A road point cloud extraction method based on BEV perception

Through BEV perception and deep model training, point cloud data processing is simplified, the problem of point cloud data conversion complexity and high computing cost is solved, fast and accurate road point cloud extraction is achieved, detection accuracy and efficiency are improved, and the Traffic safety and road management level.

CN118506016BInactive Publication Date: 2024-11-08UNIV OF JINAN
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410761196.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-13
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art has complexity and high computing costs in the three-dimensional to two-dimensional perspective conversion and BEV perception of point cloud data, making it difficult to effectively extract road point cloud information, affecting the efficiency and safety of autonomous driving and road management.

Method used

Using the BEV-perception-based road point cloud extraction method, through BEV mapping, data annotation and deep model training, point cloud data processing is simplified, and computing needs are reduced, and models such as U-Net, PSP-Net, DeepLabV3+ and HR-Net are used for this. Road feature extraction.

Benefits of technology

It realizes fast and accurate road point cloud extraction, reduces data processing complexity and calculation costs, improves detection accuracy and extraction speed, and improves traffic safety and road service levels.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118506016B_ABST
    Figure CN118506016B_ABST
Patent Text Reader

Abstract

The present invention starts from the characteristics that in the bird's-eye view (BEV) of the scene point cloud, all objects are observed from the top and roads usually appear as continuous plane areas. Through the steps of BEV coding mapping, BEV data annotation, deep model training, BEV decoding and restoration, the complex three-dimensional point cloud data is simplified into a two-dimensional image in BEV, which reduces the complexity and computing requirements of data processing, while retaining sufficient information to perform road feature extraction, simplifying the point cloud road extraction process, and realizing the accurate extraction of road point cloud.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of point cloud processing and road extraction, and in particular to a road point cloud extraction method based on BEV perception. Background Art

[0002] In the rapidly developing modern society, roads, as a key component of urban infrastructure, play an important role in connecting all corners of the city and promoting economic and social activities. With the sharp increase in the number of vehicles and the continuous increase in traffic volume, the maintenance and management of roads face unprecedented challenges. Research based on point clouds generated by LiDAR focuses on using the high precision and high efficiency characteristics of LiDAR technology to identify roads. By emitting laser pulses and receiving their reflected light, LiDAR can accurately measure the height information of the road surface and generate a detailed three-dimensional road model. This technology has shown significant advantages over traditional road extraction methods, including providing millimeter-level measurement accuracy and being able to comprehensively evaluate road conditions, including identifying road defects such as cracks and potholes.

[0003] Point cloud data is a key source of perception in autonomous driving and robotic navigation, providing accurate 3D structural information of the environment. Traditional ground-view images and data, while useful, are limited by field of view and angle and cannot provide comprehensive road information. Learning powerful representations in bird's-eye-view (BEV) for perception tasks is a trend and has attracted widespread attention from industry and academia. Traditional approaches for most autonomous driving algorithms perform detection, segmentation, tracking, etc. in the forward or forward view. As sensor configurations become increasingly complex, it becomes critical to integrate multi-source information from different sensors and represent features in a unified view. BEV perception inherits several advantages because representing the surrounding scene in BEV is intuitive and fusion-friendly; while representing objects in BEV is most needed by subsequent modules in planning and / or control. The core issues of BEV perception are: (1) how to reconstruct the lost 3D information through view-to-BEV perspective conversion; (2) how to obtain ground truth annotations in BEV meshes; and (3) how to adapt and generalize algorithms as sensor configurations change in different scenarios.

[0004] By converting point clouds into BEV images, the layout of roads and other environmental elements can be understood and analyzed more intuitively. Directly processing this data requires complex three-dimensional calculations with high computational costs. Mapping point clouds into two-dimensional BEV images can simplify the dimensions of the data and improve processing efficiency by using mature two-dimensional image processing technology. This is of great significance for subsequent accurate and efficient detection of road unevenness, ensuring traffic safety, improving road service levels, and reducing maintenance costs. Summary of the invention

[0005] In order to reduce the complexity and computing requirements of data processing, a road point cloud extraction method based on BEV perception is provided for the point cloud data generated during the driving process. Through BEV mapping, the mapped data is annotated to form a data set, and the data set is put into the image segmentation model for training, and then a pre-trained model is obtained to restore the BEV map, so as to achieve the purpose of quickly and accurately extracting road point clouds.

[0006] To achieve the above purpose, the specific implementation steps of the present invention are as follows:

[0007] S1: BEV encoding mapping;

[0008] S2: BEV data annotation;

[0009] S3: training of deep models;

[0010] S4: BEV decoding and restoration.

[0011] Furthermore, in the above S1, the specific implementation steps of the BEV coding mapping process are:

[0012] S101: Using the features of three dimensions, namely height, density and reflectivity, the point cloud is projected into a bird’s-eye view network map in 2D space with a resolution of 0.1m per pixel.

[0013] S102: The range of the LIDAR space represented by the bird's-eye view network diagram is 25.6m to the right, 25.6m to the left, and 51.2m forward; using this range with the above resolution of 0.1 results in an input shape of 512*512 for each channel; the height in the Lidar space is clipped between +2m and -2m.

[0014] S103: Map to the range of 0 to 255 to represent the pixel values ​​of the height, density, and reflectivity channels. The mapping strategy uses the Threshold-Min normalization method:

[0015]

[0016] Among them, Value is the original data value, Threshold is the set threshold, usually the maximum value or other specific value in the data set, Min is the minimum value, usually 0, Normalized Value is the normalized value, and the mapped data is backed up to the position matrix for subsequent decoding and restoration. The mapping results are shown in the attached figure. Figure 3 shown.

[0017] Furthermore, in the above S2, the specific implementation steps of the BEV data labeling process are:

[0018] S201: After obtaining the mapped BEV data, the image annotation method is used to annotate the data. To ensure the consistency of the annotation, each BEV data is annotated with only the road.

[0019] S202: The BEV dataset has 1,600 annotated images. The ratio of training set, validation set, and test set is 8:1:1. The training set contains 1,280 images, and the validation set and test set each contain 160 images.

[0020] Furthermore, in the above S3, the framework of the deep model is as shown in the attached Figure 4 As shown, the specific implementation steps of the training process are:

[0021] S301: Use a unified architecture to train point cloud BEV data. In the unified architecture, steps such as data preprocessing, feature extraction, and post-processing can be designed to be shared or partially shared, which can reduce the repeated flow of data during the processing process and reduce the overhead of data transmission and conversion.

[0022] S302: The deep model needs to be deployed, and four models, U-Net, PSP-Net, DeepLabV3+ and HR-Net, are used for deployment and training.

[0023] S303: Taking U-Net as an example, first, BEV encoding mapping is performed on the relevant features according to the channel weight of density: height: reflection intensity = 1:0:0, and data is labeled after mapping.

[0024] S304: U-Net uses ResNet50 as the backbone network for feature extraction. ResNet50 uses two basic building blocks: Basicblock and Bottleneck. Basicblock is usually used for relatively shallow networks, while Bottleneck is often used for deeper networks. ResNet50 contains multiple residual blocks, each of which consists of several convolutional layers. ResNet50 uses the powerful feature extraction capability of ResNet50 as the downsampling path of U-net, and connects the intermediate layer output of ResNet50 with the corresponding layer in the upsampling path of U-Net to improve segmentation accuracy.

[0025] S305: The input size is set to 512*512, the batch size is set to 6, the optimizer is SGD, the weight decay is set to 0.0001, the momentum parameter used in the optimizer is 0.9, the maximum learning rate is 0.0001, the minimum learning rate is 0.000001, the learning rate reduction method uses cos, the epoch is set to 500, and the weight is saved every 5 epochs.

[0026] S306: Selection accuracy , completeness , inspection quality ,Three metric indicators are used to evaluate the extraction effect of roads and road defects, accuracy Focus on the proportion of real roads in the extracted road points and their completeness Focus on extracting road points to cover the real road as much as possible and test quality Taking into account the accuracy , completeness There are two indicators. The calculation formula of the evaluation indicator is:

[0027] , where TP is actually a road defect point and is judged to be a road defect point; FN is actually a road defect point but is judged to be a non-road defect point; FP is actually a non-road defect point but is judged to be a road defect point.

[0028] In the above S4, after obtaining the pre-trained model, the model output is still the BEV feature map, which needs to be restored pixel by pixel based on the position matrix obtained in S2. The restoration requires querying the original point cloud, and the query result in the original point cloud is the road point cloud.

[0029] The beneficial effect of the present invention is that, compared with the prior art, the present invention simplifies complex three-dimensional point cloud data into two-dimensional images in BEV, reducing the complexity and computing requirements of data processing, while retaining sufficient information to perform road feature extraction, simplifying the point cloud road extraction process, and improving detection accuracy and extraction speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Design a flow chart for the present invention;

[0031] Figure 2 Schematic diagram of BEV encoding mapping process;

[0032] Figure 3 Mapping the generated BEV visualization results for the Threshold-Min normalization method;

[0033] Figure 4 A diagram of the deep model training architecture using U-Net;

[0034] Figure 5 Road extraction accuracy training curves for 500 epochs for four models;

[0035] Figure 6 Road extraction completeness training curves for 500 epochs of four models;

[0036] Figure 7Road extraction detection quality training curves for 500 epochs for four models. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] The present invention provides a road point cloud extraction method based on BEV perception. Through BEV mapping, the mapped data is annotated to form a data set, the data set is put into the image segmentation model for training, and then a pre-trained model is obtained, the BEV map is restored, and the road point cloud is restored. The specific implementation steps are as shown in the attached figure. Figure 1 shown.

[0039] The driving point cloud dataset uses the Ithaca365 dataset, which uses a novel data collection process to repeatedly record data along a 15-kilometer route in different scenes (city, highway, rural, campus), weather (snow, rain, sun), time (day / night), and traffic conditions (pedestrians, cyclists, and cars). The dataset includes images and point clouds from four cameras and LiDAR sensors, and the dataset includes 7,000 frames in 40 sets.

[0040] S1: BEV encoding mapping;

[0041] S2: BEV data annotation;

[0042] S3: training of deep models;

[0043] S4: BEV decoding and restoration.

[0044] Further, in the above S1, the BEV encoding mapping process is as shown in the attached Figure 2 As shown, the specific implementation steps are:

[0045] S101: Using the features of three dimensions, namely height, density and reflectivity, the point cloud is projected into a bird’s-eye view network map in 2D space with a resolution of 0.1m per pixel.

[0046] S102: The range of the LIDAR space represented by the bird's-eye view network diagram is 25.6m to the right, 25.6m to the left, and 51.2m forward; using this range with the above resolution of 0.1 results in an input shape of 512*512 for each channel; the height in the Lidar space is clipped between +2m and -2m.

[0047] S103: Map to the range of 0 to 255 to represent the pixel values ​​of the height, density, and reflectivity channels. The mapping strategy uses the Threshold-Min normalization method:

[0048]

[0049] Among them, Value is the original data value, Threshold is the set threshold, usually the maximum value or other specific value in the data set, Min is the minimum value, usually 0, Normalized Value is the normalized value, and the mapped data is backed up to the position matrix for subsequent decoding and restoration. The mapped BEV results are shown in the attached figure. Figure 3 shown.

[0050] Furthermore, in the above S2, the specific implementation steps of the BEV data labeling process are:

[0051] S201: After obtaining the mapped BEV data, the image annotation method is used to annotate the data. To ensure the consistency of the annotation, each BEV data is annotated with only the road.

[0052] S202: The BEV dataset has 1,600 annotated images. The ratio of training set, validation set, and test set is 8:1:1. The training set contains 1,280 images, and the validation set and test set each contain 160 images.

[0053] Furthermore, in the above S3, the framework of the deep model is as shown in the attached Figure 4 As shown in the figure, the specific implementation steps of the training process are:

[0054] S301: Use a unified architecture to train point cloud BEV data. In the unified architecture, steps such as data preprocessing, feature extraction, and post-processing can be designed to be shared or partially shared, which can reduce the repeated flow of data during the processing process and reduce the overhead of data transmission and conversion.

[0055] S302: The deep model needs to be deployed, and four models, U-Net, PSP-Net, DeepLabV3+ and HR-Net, are used for deployment and training.

[0056] S303: Taking U-Net as an example, according to the attached Figure 4 ,First, the relevant features are mapped by BEV encoding according to the ,channel weights of density: height: reflection intensity = 1: 0: 0, and the data is labeled after ,mapping.

[0057] S304: U-Net uses ResNet50 as the backbone network for feature extraction. ResNet50 uses two basic building blocks: Basicblock and Bottleneck. Basicblock is usually used for relatively shallow networks, while Bottleneck is often used for deeper networks. ResNet50 contains multiple residual blocks, each of which consists of several convolutional layers. ResNet50 uses the powerful feature extraction capability of ResNet50 as the downsampling path of U-net, and connects the intermediate layer output of ResNet50 with the corresponding layer in the upsampling path of U-Net to improve segmentation accuracy.

[0058] S305: The input size is set to 512*512, the batch size is set to 6, the optimizer is SGD, the weight decay is set to 0.0001, the momentum parameter used in the optimizer is 0.9, the maximum learning rate is 0.0001, the minimum learning rate is 0.000001, the learning rate reduction method uses cos, the epoch is set to 500, and the weight is saved every 5 epochs.

[0059] S306: Selection accuracy , completeness , inspection quality ,Three metric indicators are used to evaluate the extraction effect of roads and road defects, accuracy Focus on the proportion of real roads in the extracted road points and their completeness Focus on extracting road points to cover the real road as much as possible and test quality Taking into account the accuracy , completeness There are two indicators. The calculation formula of the evaluation indicator is:

[0060] , where TP is actually a road defect point and is judged to be a road defect point; FN is actually a road defect point but is judged to be a non-road defect point; FP is actually a non-road defect point but is judged to be a road defect point.

[0061] S307: Replace U-Net in S303 with PSP-Net network, use Mobile-net as the backbone network, set the input image to 512*512, set the downsampling multiple to 8, set the batch size to 6, use SGD as the optimizer, set the weight decay to 0.0001, set the momentum parameter used in the optimizer to 0.9, set the maximum learning rate to 0.01, set the minimum learning rate to 0.0001, use cosine as the learning rate reduction method, set the epoch to 500, and save the weight every 5 epochs.

[0062] S308: Replace U-Net in S303 and use DeeplabV3+ network. The backbone network uses Xception, the input image is set to 512*512, the downsampling multiple is set to 8, the batch size is set to 6, the optimizer is SGD, the weight decay is 0.0001, the momentum parameter used in the optimizer is set to 0.9, the maximum learning rate is 0.007, the minimum learning rate is 0.00007, the learning rate reduction method uses cos, the epoch is set to 500, and the weight is saved every 5 epochs.

[0063] S309: Replace U-Net in S303 with HR-Net network. The backbone network uses hrnetv2_w18. The input image is set to 512*512, the batch size is set to 6, the optimizer is SGD, the weight decay is 0.0001, the momentum parameter used in the optimizer is set to 0.9, the maximum learning rate is 0.004, the minimum learning rate is 0.00004, the learning rate reduction method uses cos, the epoch is set to 500, and the weight is saved every 5 epochs.

[0064] S310: Train the above four models and the accuracy of the four models , completeness , inspection quality The training comparison results are shown in the attached Figure 5 , attached Figure 6 , attached Figure 7 As shown in the figure, according to the training curves of the four models, the network model with the highest road extraction accuracy is PSP-Net, which reaches 84.31%, followed by HR-Net; the network model with the highest road extraction completeness is DeeplabV3+, which reaches 86.67%, followed by HR-Net; the network model with the highest road detection quality is HR-Net, which reaches 80.67%, followed by U-Net. In terms of comprehensive performance, HR-Net and U-Net are better than PSP-Net and DeeplabV3+. From the perspective of road detection quality, HR-Net is a better choice.

[0065] In the above S4, after obtaining the pre-trained model, the model output is still the BEV feature map, which needs to be restored pixel by pixel based on the position matrix obtained in S2. The restoration requires querying the original point cloud, querying the result in the original point cloud, and finally obtaining the road point cloud.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A road point cloud extraction method based on BEV perception, characterized in that: The specific implementation steps are as follows: S1: BEV encoding mapping; S2: BEV data annotation; S3: training of deep models; S4: BEV decoding and restoration; In the above S1, the specific implementation steps of the BEV coding mapping process are: S101: Using the features of three dimensions, namely height, density and reflectivity, the point cloud is projected into a bird’s-eye view network map in 2D space with a resolution of 0.1m per pixel; S102: The range of LIDAR space represented by the bird's-eye view network diagram is 25.6m to the right, 25.6m to the left, and 51.2m forward; using this range with the above resolution of 0.1 results in an input shape of 512*512 for each channel; the height in the Lidar space is clipped between +2m and -2m; S103: Mapping to the range of 0 to 255 to represent the pixel values ​​of the height, density, and reflectivity channels; the mapping strategy uses the Threshold-Min normalization method: ; Among them, Value is the original data value, Threshold is the set threshold, usually the maximum value or other specific value in the data set, Min is the minimum value, usually 0, Normalized Value is the normalized value, and the mapped data is backed up in the position matrix for subsequent decoding and restoration; In the above S2, the specific implementation steps of the BEV data labeling process are: S201: After obtaining the mapped BEV data, the image annotation method is used to annotate the data. To ensure the consistency of the annotation, each BEV data is annotated with only the road; S202: The BEV dataset has 1,600 annotated images. The ratio of training set, validation set, and test set is 8:1:

1. The training set contains 1,280 images, and the validation set and test set each contain 160 images. In the above S3, the specific implementation steps of the training process are: S301: Use a unified architecture to train point cloud BEV data. In the unified architecture, steps such as data preprocessing, feature extraction, and post-processing can be designed to be shared or partially shared, which can reduce the repeated flow of data during the processing and reduce the overhead of data transmission and conversion; S302: The deep model needs to be deployed, and four models, U-Net, PSP-Net, DeepLabV3+ and HR-Net, are used for deployment and training; In the above S4, after obtaining the pre-trained model, the model output is still the BEV feature map, which needs to be restored pixel by pixel based on the position matrix obtained in S2. The restoration requires querying the original point cloud, and the query result in the original point cloud is the road point cloud.