Vehicle panoramic 3D intelligent sensing system and method based on BEVDet

Abstract

The invention discloses a BEVDet-based vehicle panorama 3D intelligent sensing system and method, and the system comprises a data collection module which employs six cameras and a laser radar, carries out the arrangement according to a 360-degree panorama requirement, and collects image data and laser radar point cloud data used for 3D marking; and the data processing module is responsible for sensor calibration and data annotation, including camera calibration, camera and laser radar joint calibration and target annotation of image data. According to the system, a self-made data set is adopted for model design, and a panoramic 3D sensing algorithm is constructed through a view encoder, a view converter and a BEV encoder. In the model training stage, a bevdet-r50-cbgs parameter file is used for training, and data enhancement and model optimization are carried out for an overfitting problem in BEV space learning. And finally, the model is deployed on an NVIDIA AGX Xavier platform and is optimized and accelerated through TensorRT, so that the reasoning speed and the real-time performance of the model are improved. The method is suitable for mining area and port vehicle early warning, active safety anti-collision and other scenes, and has wide application prospects.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent assisted driving, and specifically relates to a vehicle panoramic 3D intelligent perception system and method based on BEVDet. Background Art

[0002] Vehicle panoramic 3D intelligent perception technology refers to the use of multiple sensors, such as lidar, camera or radar, to obtain three-dimensional information about the vehicle's surroundings, including the location, size and category of objects, to achieve all-round perception and understanding of the vehicle's surroundings, and assist in completing the vehicle's personnel approach warning function. At present, the following two methods are mostly used for panoramic 3D intelligent perception technology, but each method has its shortcomings:

[0003] (1) Perform detection and perception tasks independently on multiple cameras, and then fuse these independent perception results together. This method greatly increases the complexity of the entire system because each camera has a unique field of view and observation results; in addition, the depth estimation results may be greatly biased by the camera's viewing angle, resolution, and other factors.

[0004] (2) Intelligent perception based on LiDAR point cloud: LiDAR can accurately obtain three-dimensional point cloud data of the surrounding environment by emitting laser beams and measuring the time and intensity of their reflection. However, this method has high technical costs, uneven data density obtained by LiDAR, and low real-time detection performance. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a vehicle panoramic 3D intelligent perception system and method based on BEVDet.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A vehicle panoramic 3D intelligent perception system based on BEVDet includes: a data acquisition module, which is equipped with multiple cameras and laser radars and installed on the front, back, left and right sides of the vehicle, wherein the laser radar is installed on the roof of the vehicle and is used to collect 360° panoramic images and point cloud data; a data processing module, which is connected to the data acquisition module and is used to process the collected original image data; a model design module, which is connected to the data processing module and is used to design a panoramic 3D perception algorithm based on a BEVDet model; a model training module, which is connected to the model design module and is used to train the BEVDet model; a model performance evaluation and optimization module, which is connected to the model training module and is used to deploy the trained model and optimize and accelerate it to improve the reasoning speed and finally realize the real-time reasoning detection task.

[0008] As a preferred embodiment of the above scheme, the data acquisition module is configured with six cameras and laser radars, which are respectively installed on the front left, front, front right, rear left, rear, and rear right of the vehicle.

[0009] As a preferred embodiment of the above solution, the model design module includes an image view encoder, a view converter, a BEV encoder and a 3D detection head which are interconnected.

[0010] A vehicle panoramic 3D intelligent perception method based on BEVDet includes the following steps:

[0011] S1. Install six cameras and laser radar on the vehicle to obtain 360° panoramic images and 3D frame annotations of the images;

[0012] S2. After data collection, data processing is performed and data sets are produced, including sensor calibration and data labeling, to obtain calibration data and labeling data required for model training respectively;

[0013] S3. Design a panoramic 3D perception algorithm suitable for the self-made dataset based on the BEVDet model, and use the surround view 2D image to convert into BEV view for 3D object detection;

[0014] S4, train the BEVDet model, perform data enhancement operations on the overfitting problem in BEV space learning, upgrade the classic non-maximum suppression NMS, and train the optimized model;

[0015] S5. After training, the model is deployed on the NVIDIA AGX Xavier edge computing platform to evaluate model performance and further optimize it, and optimized and accelerated through TensorRT.

[0016] As a preferred embodiment of the above solution, in step S2, the sensor calibration includes camera calibration and camera and lidar joint calibration, and the data annotation includes the category, 3D frame position, size and direction information of the target.

[0017] As a preferred embodiment of the above solution, in step S3, the panoramic 3D perception algorithm specifically includes:

[0018] S31, extracting features from the input surround view image;

[0019] S32, converting features from image view to BEV view;

[0020] S33, further encode features and perform 3D target detection.

[0021] As a preferred embodiment of the above scheme, in step S4, the model training module uses the bevdet-r50-cbgs parameter file for training and performs data enhancement operations.

[0022] Due to the above structure, the beneficial effects of the present invention are:

[0023] The present invention designs a vehicle panoramic 3D intelligent perception system and method based on the BEVDet model. It is based on a visual camera and involves steps such as data acquisition, data processing, model design, model training and performance optimization. In order to reduce the large amount of algorithm calculation and long time consumption caused by the system simultaneously acquiring 6-way camera data and detecting, the model reasoning speed is improved by optimizing the deployment of the model; in order to improve the practicality of the model, the interface is designed using the method of software and hardware collaborative design, and related issues are studied to optimize the real-time performance of the system. The technical solution of the present invention can be applied to multiple fields and has a high promotion and application value. For relevant personnel in the technical field, various improvements and changes can be made according to the spirit and principles of the present invention to adapt to different application requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below.

[0025] Figure 1 A schematic diagram of the hardware platform interface design of the system of the present invention;

[0026] Figure 2 A schematic diagram of the hardware platform of the system of the present invention;

[0027] Figure 3 The panoramic image data collected are: (a) is the front left camera, (b) is the front camera, (c) is the front right camera, (d) is the rear right camera, (e) is the rear camera, and (f) is the rear left camera;.

[0028] Figure 4 Calibrate the interface for the camera;

[0029] Figure 5 Joint calibration interface for camera and lidar;

[0030] Figure 6 Provides an interface for 3D box annotation;

[0031] Figure 7 To obtain the final loss function after training;

[0032] Figure 8 Comparison of the accuracy before and after the model TensorRT optimization acceleration, (a) is before optimization acceleration, (b) is after optimization acceleration. DETAILED DESCRIPTION

[0033] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention.

[0034] BEV (Bird's Eye View) is a perspective for viewing an object or scene from above, just like a bird looking down at the ground in the air. The data obtained by LiDAR or camera is converted into BEV representation, which can better perform tasks such as 3D detection and perception of objects. BEV can simplify complex three-dimensional environments into two-dimensional images, which can effectively improve the real-time performance of detection; in addition, BEV space provides a unique visual effect, making the objects and spatial relationships in the scene more clearly visible, which is more conducive to the depth estimation of objects and reduces the error of object distance detection.

[0035] There are many 3D object detection models based on BEV space, such as BEVDet, BEVDepth, and BEVFusion, etc. The BEVDet model can predict the position, size, and category information of objects around the vehicle from images transmitted by cameras from multiple perspectives; the BEVDepth model focuses more on predicting the precise distance information of objects in the vehicle's surrounding environment; the BEVFusion model combines the camera's images and the LiDAR's point cloud data to obtain more comprehensive and consistent environmental perception information, but the cost is also higher.

[0036] Compared with the BEVDepth model and the BEVFusion model, the BEVDet model perceives richer 3D information about objects, and only requires camera image data, not point cloud data, with lower costs, more mature technology, and better accuracy and real-time performance. Therefore, this embodiment designs a panoramic 3D visual perception algorithm based on BEVDet to perceive the 3D frame position, size, category, and distance of objects within a 360° range around the vehicle.

[0037] Specifically, Figure 1 As shown, this embodiment provides a vehicle panoramic 3D intelligent perception system based on BEVDet, including:

[0038] The data acquisition module is equipped with multiple cameras and laser radars, which are installed at the front, back, left and right of the vehicle. The laser radar is installed on the roof to collect 360° panoramic images and point cloud data for 3D annotation. In this embodiment, the data acquisition module is equipped with six cameras and laser radars, which are installed at the front left, front, front right, rear left, rear and rear right of the vehicle respectively;

[0039] A data processing module, connected to the data acquisition module, for processing the acquired raw image data;

[0040] The model design module is connected to the data processing module and is used to design a panoramic 3D perception algorithm based on the BEVDet model. Specifically, it includes interconnected image view encoders, view converters, BEV encoders, and 3D detection heads. The image view encoder is used to extract features from the input surround view image, the view converter converts features from image views to BEV views, the BEV encoder is used to further encode features, and the 3D detection head is used for target detection tasks.

[0041] The model training module is connected with the model design module and is used to train the BEVDet model;

[0042] The model performance evaluation and optimization module is connected to the model training module to deploy the trained model and optimize and accelerate it to improve the reasoning speed and ultimately achieve real-time reasoning detection tasks.

[0043] This embodiment also provides a vehicle panoramic 3D intelligent perception method based on BEVDet, comprising the following steps:

[0044] S1. Build the data collection module as follows Figure 2 As shown in the figure, according to the requirements of 360° panoramic view, six cameras are installed in the front, back, left and right of the car, divided into front left, front, front right, rear left, rear and rear right. The laser radar is installed on the roof, and the laser radar point cloud data is used for 3D data annotation. The installation requirements should meet the requirements that the images taken by each camera have overlapping parts with the adjacent cameras, and the collected image data is as follows Figure 3 shown.

[0045] S2. After data collection, the collected raw image data is processed and a data set is created. The self-made data set includes sensor calibration and data annotation. Sensor calibration obtains the calibration data required for model training, and data annotation obtains the annotation data required for model training. Among them, sensor calibration includes camera calibration and camera and lidar joint calibration. Data annotation includes target category, 3D frame position, size, direction and other information. Specifically:

[0046] 1) Sensor calibration

[0047] In order to obtain the camera parameters (calibration data) required by the BEVDet model, sensor calibration is required. Sensor calibration includes camera calibration and camera and lidar joint calibration. Camera calibration obtains the camera intrinsic parameters required by the model. This parameter reflects the mapping relationship between environmental information and image information. It can be obtained using the camera_calibration tool in ROS, such as Figure 4As shown in the figure, the camera and lidar are calibrated together to obtain the external parameters required by the model. This parameter reflects the position relationship between the two sensors and can be obtained using the SensorsCalibration tool, such as Figure 5 shown.

[0048] Table 1 Annotation files obtained by SUSTechPOINTS annotation tool

[0049]

[0050] 2) Data labeling

[0051] Data annotation is to annotate the collected images and obtain annotation data. The annotation content usually includes the category of the target (such as vehicles, pedestrians, etc.), 3D frame position, size, direction and other information. The annotation data obtained by the SUSTechPOINTS annotation tool is based on the principle of annotating the 3D frame in the point cloud data, and then projecting the 3D annotation frame onto the image using the camera internal and external parameters, such as Figure 6 The information contained in the labeled data is shown in Table 1. Data labeling is the core link in making a dataset, and the accuracy of labeling directly affects the training effect of the model and the final detection performance.

[0052] S3. Design a panoramic 3D perception algorithm suitable for self-made data sets based on the BEVDet model. The BEVDet model is a perception framework that uses surround 2D images to detect 3D targets under the BEV bird's-eye view. It uses a pure visual solution (without relying on lidar) to perceive the environment, and maps 2D images into 3D images through the view cone to construct pseudo point cloud data for BEV spatial encoding. It is low-cost and efficient. The BEVDet algorithm mainly includes the following four parts:

[0053] 1) Image view encoder

[0054] The main function of the image view encoder is to extract features from the input surround view image, including a feature extraction network (backbone network) and a feature fusion network.

[0055] The backbone network is explained using ResNet-50 as an example. First, the backbone network extracts multi-scale features from the input surround image (denoted as Tensor([bs,N,3,H,W])); where bs = batchsize, N is the number of surround images, and H and W are the width and height of the input image. The resolution of the multi-scale features is as follows:

[0056]

[0057] The multi-scale features extracted by the backbone network are fused, and the fused features are:

[0058]

[0059] The mathematical description of the feature fusion part is as follows:

[0060] F=Conv 3×3 (Up(Conv 1×1 (L 1 ))+Conv 1×1 (L 0 )),

[0061] Where Up(·) represents upsampling, Conv 1×1 (·) for dimensionality reduction, Conv 3×3 (·) is used to further extract features from the fused feature map.

[0062] 2) View Converter

[0063] The view converter converts features from image views to BEV views. This module mainly completes the following three functions: using the depth estimation network to predict the discrete depth information and semantic features of each cell in the feature map, and using the outer product operation to construct 3D cone features; generating cone 3D coordinate points and completing the projection mapping to BEV space; and obtaining BEV space features based on the mapped 3D coordinate points and the generated cone features.

[0064] 3) BEV encoder

[0065] The BEV encoder further encodes features in the BEV space. The role of this module is to further extract and fuse the features of the BEV space output by the view converter, including the BEV backbone network and the BEV feature fusion network.

[0066] The BEV backbone network extracts multi-scale features from the output BEV features. The resolution of the multi-scale features is as follows:

[0067] B0=Tensor([bs,160,64,64]),

[0068] B1=Tensor([bs,320,32,32]),

[0069] B2 = Tensor([bs,640,16,16]).

[0070] The BEV feature fusion network mainly fuses the multi-scale features output by the BEV backbone network. The features output by the feature fusion network correspond to Tensor ([bs, 256, 128, 128]). The mathematical description of the corresponding logic is as follows:

[0071] F=Up(Conv(Concat(B2,B0))),

[0072] Concat(·) is used to merge tensor data in the channel dimension, Conv(·) is used for data dimensionality reduction, and Up(·) is used for data upsampling.

[0073] 4) 3D detection head

[0074] The 3D detection head is built based on a specific target detection task. The 3D detection head in this system is designed for the position and distance of movable objects such as pedestrians.

[0075] S4. Train the BEVDet model. During the training phase, the bevdet-r50-cbgs parameter file is used for training. To address the overfitting problem in BEV spatial learning, additional data enhancement operations such as flipping, scaling, and rotation are performed to improve the robustness of the model. In addition, this system upgrades the classic non-maximum suppression (NMS) strategy to improve its adaptability in 3D object detection scenarios. It further speeds up the reasoning process by removing operators that are executed sequentially. The final loss function obtained after training is as follows: Figure 7 As shown in the figure, the change curves of the object classification loss and object regression box loss function show a decreasing trend, and drop to about 0.2 after 30,000 iterations.

[0076] S5. After training, first, deploy the trained model on the NVIDIA AGX Xavier edge computing platform, and configure the environment as CUDA11.4+Pytorch1.11. After that, optimize and accelerate it through TensorRT. This method greatly improves the reasoning speed while ensuring accuracy, making it suitable for real-time detection tasks, and improves the practicality of the system, making it suitable for application scenarios such as mining and port vehicle safety warnings. Finally, real-time reasoning detection is performed. The input size of the image during detection is (256, 704, 3), and each frame simultaneously inputs 6 images of "CAMERA_FRONT_LEFT", "CAMERA_FRONT", "CAMERA_FRONT_RIGHT", "CAMERA_BACK_LEFT", "CAMERA_BACK", and "CAMERA_BACK_RIGHT". The reasoning performance of the model is shown in Table 2 below:

[0077] Table 2 Comparison of model performance before and after TensorRT acceleration

[0078] Model Accuracy (%) FPS Model size Bevdet-r50-cbgs.pth 56.06 3.6 169M Trtbevdet_int8_fuse.engine 55.87 26.3 46.3M

[0079] The performance of the detection model was tested on the test set. Before the model was optimized and accelerated using TensorRT, the accuracy on the main categories of pedestrians and vehicles was 0.578 and 0.420 respectively; after the model was optimized and accelerated using TensorRT, the accuracy on the main categories of pedestrians and vehicles was 0.441 and 0.330 respectively, with a slight decrease in accuracy. Figure 8 shown.

[0080] Model computing power (FLOPs) and parameter volume (Params) are two important indicators for measuring the complexity of deep learning algorithms. They can be used to evaluate the performance and practicality of a model. The number of parameters required for model training is calculated to be 44.25M, the model computing power FLOPs is 148.77GFLOPs, and the computing power of the deployed NVIDIA AGX Xavier platform is 30TOPS, which meets the platform deployment requirements.

[0081] The vehicle panoramic 3D intelligent perception system proposed in the present invention is based on visual cameras, which collects, calibrates, and produces data sets, and designs a visual 3D panoramic perception algorithm suitable for self-made data sets based on the BEVDet model. At the same time, in order to reduce the problems of large algorithm calculation and long time consumption caused by the system simultaneously acquiring six cameras and detecting, the model reasoning speed is improved by optimizing the deployment of the model. In order to improve the practicality of the model, the interface design is carried out by adopting the method of software and hardware collaborative design, and the relevant issues are studied, and the real-time performance of the system is studied. Through the development of this system, not only the visual 3D perception function can be realized, but also more downstream tasks can be served. This system has good reasoning speed and accuracy, is suitable for real-time vehicle perception and warning tasks, and can be productized and promoted in the following fields: 1) Approaching warning of vehicles and personnel in mining areas and ports; 2) Active safety collision avoidance of vehicles in mining areas and ports.

[0082] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A vehicle panoramic 3D intelligent perception system based on BEVDet, characterized in that: include: The data acquisition module is equipped with multiple cameras and laser radars, which are installed on the front, back, left and right sides of the vehicle. The laser radar is installed on the roof to collect 360° panoramic images and point cloud data; A data processing module, connected to the data acquisition module, for processing the acquired raw image data; The model design module is connected to the data processing module and is used to design a panoramic 3D perception algorithm based on the BEVDet model; The model training module is connected with the model design module and is used to train the BEVDet model; The model performance evaluation and optimization module is connected to the model training module to deploy the trained model and optimize and accelerate it to improve the reasoning speed and ultimately achieve real-time reasoning detection tasks.

2. A vehicle panoramic 3D intelligent perception system based on BEVDet according to claim 1, characterized in that: The data acquisition module is equipped with six cameras and laser radars, which are installed on the front left, front, front right, rear left, rear, and rear right of the vehicle respectively.

3. The vehicle panoramic 3D intelligent perception system based on BEVDet according to claim 1 is characterized in that: The model design module includes an image view encoder, a view converter, a BEV encoder, and a 3D detection head that are interconnected.

4. A vehicle panoramic 3D intelligent perception method based on BEVDet according to any one of claims 1 to 3, characterized in that: The following steps are involved: S1. Install six cameras and laser radar on the vehicle to obtain 360° panoramic images and 3D frame annotations of the images; S2. After data collection, data processing is performed and data sets are produced, including sensor calibration and data labeling, to obtain calibration data and labeling data required for model training respectively; S3. Design a panoramic 3D perception algorithm suitable for the self-made dataset based on the BEVDet model, and use the surround view 2D image to convert into BEV view for 3D object detection; S4, train the BEVDet model, perform data enhancement operations on the overfitting problem in BEV space learning, upgrade the classic non-maximum suppression NMS, and train the optimized model; S5. After training, the model is deployed on the NVIDIA AGX Xavier edge computing platform to evaluate model performance and further optimize it, and optimized and accelerated through TensorRT.

5. The vehicle panoramic 3D intelligent perception method based on BEVDet according to claim 4 is characterized in that: In step S2, sensor calibration includes camera calibration and camera and lidar joint calibration, and data annotation includes target category, 3D frame position, size and direction information.

6. The vehicle panoramic 3D intelligent perception method based on BEVDet according to claim 4 is characterized in that: In step S3, the panoramic 3D perception algorithm specifically includes: S31, extracting features from the input surround view image; S32, converting features from image view to BEV view; S33, further encode features and perform 3D target detection.

7. The vehicle panoramic 3D intelligent perception method based on BEVDet according to claim 4 is characterized in that: In step S4, the model training module uses the bevdet-r50-cbgs parameter file for training and performs data enhancement operations.

Images