A roadside 3D target detection deep learning algorithm based on combination of a camera and a laser radar

Abstract

This invention relates to the field of 3D target detection technology, and in particular to the design of a deep learning algorithm for roadside 3D target detection based on the combination of cameras and LiDAR. The specific steps are as follows: Step S1, data construction and data preprocessing; Step S2, image feature extraction; Step S3, LiDAR feature extraction; Step S4, feature fusion; Step S5, task head decoding. This invention effectively solves the current situation where 3D detection algorithms neglect the role of roadside cameras and radar in road traffic and the successful application of LiDAR data in multi-sensor fusion.

Description

Technical Field

[0001] This invention relates to the field of 3D target detection technology, specifically a deep learning algorithm for roadside 3D target detection based on a combination of camera and lidar. Technical Background

[0002] 3D object detection algorithms have developed rapidly in recent years. Currently popular detection methods mainly cover 2D monocular images, stereo cameras, radar, and other monocular or multi-sensor fusion methods, which have achieved good accuracy in 3D object detection. However, in general, current methods are based on building networks for 3D object detection using data from vehicle-mounted cameras or radar, neglecting the role of important roadside sensors such as roadside cameras and radar. Applying existing advanced vision-centric bird's-eye view detection methods to roadside cameras has revealed poor performance. The reason is that these methods mainly predict the center depth of objects in the bird's-eye view, but the depth difference decreases rapidly as the car moves away from the roadside camera, resulting in a significant drop in recognition accuracy.

[0003] In recent years, numerous 3D detection methods using monocular cameras have emerged, with methods projecting 2D images into 3D space showing strong momentum. FCOS3D treats 3D object detection as a 2D problem, perceiving only within the image view. Due to the strong spatial correlation between target attributes and image appearance, it can predict this well. Furthermore, utilizing the bird's-eye view (BEV) feature space to reduce the Z-axis degree of freedom can significantly improve the perception performance of the vision-centric system. However, these methods still lag significantly behind the accuracy of multi-sensor fusion methods. Cameras contain rich semantic information, compensating for the lack of semantic information in radar data, while radar contains rich depth information, correcting the insufficient depth estimation of cameras. Therefore, multi-sensor fusion is increasingly gaining popularity in the field of 3D object detection.

[0004] In summary, this invention reduces the depth estimation error of distant objects by designing a deep learning algorithm for roadside 3D target detection based on a combination of camera and LiDAR, and achieves high-precision roadside 3D target detection by integrating multiple sensors. Summary of the Invention

[0005] In view of the current situation where 3D detection algorithms neglect the role of roadside cameras and radar in road traffic and the successful application of LiDAR data in multi-sensor fusion, the purpose of this invention is to provide a deep learning algorithm for roadside 3D target detection based on the combination of cameras and LiDAR, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A deep learning algorithm for roadside 3D target detection based on a combination of camera and LiDAR, characterized by the following steps:

[0008] Step S1, Data Construction and Preprocessing: The open-source datasets for 3D object detection in road tests are mainly DAIR-V2X-IDataset, A9 Intersection Dataset and Repo3D Dataset. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above three datasets. The common practice is to convert the above data into the KITTI dataset format and then input it into the model. After converting the dataset, feature extraction is performed.

[0009] Step S2, Image Feature Extraction: Image features are extracted by constructing an Image HeightProjector Network (IHPN). Using RGB images as input data, a ResNet101 network is used for feature extraction, and then an FPN is used to perform multi-scale fusion of the extracted features to obtain the extracted image features.

[0010] Based on image features, the features are divided into a semantic information branch and a pixel height prediction branch. For the semantic information branch, a compression and activation network layer, namely the SE-layer, is used to extract the semantic information of the region of interest in the image. For the pixel height prediction branch, multiple residual blocks plus a deformable convolutional layer are used to predict the height of each pixel. Then, the semantic information branch and the pixel height prediction branch are fused by dot product to obtain the final 2D image features. Next, the 2D image features are mapped to 3D features. By creating a virtual coordinate system with a depth of 1, the 2D features are associated with the virtual coordinate system. Then, according to the principle of similar triangles, the 2D features of the road test camera are transformed into 3D space. Finally, Voxel Pooling is used to generate BEV features of points. At this point, the image pixel height projection network, namely IHPN, is completed, and the image features are transformed from 2D to 3D BEV features.

[0011] Step S3, LiDAR feature extraction: For LiDAR data feature extraction, the input point cloud data is first cropped, retaining only four dimensions, namely X, Y, Z and density. The scattering method is used to increase the density of the single frame point cloud scan.

[0012] Then, the point cloud is divided into ground point cloud and non-ground point cloud. Outlier removal is then performed on the ground point cloud to filter noise. After preprocessing, VoxelPooling features are extracted from the point cloud to generate BEV features of the point cloud.

[0013] Step S4, Feature Fusion: Based on the extracted image BEV features and point cloud BEV features, the semantic features generated from the RGB image are fused with the geometric features of the LiDAR point cloud. The original features are then upsampled twice and downsampled once. Feature maps of the same size are fused to obtain the final fused feature map.

[0014] Step S5, Task Head Decoding: Based on the feature map above, after one upsampling and shared convolution, and using ScaleNMS to perform non-maximum suppression on the 3D bounding boxes, six non-shared task heads are finally decoded. Each task head contains six branches: branch one is the offset along the x and y axes in the 2D dimension, i.e., the reg branch; branch two is the z-axis, which is the height information of the predicted object, i.e., the height branch; branch three is the size information of the object, i.e., the dim branch, where the size information is length-width-height; branch four is the sine and cosine of the yaw angle of the object, i.e., the rot branch; branch five is the velocity of the object along the x and y axes, i.e., the vel branch; and branch six is ​​the classification confidence, i.e., the heatmap branch.

[0015] As a preferred embodiment of the present invention, the camera image in S1 has a length of 1920 and a width of 1080, 10 types of real labels and 2D bounding boxes, and point cloud data with dimensions N*3, where N is the number of point clouds and 3 is the number of channels.

[0016] As a preferred embodiment of the present invention, the input image in S2 is cropped to a size of 704*256 with a depth of 3. The cropped image is then input into the ResNet101 network to extract image features. The extracted features are then fused at multiple scales using FPN to obtain the extracted image features.

[0017] Based on image features, the features are divided into a semantic information branch and a pixel height prediction branch. For the semantic information branch, a compression and activation network layer, namely the SE-layer, is used to extract the semantic information of the region of interest in the image. The semantic information branch is represented as F. context The pixel height prediction branch uses four residual blocks and one deformable convolutional layer to predict the height of each pixel. The pixel height prediction branch is represented as PH. prediction To optimize computation speed, one-hot encoding is introduced, which encodes continuous heights PH. prediction Discretization is performed, and the specific formula is as follows:

[0018]

[0019] Where h represents the height from the ground to the camera, h max and h min Indicates the maximum and minimum height values, M represents the number of discrete height columns, and hi This represents the i-th height column. ε is a hyperparameter that controls the density of the height columns;

[0020] The semantic information branch and the pixel height prediction branch are then fused together using a dot product to obtain the final 2D image features. The fusion formula is as follows:

[0021] F fuse =F context ⊙PH prediction (2)

[0022] Next, the 2D features of the image are mapped to 3D features. For a pixel P(u,v) in the image plane,

[0023] First, create a virtual coordinate system with the origin of the camera coordinate system, the y-axis perpendicular to the ground, and a depth d of 1. Then, use the transformation formula from pixel coordinate system to camera coordinate system to transform the pixels from pixel space to camera space:

[0024] P cam =K -1 d[u,v,1] T =K -1 [u,v,1] T (3)

[0025] Where K is the rigid transformation and d is the depth with a value of 1; let T be the transformation matrix from the camera coordinate system to the virtual coordinate system. virtual This allows you to map points from the camera coordinate system to the virtual coordinate system.

[0026] P cam-virtual =T virtual *P cam (4)

[0027] At this point, the 2D feature pixel P is associated with the virtual coordinate system, and then, based on the principle of similar triangles, the virtual coordinates of the height of the i-th projected 3D point are obtained:

[0028]

[0029] Finally, the 2D features from the road test camera are converted to 3D space, using the following formula:

[0030]

[0031] Finally, VoxelPooling is used to generate BEV features of the points. At this point, the Image Height Projection Network (IHPN) is completed, and the image features are transformed from 2D to 3D BEV features.

[0032] As a preferred embodiment of the present invention, in step S3, for the feature extraction of lidar data, the input point cloud data is first cropped, retaining only four dimensions, namely X, Y, Z and density, and a scattering method is used to increase the density of a single frame point cloud scan.

[0033] Then, the point cloud is divided into ground point cloud and non-ground point cloud. Outlier removal is then performed on the ground point cloud to filter noise. After preprocessing, VoxelPooling features are extracted from the point cloud to generate BEV features of the point cloud.

[0034] As a preferred embodiment of the present invention, in step S4, the semantic features generated from the RGB image and the geometric features of the LiDAR point cloud are fused based on the extracted BEV features of the image and the BEV features of the point cloud. Then, the original features are upsampled twice and downsampled once, and feature maps of the same size are fused to obtain the final fused feature map.

[0035] As a preferred embodiment of the present invention, in step S5, based on the above feature map, after one upsampling and shared convolution, and using ScaleNMS to perform non-maximum suppression on the 3D bounding box, six non-shared task heads are finally decoded. Each task head contains six branches: branch one is the offset of the 2D dimension along the x and y axes, i.e., the reg branch; branch two is the z-axis, which is the height information of the predicted object, i.e., the height branch; branch three is the size information of the object, i.e., the dim branch, where the size information is length-width-height; branch four is the sine and cosine of the yaw angle of the object, i.e., the rot branch; branch five is the velocity of the object along the x and y axes, i.e., the vel branch; and branch six is ​​the classification confidence, i.e., the heatmap branch.

[0036] Compared with the prior art, the beneficial effects of the present invention are:

[0037] This invention provides a deep learning algorithm for roadside 3D target detection based on a combination of camera and LiDAR. First, a ResNet101 network is used to extract image features. Then, an FPN is used to perform multi-scale fusion of the extracted features to obtain the extracted image features. Based on the image features, the features are divided into a semantic information branch and a pixel height prediction branch. The semantic information branch uses a compression and activation network layer (SE-layer) to extract the semantic information of the region of interest in the image. The pixel height prediction branch uses four residual blocks and a deformable convolutional layer to predict the height of each pixel. Finally, the semantic information branch and the pixel height prediction branch are fused to obtain the final image features, which are then voxelized. Secondly, for feature extraction from LiDAR data, the input point cloud data is first cropped, then divided into ground point clouds and non-ground point clouds. Outlier removal is then performed on the ground point clouds to filter noise. After preprocessing, VoxelPooling features are extracted from the point clouds to generate BEV features. Finally, the voxelized image features and point cloud features are fused, followed by one upsampling and shared convolution. ScaleNMS is used to suppress non-maximum values ​​in the 3D bounding boxes. Finally, six non-shared task heads are decoded to obtain the final high-precision 3D object detection result. This results rank among the top on open-source datasets. Attached Figure Description

[0038] Figure 1 This invention relates to an Image Height Projection Network (IHPN).

[0039] Figure 2 This is a diagram of the overall network structure of the present invention. Detailed Implementation

[0040] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0041] Please see Figure 1-2This invention provides a deep learning algorithm for roadside 3D target detection based on a combination of camera and LiDAR. First, a ResNet101 network is used to extract image features. Then, an FPN is used to perform multi-scale fusion of the extracted features to obtain the extracted image features. Based on the image features, the features are divided into a semantic information branch and a pixel height prediction branch. The semantic information branch uses a compression and activation network layer (SE-layer) to extract the semantic information of the region of interest in the image. The pixel height prediction branch uses four residual blocks and a deformable convolutional layer to predict the height of each pixel. Finally, the semantic information branch and the pixel height prediction branch are fused to obtain the final image features, which are then voxelized. Secondly, for LiDAR data feature extraction, the input point cloud data is first cropped, then divided into ground point clouds and non-ground point clouds. Outlier removal is then performed on the ground point clouds to filter noise. After preprocessing, VoxelPooling features are extracted from the point clouds to generate BEV features. Finally, the voxelized image features and point cloud features are fused, followed by one upsampling and shared convolution. ScaleNMS is used to suppress non-maximum values ​​in the 3D bounding boxes. Finally, six non-shared task heads are decoded to obtain the final high-precision 3D target detection result. This invention mainly includes the following steps: data construction, data preprocessing, image feature extraction, LiDAR feature extraction, feature fusion, and task head decoding.

[0042] The technical solution adopted is described below:

[0043] Step S1: Data construction and data preprocessing:

[0044] The main open-source datasets for 3D object detection in road tests are DAIR-V2X-I Dataset, A9 InteractionDataset, and Repo3D Dataset. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above three datasets. A common practice is to convert the above data into the KITTI dataset format and then input it into the model. After converting the dataset, feature extraction is performed.

[0045] Step S2; Image feature extraction:

[0046] Image feature extraction is achieved by creating an Image Height Projector Network (IHPN). The input image is cropped to 704*256 pixels with a depth of 3. The cropped image is then fed into a ResNet101 network to extract image features. A Multi-Scale Fusion Network (FPN) is then used to fuse the extracted features at multiple scales to obtain the final image features. Based on these features, the network is divided into a semantic information branch and a pixel height prediction branch. For the semantic information branch, a compression and activation network layer (SE-layer) is used to extract the semantic information of the regions of interest (ROIs) in the image. The semantic information branch is represented as F... context The pixel height prediction branch uses four residual blocks and one deformable convolutional layer to predict the height of each pixel. The pixel height prediction branch is represented as PH. prediction To optimize computation speed, one-hot encoding is introduced, which encodes continuous heights PH. prediction Discretization is performed, and the specific formula is as follows:

[0047]

[0048] Where h represents the height from the ground to the camera, h max and h min Indicates the maximum and minimum height values, M represents the number of discrete height columns, and h i Let represent the i-th height column. ε is a hyperparameter that controls the density of height columns.

[0049] The semantic information branch and the pixel height prediction branch are then fused together using a dot product to obtain the final 2D image features. The fusion formula is as follows:

[0050] F fuse =F context ⊙PH prediction (2)

[0051] Next, the 2D features of the image are mapped to 3D features. For a pixel P(u,v) in the image plane,

[0052] First, create a virtual coordinate system with the origin of the camera coordinate system, the y-axis perpendicular to the ground, and a depth d of 1. Then, use the transformation formula from pixel coordinate system to camera coordinate system to transform the pixels from pixel space to camera space:

[0053] P cam =K -1 d[u,v,1] T =K -1 [u,v,1] T (3)

[0054] Where K represents the rigid transformation, and d is the depth with a value of 1. Let T be the transformation matrix from the camera coordinate system to the virtual coordinate system. virtual This allows you to map points from the camera coordinate system to the virtual coordinate system.

[0055] P cam-virtual =T virtual *P cam (4)

[0056] At this point, the 2D feature pixel P is associated with the virtual coordinate system, and then, based on the principle of similar triangles, the virtual coordinates of the height of the i-th projected 3D point are obtained:

[0057]

[0058] Finally, the 2D features from the road test camera are converted to 3D space, using the following formula:

[0059]

[0060] Finally, Voxel Pooling is used to generate BEV features for the points. At this point, the Image Height Projection Network (IHPN) is complete, and the image features are transformed from 2D to 3D BEV features. The network structure diagram is shown below. Figure 1 :

[0061] Step S3; LiDAR Feature Extraction:

[0062] For feature extraction from LiDAR data, the input point cloud data is first cropped, retaining only four dimensions (X, Y, Z, and density), and a scattering method is used to increase the density of a single frame of point cloud scanning. Then, the point cloud is divided into ground point cloud and non-ground point cloud, and outlier removal is performed on the ground point cloud to filter noise. After preprocessing, VoxelPooling features are extracted from the point cloud to generate BEV features of the point cloud.

[0063] Step S4; Feature fusion:

[0064] Based on the extracted BEV features of the image and the BEV features of the point cloud, the semantic features generated from the RGB image are fused with the geometric features of the LiDAR point cloud. The original features are then upsampled twice and downsampled once. The feature maps of the same size are fused to obtain the final fused feature map.

[0065] Step S5; Task Head Decoding;

[0066] After one upsampling and shared convolution, and using ScaleNMS to perform non-maximum suppression on the 3D bounding boxes, six non-shared task heads are finally decoded. Each task head contains six branches: branch 1 is the 2D dimension offset along the x and y axes (reg branch); branch 2 is the z-axis, i.e., the predicted height information of the object (height branch); branch 3 is the object's size information (length-width-height) (dim branch); branch 4 is the sine and cosine of the object's yaw angle (rot branch); branch 5 is the object's velocity along the x and y axes (vel branch); and branch 6 is the classification confidence (heatmap branch), yielding the final 3D object detection result. The overall network structure diagram is shown below. Figure 2 :

[0067] The implementation of the monocular camera depth estimation algorithm based on piecewise plane priors consists of the following parts:

[0068] Step 1; Data Preparation:

[0069] The main open-source datasets for 3D object detection in road tests are DAIR-V2X-I Dataset, A9 InteractionDataset, and Repo3D Dataset. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above three datasets. The common practice is to convert the above data into the KITTI dataset format and then input it into the model. After converting the dataset, feature extraction is performed.

[0070] Step 2; Training Phase:

[0071] The input image size is 704*256*3, and the number of image feature channels is [64,128,256,512]. Image features are extracted using ResNet101 and multi-layer features are fused using FPN. The input LiDAR point cloud data points are 16384, and the number of point cloud feature channels is [96,256,512,1024]. Based on the image features, the features are divided into a semantic information branch and a pixel height prediction branch. The semantic information branch uses a compression and activation network layer (SE-layer) to extract the semantic information of the region of interest in the image. The pixel height prediction branch uses four residual blocks and a deformable convolutional layer to predict the height of each pixel. Finally, the semantic information branch and the pixel height prediction branch are fused to obtain the final image features, and then the image features are voxelized. Secondly, for feature extraction from LiDAR data, the input point cloud data is first cropped, then the point cloud is divided into ground point cloud and non-ground point cloud. Outlier removal is then performed on the ground point cloud to filter noise. After preprocessing, VoxelPooling features are extracted from the point cloud to generate BEV features of the point cloud. Finally, the voxelized image features and point cloud features are fused. Gaussian FocalLoss is used to supervise the detection category, and L1Loss is used to supervise the detection box. After one upsampling and shared convolution, ScaleNMS is used to suppress non-maximum values ​​in the 3D boxes. Finally, six non-shared task heads are decoded to obtain the final high-precision 3D target detection result.

[0072] Step 3; Testing Phase:

[0073] For the test image, it is fed into the network for inference. After fusing image features and LiDAR features, six branches are decoded. Branch 1 is the offset along the x and y axes in 2D dimension. Branch 2 is the height information of the predicted object along the z axis. Branch 3 is the size information of the object (length-width-height). Branch 4 is the sine and cosine of the object's yaw angle. Branch 5 is the velocity of the object along the x and y axes. Branch 6 is the classification confidence. The 3D object detection prediction value obtained from this is compared with the ground truth value to calculate mAP, which gives the final accuracy.

[0074] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A deep learning algorithm for roadside 3D target detection based on a combination of camera and LiDAR, characterized in that, Includes the following steps: Step S1, Data Construction and Data Preprocessing: The open-source datasets for 3D object detection in road tests include DAIR-V2X-I Dataset, A9 Intersection Dataset and Repo3D Dataset. When training on your own dataset, prepare the data format to be consistent with the above three datasets, and convert the above data into KITTI dataset format for input into the model. After converting the dataset, perform feature extraction. Step S2, Image Feature Extraction: Image features are extracted by constructing an Image HeightProjector Network (IHPN). Using RGB images as input data, a ResNet101 network is used for feature extraction, and then an FPN is used to fuse the extracted features at multiple scales to obtain the extracted image features. Based on image features, the features are divided into a semantic information branch and a pixel height prediction branch. For the semantic information branch, a compression and activation network layer, namely the SE-layer, is used to extract the semantic information of the region of interest in the image. For the pixel height prediction branch, multiple residual blocks plus a deformable convolutional layer are used to predict the height of each pixel. Then, the semantic information branch and the pixel height prediction branch are fused by dot product to obtain the final 2D image features. Next, the 2D image features are mapped to 3D features. By creating a virtual coordinate system with a depth of 1, the 2D features are associated with the virtual coordinate system. Then, according to the principle of similar triangles, the 2D features of the road test camera are transformed into 3D space. Finally, Voxel Pooling is used to generate BEV features of points. At this point, the image pixel height projection network, namely IHPN, is completed, and the image features are transformed from 2D to 3D BEV features. Step S3, LiDAR feature extraction: For LiDAR data feature extraction, the input point cloud data is first cropped, retaining only four dimensions, namely X, Y, Z and density. The scattering method is used to increase the density of the single frame point cloud scan. Then, the point cloud is divided into ground point cloud and non-ground point cloud. Outlier removal is then performed on the ground point cloud to filter noise. After preprocessing, VoxelPooling features are extracted from the point cloud to generate BEV features of the point cloud. Step S4, Feature Fusion: Based on the extracted image BEV features and point cloud BEV features, the semantic features generated from the RGB image are fused with the geometric features of the LiDAR point cloud. The original features are then upsampled twice and downsampled once, and feature maps of the same size are fused to obtain the final fused feature map. Step S5, Task Head Decoding: Based on the above feature map, after one upsampling and shared convolution, and using ScaleNMS to perform non-maximum suppression on the 3D bounding boxes, six non-shared task heads are finally decoded. Each task head contains six branches: branch 1 is the offset along the x and y axes in the 2D dimension, i.e., the reg branch; branch 2 is the height information of the predicted object along the z axis, i.e., the height branch; branch 3 is the size information of the object, i.e., the dim branch, where the size information is length-width-height; branch 4 is the sine and cosine of the yaw angle of the object, i.e., the rot branch; branch 5 is the velocity of the object along the x and y axes, i.e., the vel branch; and branch 6 is the classification confidence, i.e., the heatmap branch.

2. The deep learning algorithm for roadside 3D target detection based on a combination of camera and lidar as described in claim 1, characterized in that, The camera image in S1 is 1920 pixels long and 1080 pixels wide. The data annotation information includes category labels for 10 types of targets and corresponding 2D annotation boxes. The point cloud data has dimensions N*3, where N is the number of point clouds and 3 is the number of channels.

3. The deep learning algorithm for roadside 3D target detection based on a combination of camera and lidar as described in claim 1, characterized in that, In step S2, the input image is cropped to 704*256 pixels with a depth of 3. The cropped image is then input into a ResNet101 network to extract image features. The extracted features are then fused using a multi-scale FPN to obtain the final image features. Based on these features, the features are divided into a semantic information branch and a pixel height prediction branch. For the semantic information branch, a compression and activation network layer (SE-layer) is used to extract the semantic information of the region of interest in the image. The semantic information branch is represented as... The pixel height prediction branch uses four residual blocks and one deformable convolutional layer to predict the height of each pixel. The pixel height prediction branch is represented as follows: To optimize computation speed, one-hot encoding is introduced to encode continuous heights. Discretization is performed, and the specific formula is as follows: (1) in, This represents the height value from the ground to the camera. and This represents the maximum and minimum height values, where M represents the number of discrete height columns. This represents the i-th columnar column. It is a hyperparameter that controls the density of the columnar structures. The semantic information branch and the pixel height prediction branch are then fused together using a dot product to obtain the final 2D image features. The fusion formula is as follows: (2) Next, the 2D features of the image are mapped to 3D features, for each pixel in the image plane. First, create the origin, which is the origin of the camera coordinate system, with the y-axis perpendicular to the ground, and the depth... Using a virtual coordinate system of 1, the transformation formula from pixel coordinate system to camera coordinate system is used to transform pixels from pixel space to camera space: = (3) in It is a rigid transformation. The depth is 1; let the transformation matrix from the camera coordinate system to the virtual coordinate system be... This allows you to map points from the camera coordinate system to the virtual coordinate system: (4) At this point, the 2D feature pixels are... By associating points with the virtual coordinate system, and then applying the principle of similar triangles, the first... Virtual coordinates of the height of each projected 3D point: (5) Finally, the 2D features from the road test camera are converted to 3D space, using the following formula: (6) Finally, VoxelPooling is used to generate BEV features of the points. At this point, the image height projection network is completed, and the image features are transformed from 2D to 3D BEV features.

4. The deep learning algorithm for roadside 3D target detection based on a combination of camera and lidar as described in claim 1, characterized in that, In S3, for the feature extraction of lidar data, the input point cloud data is first cropped, retaining only four dimensions, namely X, Y, Z and density, and a scattering method is used to increase the density of a single frame point cloud scan. Then, the point cloud is divided into ground point cloud and non-ground point cloud. Outlier removal is then performed on the ground point cloud to filter noise. After preprocessing, VoxelPooling features are extracted from the point cloud to generate BEV features of the point cloud.

5. The deep learning algorithm for roadside 3D target detection based on a combination of camera and lidar as described in claim 1, characterized in that, In step S4, based on the extracted image BEV features and point cloud BEV features, the semantic features generated from the RGB image are fused with the geometric features of the LiDAR point cloud. Then, the original features are upsampled twice and downsampled once, and feature maps of the same size are fused to obtain the final fused feature map.

Images