A real-time target detection method based on lidar and camera data fusion

Abstract

The present invention discloses a real-time target detection method based on the fusion of laser radar and camera data. The method is implemented using a convolutional neural network model, specifically including: constructing a sensor joint calibration matrix to map the 3D laser radar point cloud image to a 2D plane and convert it into a depth map; building a twin convolutional network model including two identical parallel branches to extract feature tensors from the depth map and color map respectively; on this basis, a data fusion method based on feature layer fusion is proposed, the extracted multi-scale feature tensors are subjected to multimodal data fusion through superposition operations, and the fused data is input into the detection head to realize target detection classification and position regression. The target detection method based on fused data proposed in the present invention improves the detection accuracy while ensuring good real-time performance, and its comprehensive performance is better than that of existing target detection methods.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and intelligent information processing, and specifically relates to a real-time target detection method based on laser radar and camera data fusion. Background Art

[0002] Object detection is a crucial component of environmental perception systems for intelligent robots, finding widespread application in autonomous driving, household robots, drones, and automated guided vehicles (AGVs). However, due to the complexities and vastness of real-world environments, most object detection algorithms suffer from low detection efficiency, making it difficult to meet real-time and high-accuracy recognition requirements. Therefore, developing high-precision and real-time object detection algorithms is an urgent challenge.

[0003] Currently, common sensors used in target detection modules include RGB cameras and LiDAR. Cameras offer fast detection speeds and can capture rich texture information about the target being detected, but they struggle to directly measure the object's shape and position. Furthermore, as passive sensors, they are easily affected by changes in ambient light intensity. Compared to RGB cameras, LiDAR uses lasers to detect the surrounding environment, accurately measuring the distance and shape of objects and demonstrating strong robustness to light variations. However, even high-resolution LiDARs collect relatively sparse point cloud data. Therefore, fusing RGB images with point cloud images yields data that is rich in semantic information, accurate in distance information, and insensitive to light variations. This provides reliable source data for detection algorithms, thereby improving detection accuracy.

[0004] Multimodal data fusion strategies include data-level fusion, feature-level fusion, and decision-level fusion. Data-level fusion converts RGB images and depth images into multidimensional tensor graphs, which are then spliced ​​and fused in the tensor depth dimension. However, this method produces large amounts of fused tensor data, and the convolution processing is time-consuming, making it difficult to meet real-time requirements. As a shallow fusion method, data-level fusion only achieves simple splicing and fails to establish connections between multimodal data, which can reduce fusion performance. Decision-level fusion uses two independent convolutional neural networks to process RGB and depth images to obtain detection results. The final decision depends on the integration of these two results, but the two networks' detection results may be mutually exclusive, which can lead to poor detection performance. In contrast, feature-level fusion extracts abstract feature maps from source data collected by multiple sensors and fuses them into a single feature vector, which is then detected and recognized through convolution. Abstract feature maps are smaller than unprocessed raw images, so processing is time-consuming. Furthermore, feature-level fusion allows for multiple fusions between the two convolutional neural network structures, strengthening the correlation between multimodal data. Summary of the Invention

[0005] To address the issues mentioned in the technical background, this paper proposes a real-time target detection method based on the fusion of LiDAR and camera data. This method uses a feature-level fusion approach to effectively reduce the amount of data computation and improve the real-time performance of the algorithm. Furthermore, the fusion strengthens the correlation of multimodal data, increasing the richness of the information contained and the accuracy of detection. Specifically, it includes the following steps:

[0006] 1) Prepare the color camera image data and corresponding lidar point cloud data to be processed;

[0007] 2) The original point cloud data in step 1) is calibrated by multi-sensor joint calibration, and each data point in the point cloud is projected onto the RGB image plane to convert it into a depth image, so that the RGB image and the depth image have the same field of view;

[0008] 3) Build an object detection model based on a twin neural network. The model consists of two branches, one for convolution processing of RGB images and the other for depth images.

[0009] 4) Construct a cross-fusion module based on feature layer fusion. The two parallel branches of the Siamese network model are composed of multiple convolutional layers. The model extracts feature maps of different sizes through multi-layer convolution. A fusion layer is added after the multiple convolutional layers of the parallel branches to complete multimodal data fusion by superimposing the feature maps of non-homogeneous data;

[0010] 5) Input the RGB image and the corresponding depth image into the neural network for training, and perform detection after the model converges to obtain the final detection result.

[0011] Furthermore, the point cloud to depth map method in step 2) first rotates and translates the point cloud data from the lidar coordinate system to the camera coordinate system according to formula 1; then, the transposed data is projected from the camera coordinate system to the image coordinate system according to formula 2; and finally, the projected data is scaled from the image coordinate system to the pixel coordinate system according to formula 3.

[0012]

[0013] where [X C ,Y C ,Z C ] is the coordinate in the laser radar coordinate system, [X L ,Y L ,Z L ] is the coordinate in the camera coordinate system, R is the rotation matrix, and T is the translation matrix.

[0014]

[0015] Where f is the focal length of the camera, and [x,y] is the coordinate in the image coordinate system.

[0016]

[0017] Among them, [u,v] is the coordinate in the pixel coordinate system, and [u0,v0] is the origin of the pixel coordinate system.

[0018] Furthermore, the twin neural network target detection model in step 3) is composed of two identical parallel branches, wherein branch 1 is used to convolutionally process RGB images, and branch 2 is used to convolutionally process depth images, and the sizes of both input images are (W, H).

[0019] Furthermore, the two branches of the twin neural network model, the feature extraction network of each branch is Darknet53, including 6 convolutional layers and 5 groups of residual modules, which are as follows: the first convolutional layer, the convolution kernel size is 3×3, the step size is 1, the number of channels is 32, and the output size is The second convolution layer has a kernel size of 3×3, a stride of 2, 64 channels, and an output size of The first group of residual modules consists of 1 residual module, and the output size is The third convolution layer has a kernel size of 3×3, a stride of 2, 128 channels, and an output size of The second group of residual modules consists of 2 residual modules, and the output size is The fourth convolution layer has a kernel size of 3×3, a stride of 2, a channel number of 256, and an output size of The third group of residual modules consists of 8 residual modules with an output size of The fifth convolution layer has a kernel size of 3×3, a stride of 2, a channel number of 512, and an output size of The fourth group of residual modules consists of 8 residual modules, with 512 channels and an output size of The sixth convolution layer has a kernel size of 3×3, a stride of 2, a channel number of 1024, and an output size of The fifth group of residual modules consists of 4 residual modules, and the output size is

[0020] Furthermore, among the five groups of residual modules, each residual module in the first group of residual modules consists of two convolutional layers and one residual unit. The convolution kernel size of the first convolutional layer is 1×1, the stride is 1, and the number of channels is 32; the convolution kernel size of the second convolutional layer is 3×3, the stride is 1, and the number of channels is 64. Each residual module in the second group of residual modules consists of two convolutional layers and one residual unit. The convolution kernel size of the first convolutional layer is 1×1, the stride is 1, and the number of channels is 64; the convolution kernel size of the second convolutional layer is 3×3, the stride is 1, and the number of channels is 128. Each residual module in the third group of residual modules consists of two convolutional layers and one residual unit. The first convolutional layer has a convolution kernel size of 1×1, a stride of 1, and 128 channels; the second convolutional layer has a convolution kernel size of 3×3, a stride of 1, and 256 channels. Each residual module in the fourth group of residual modules consists of two convolutional layers and one residual unit. The first convolutional layer has a convolution kernel size of 1×1, a stride of 1, and 256 channels; the second convolutional layer has a convolution kernel size of 3×3, a stride of 1, and 512 channels. Each residual module in the fifth group of residual modules consists of two convolutional layers and one residual unit. The first convolutional layer has a convolution kernel size of 1×1, a stride of 1, and 512 channels; the second convolutional layer has a convolution kernel size of 3×3, a stride of 1, and 1024 channels.

[0021] Furthermore, the cross fusion module based on feature layer fusion is composed of three addition layers. The addition layer adds the eigenvalues ​​in the multi-dimensional feature vector map obtained by the convolution of the two branches, and fuses the feature map information of the depth image and the RGB image without changing the dimension of the feature map. Multiple fusions can deepen the degree of fusion and fully fuse the two non-homogeneous data. Among them, the first addition layer is placed after the third group of residual modules of the two branches, and the size of the two branches is Feature Map C 13 and C 23 The size is generated by adding Feature map C 1, ; The second addition layer is placed after the fourth group of residual modules of the two branches, and the size of the two branches is Feature map C 14 and C 24 The size is generated by adding Feature map C 1, ; The third addition layer is placed after the fifth group of residual modules of the two branches, and the size of the two branches is Feature map C 15 and C 25 The size is generated by adding Feature map C 1, .

[0022] Furthermore, the object detection algorithm head uses the YOLO series' one-stage architecture to simultaneously predict the category and location of the detected object. The number of output channels is 3 (K + 5), where 3 represents three prediction boxes of different sizes for each predicted location, K represents the number of predicted categories, and 5 represents the four coordinates of the predicted box and one confidence level.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] 1. The present invention preprocesses the lidar point cloud data and maps the point cloud to a 2D plane to construct a depth map by constructing a joint calibration matrix, thereby reducing the amount of data computation while retaining the depth information.

[0025] 2. The twin neural network model constructed by the present invention consists of two identical parallel branches, which are used to convolutionally process RGB images and depth images respectively, and can effectively capture more common abstract features of objects to be detected in multimodal data.

[0026] 3. The cross-fusion module constructed by this invention, based on feature layer fusion, feeds the multi-scale feature maps generated by the convolution of two neural network branches into the addition layer for multimodal data fusion. During training, the network model adjusts weights based on the fused data, strengthening the correlation of heterogeneous data. This fusion method reduces the size of feature maps and the amount of data computation, while improving real-time processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is an overall flow chart of an embodiment of the present invention.

[0028] Figure 2 This is the twin neural network model of an embodiment of the present invention.

[0029] Figure 3 2 is a diagram showing the detection results of an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be described in detail below with reference to the accompanying drawings in the embodiments of the present invention:

[0031] Step 1: Prepare the color camera image data and corresponding lidar point cloud data to be processed. The 7,481 RGB images and point clouds officially released by KITTI, along with their corresponding labels, were divided into training, validation, and test sets in an 8:1:1 ratio. The data labels were then processed and converted to the standard VOC format.

[0032] Step 2: Convert the raw point cloud data into a depth image. A joint calibration matrix is ​​constructed based on the intrinsic and extrinsic parameters of the LiDAR and camera. First, the raw point cloud data is converted from the LiDAR coordinate system to the camera coordinate system through spatial rotation and translation according to Formula 1. Then, the rotated and translated data is transferred from the camera coordinate system to the image coordinate system through transmission projection according to Formula 2. Finally, the projected data is transferred from the image coordinate system to the pixel coordinate system through scaling and translation according to Formula 2 to obtain the depth image.

[0033]

[0034] where [X C ,Y C ,Z C ] is the coordinate in the laser radar coordinate system, [X L ,Y L ,Z L ] is the coordinate in the camera coordinate system, R is the rotation matrix, and T is the translation matrix.

[0035]

[0036] Where f is the focal length of the camera, and [x,y] is the coordinate in the image coordinate system.

[0037]

[0038] Among them, [u,v] is the coordinate in the pixel coordinate system, and [u0,v0] is the origin of the pixel coordinate system.

[0039] Step 3: Build an object detection model based on a twin neural network. The model consists of two identical parallel branches, which convolute the RGB image and the depth image respectively. The feature extraction network of each branch is Darknet53, which includes 6 convolutional layers and 5 groups of residual modules, as follows: The first convolutional layer has a convolution kernel size of 3×3, a stride of 1, a channel number of 32, and an output size of The second convolution layer has a kernel size of 3×3, a stride of 2, 64 channels, and an output size of The first group of residual modules consists of 1 residual module, and the output size is The third convolution layer has a kernel size of 3×3, a stride of 2, 128 channels, and an output size of The second group of residual modules consists of 2 residual modules, and the output size is The fourth convolution layer has a kernel size of 3×3, a stride of 2, a channel number of 256, and an output size of The third group of residual modules consists of 8 residual modules with an output size of The fifth convolution layer has a kernel size of 3×3, a stride of 2, a channel number of 512, and an output size of The fourth group of residual modules consists of 8 residual modules, with 512 channels and an output size of The sixth convolution layer has a kernel size of 3×3, a stride of 2, a channel number of 1024, and an output size of The fifth group of residual modules consists of 4 residual modules, and the output size is

[0040] Each residual module in the first group of residual modules consists of two convolutional layers and one residual unit. The convolution kernel size of the first convolutional layer is 1×1, the stride is 1, and the number of channels is 32; the convolution kernel size of the second convolutional layer is 3×3, the stride is 1, and the number of channels is 64. Each residual module in the second group of residual modules consists of two convolutional layers and one residual unit. The convolution kernel size of the first convolutional layer is 1×1, the stride is 1, and the number of channels is 64; the convolution kernel size of the second convolutional layer is 3×3, the stride is 1, and the number of channels is 128. Each residual module in the third group of residual modules consists of two convolutional layers and one residual unit. The convolution kernel size of the first convolutional layer is 1×1, the stride is 1, and the number of channels is 128; the convolution kernel size of the second convolutional layer is 3×3, the stride is 1, and the number of channels is 256. Each residual module in the fourth group of residual modules consists of two convolutional layers and one residual unit. The convolution kernel size of the first convolutional layer is 1×1, the stride is 1, and the number of channels is 256; the convolution kernel size of the second convolutional layer is 3×3, the stride is 1, and the number of channels is 512. Each residual module in the fifth group of residual modules consists of two convolutional layers and one residual unit. The convolution kernel size of the first convolutional layer is 1×1, the stride is 1, and the number of channels is 512; the convolution kernel size of the second convolutional layer is 3×3, the stride is 1, and the number of channels is 1024.

[0041] Step 4: Construct a cross-fusion module based on feature layer fusion. The fusion method is to build multi-layer connections between the convolutional layers of the neural network and fuse the feature maps of multimodal data. By fusing feature maps with less data instead of simply superimposing the original data, the amount of data calculation can be effectively reduced and the real-time performance of the algorithm can be improved. Multi-layer connections enable the network to update the network weights according to non-homogeneous data, thereby strengthening the correlation of the data. The cross-fusion module consists of three addition layers: the first addition layer is placed after the third group of residual modules of the two branches, and the size of the two branches is Feature map C 13 and C 23 The size is generated by adding Feature map C 1, ; The second addition layer is placed after the fourth group of residual modules of the two branches, and the size of the two branches is Feature map C 14and C 24 The size is generated by adding Feature map C 1, ; The third addition layer is placed after the fifth group of residual modules of the two branches, and the size of the two branches is Feature map C 15 and C 25 The size is generated by adding Feature map C 1, .

[0042] Step 5: Build a one-stage detection head based on the YOLO series, and output the category and location of the target to be detected. The head structure has 3 output channels (K+5), where 3 represents three prediction boxes of different sizes for each predicted location, K represents the number of predicted categories, and 5 represents the four coordinates and one confidence level of the prediction box. The RGB images and corresponding depth images in the training set are input into the constructed network model for training and validation. After the model converges, it is tested on the test set and compared with other algorithms, including the following:

[0043] F-PointNet, please refer to Qi CR, Liu W, Wu C, et al. Frustum pointnets for 3dobject detection from rgb-d data[C]. IEEE conference on computer vision and pattern recognition, 2018:918-927.

[0044] 3D-CVF atSPA, please refer to Yoo JH, Kim Y, Kim J, Choi J W.3d-cvf: Generatingjoint camera and lidar features using cross-view spatial feature fusion for3d object detection[C].In European Conference on Computer Vision, 2020:720-736.

[0045] CLCOs. For reference, see Pang S, Morris D, Radha H. CLOCs: Camera-LiDAR object candidates fusion for 3D object detection[C]. IEEE International Conference on Intelligent Robots and Systems, 2020: 10386-10393.

[0046] ImvoxelNet. For reference, see Rukhovich D, Vorontsova A, Konushin A. Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection[C]. Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, 2020: 2397-2406.

[0047] Pointpillars. For reference, see Lang A H, Vora S, Caesar H, Zhou L, Yang J, Beijbom O. Pointpillars: Fast encoders for object detection from point clouds[C]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019:12697-12705.

[0048] PointRCNN. For reference, see Shi S, Wang X, Li H. Pointrcnn: 3d object proposal generation and detection from point cloud[C]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019: 770-779.

[0049] PointRCNN, please refer to Yang Z, Sun Y, Liu S, Jia J.3dssd: Point-based 3d singlestage object detector[C]. Proceedings of the IEEE / CVF conference on computervision and pattern recognition, 2020:11040-11048.

[0050] The final test results are shown in Table 1.

[0051] Table 1 Comparative experimental results of the method of the present invention with other methods on the KITTI test set

[0052]

[0053]

[0054] Note: L stands for laser radar, R stands for color camera

[0055] Table 1 presents comparative experimental results of the proposed method against other methods on the KITTI test set. Compared with object detection methods based on point cloud image fusion and single point cloud, the proposed method achieves superior detection performance, achieving higher detection accuracy than other methods on the three official KITTI baselines: easy, medium, and difficult. Furthermore, the proposed method runs in a mere 0.05 seconds, significantly faster than other algorithms, demonstrating that it achieves both improved detection performance and real-time performance.

[0056] The above description is merely an embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A real-time target detection method based on laser radar and camera data fusion, characterized in that: The following steps are involved: 1) Prepare the color camera image data and corresponding lidar point cloud data to be processed; 2) The original point cloud data in step 1) is calibrated by multi-sensor joint calibration, and each data point in the point cloud is projected onto the RGB image plane to convert it into a depth image, so that the RGB image and the depth image have the same field of view; 3) Build an object detection model based on a twin neural network. The model consists of two branches, one for convolution processing of RGB images and the other for depth images. 4) Construct a cross-fusion module based on feature layer fusion. The two parallel branches of the Siamese network model are composed of multiple convolutional layers. The model extracts feature maps of different sizes through multi-layer convolution. A fusion layer is added after the multiple convolutional layers of the parallel branches to complete multimodal data fusion by superimposing the feature maps of non-homogeneous data; 5) Input the RGB image and the corresponding depth image into the neural network for training, and perform detection after the model converges to obtain the final detection result; The cross fusion module based on feature layer fusion in step 4) consists of three addition layers; the addition layer adds the eigenvalues ​​in the multidimensional feature vector map obtained by the convolution of the two branches, and fuses the feature map information of the depth image and the RGB image without changing the dimension of the feature map. Multiple fusions can deepen the degree of fusion and fully fuse the two non-homogeneous data; among them, the first addition layer is placed after the third group of residual modules of the two branches, and the size of the two branches is Feature map C 13 and C 23 The size is generated by adding Feature map C 1,23 ; The second addition layer is placed after the fourth group of residual modules of the two branches, and the size of the two branches is Feature map C 14 and C 24 The size is generated by adding Feature map C 1,24 ; The third addition layer is placed after the fifth group of residual modules of the two branches, and the size of the two branches is Feature map C 15 and C 25 The size is generated by adding Feature map C 1,25 .

2. The real-time target detection method based on laser radar and camera data fusion according to claim 1, characterized in that: In step 2), the point cloud data is converted into a depth map through multi-sensor joint calibration. First, the point cloud data is rotated and translated from the lidar coordinate system to the camera coordinate system according to formula 1. Then, the transposed data is projected from the camera coordinate system to the image coordinate system according to formula 2. Finally, the projected data is scaled from the image coordinate system to the pixel coordinate system according to formula 3. where [X C ,Y C ,Z C ] is the coordinate in the laser radar coordinate system, [X L ,Y L ,Z L ] is the coordinate in the camera coordinate system, R is the rotation matrix, and T is the translation matrix; Where f is the focal length of the camera, and [x,y] is the coordinate in the image coordinate system; Among them, [u,v] is the coordinate in the pixel coordinate system, and [u0,v0] is the origin of the pixel coordinate system.

3. The real-time target detection method based on laser radar and camera data fusion according to claim 1, characterized in that: The twin neural network target detection model in step 3) consists of two identical parallel branches, where branch 1 is used to convolve RGB images and branch 2 is used to convolve depth images. The sizes of both input images are (W, H).

4. The real-time target detection method based on laser radar and camera data fusion according to claim 3, characterized in that: The feature extraction network of each branch is Darknet53, which includes 6 convolutional layers and 5 groups of residual modules, which are as follows: the first convolutional layer, the convolution kernel size is 3×3, the step size is 1, the number of channels is 32, and the output size is The second convolution layer has a kernel size of 3×3, a stride of 2, 64 channels, and an output size of The first group of residual modules consists of 1 residual module, and the output size is The third convolution layer has a kernel size of 3×3, a stride of 2, 128 channels, and an output size of The second group of residual modules consists of 2 residual modules, and the output size is The fourth convolution layer has a kernel size of 3×3, a stride of 2, a channel number of 256, and an output size of The third group of residual modules consists of 8 residual modules with an output size of The fifth convolution layer has a kernel size of 3×3, a stride of 2, a channel number of 512, and an output size of The fourth group of residual modules consists of 8 residual modules, with 512 channels and an output size of The sixth convolution layer has a kernel size of 3×3, a stride of 2, a channel number of 1024, and an output size of The fifth group of residual modules consists of 4 residual modules, and the output size is 5. The real-time target detection method based on laser radar and camera data fusion according to claim 4, characterized in that: Each residual module in the first group of residual modules consists of 2 convolutional layers and 1 residual unit. The convolution kernel size of the first convolutional layer is 1×1, the step size is 1, and the number of channels is 32; the convolution kernel size of the second convolutional layer is 3×3, the step size is 1, and the number of channels is 64; each residual module in the second group of residual modules consists of 2 convolutional layers and 1 residual unit. The convolution kernel size of the first convolutional layer is 1×1, the step size is 1, and the number of channels is 64; the convolution kernel size of the second convolutional layer is 3×3, the step size is 1, and the number of channels is 128; each residual module in the third group of residual modules consists of 2 convolutional layers and 1 residual unit. The convolution kernel size of the first convolutional layer is 1×1 , the stride is 1, and the number of channels is 128; the convolution kernel size of the second convolution layer is 3×3, the stride is 1, and the number of channels is 256; each residual module in the fourth group of residual modules consists of 2 convolution layers and 1 residual unit, the convolution kernel size of the first convolution layer is 1×1, the stride is 1, and the number of channels is 256; the convolution kernel size of the second convolution layer is 3×3, the stride is 1, and the number of channels is 512; each residual module in the fifth group of residual modules consists of 2 convolution layers and 1 residual unit, the convolution kernel size of the first convolution layer is 1×1, the stride is 1, and the number of channels is 512; the convolution kernel size of the second convolution layer is 3×3, the stride is 1, and the number of channels is 1024.

6. The real-time target detection method based on laser radar and camera data fusion according to claim 1, characterized in that: In step 5), the one-stage head structure of the YOLO series is used to simultaneously predict the category and position of the target to be detected; the number of output channels is 3 (K + 5), where 3 means that each predicted position has three prediction boxes of different sizes, K is the number of predicted categories, and 5 represents the four coordinates of the prediction box and one confidence level.

Images