A multi-sensor fusion all-day three-dimensional target detection method

Abstract

A multi-sensor fusion all-day three-dimensional target detection method belongs to the field of target detection technology in computer vision. First, the visible light camera, infrared camera and laser radar are temporally and spatially aligned; then, target detection is performed on the visible light image and infrared image using the trained target detection deep network model, and a two-dimensional target list in the visible light image and infrared image is obtained, including the target category and two-dimensional bounding box; again, the corresponding three-dimensional point cloud cone is extracted for each two-dimensional target bounding box detected in the visible light image and infrared image, and three-dimensional point cloud target detection is performed in the cone; finally, non-maximum suppression processing is performed on the three-dimensional targets detected in all cones to eliminate repeated detection of the same target, and the final three-dimensional target detection result is output. The present invention integrates visible light cameras, infrared cameras and laser radar sensors, and can stably detect three-dimensional targets in all-day environments during the day and at night.

Description

Technical Field

[0001] The present invention relates to target detection technology in computer vision, and in particular to a three-dimensional target tracking method based on multi-sensor fusion. Background Art

[0002] With the rapid development of deep learning, object detection technology based on images and 3D point cloud data has made significant progress. However, single-sensor object detection methods often have limitations in specific scenarios due to the inherent characteristics of the sensor. Visible light cameras can provide dense information about the target's color, texture, and edges under good lighting conditions. However, image quality degrades dramatically under low lighting conditions at night, making object detection based on visible light images often ineffective. Infrared cameras can capture thermal radiation images of scenes at night and are particularly suitable for detecting targets such as pedestrians and moving vehicles. However, infrared images typically have low resolution, making it difficult to detect targets at long distances or with small temperature differences from the surrounding environment. Image-based object detection has rapidly developed in recent years, resulting in a series of mature object detection models. These models can achieve excellent detection results quickly when image quality is good, but their performance degrades significantly when image quality is poor due to environmental and sensor performance. LiDAR is an active sensor that is not easily affected by ambient light and temperature. However, LiDAR point clouds are sparse, disordered, and unstructured. Traditional two-dimensional convolutional neural networks cannot be directly applied to three-dimensional point clouds. Currently, the commonly used approaches include multi-view projection, three-dimensional voxel partitioning, and direct point cloud processing. However, multi-view projection loses a large amount of three-dimensional information; the three-dimensional voxel partitioning method cannot process large-scale scenes at high resolution because its spatiotemporal complexity increases cubically with the number of grid cells in each dimension; current direct point cloud processing methods typically require repeated neighborhood searches and set abstractions, which also significantly increase the computational complexity as the number of point clouds increases.

[0003] To achieve efficient 3D object detection, the paper (Qi, CR, Liu, W., Wu, C., Su, H., & Guibas, LJ (2017). Frustum PointNets for 3D Object Detection from RGB-D Data.) proposes a 3D object detection method that integrates cameras and lidar. This method first uses an image-based object detection network to obtain a 2D detection bounding box on the image. Then, by registering the point cloud with the image, the frustum point cloud corresponding to the 2D detection bounding box is extracted. The frustum point cloud is then used with a PointNet network to extract foreground segmentation and predict 3D object bounding boxes. This method achieves efficient 3D object detection because it first utilizes a mature image detection network for screening, while 3D detection only requires computation on a smaller number of frustum point clouds. The literature (Wang, Z., & Jia, K. (2020). Frustum ConvNet: Sliding Frustums to Aggregate Local Point-WiseFeatures for Amodal 3D Object Detection (pp. 1742–1749).) improves the above method by dividing the frustum point cloud along the frustum axis, extracting features from each frustum using PointNet, and then using a convolutional network for object classification and bounding box regression. Although this Frustum-based visible light camera-lidar fusion target detection method improves detection efficiency, it is highly dependent on the detection results of visible light images. Once the visible light image is missed, it is bound to lead to missed detection of three-dimensional targets. Under complex lighting conditions, the degradation of image quality often leads to poor image target detection results. In order to alleviate this problem, the present invention realizes all-day three-dimensional target detection by fusing three sensors: visible light camera, infrared camera and lidar. Summary of the Invention

[0004] The purpose of the present invention is to address the problems existing in existing target detection technologies and provide a new three-dimensional target detection method based on multi-sensor fusion. This method integrates three sensors: visible light camera, infrared camera and lidar, and can stably detect three-dimensional targets in all-day environments such as day and night.

[0005] In order to achieve the above object, the technical solution provided by the present invention is:

[0006] A multi-sensor fusion all-day three-dimensional target detection method, characterized by comprising the following steps:

[0007] S1. Perform spatiotemporal registration of visible light camera, infrared camera and lidar;

[0008] S2. Perform target detection on the visible light image and infrared image using the trained deep network model to obtain a list of two-dimensional targets in the visible light image and infrared image, including the target category and two-dimensional bounding box;

[0009] S3. extract the corresponding 3D point cloud frustum for each 2D target bounding box detected in the visible light image and the infrared image, and perform 3D point cloud target detection in the frustum;

[0010] S31. Enlarging the two-dimensional bounding box of the target in the visible light image or infrared image;

[0011] S32. Project the point cloud to the visible light camera coordinate system or the infrared camera coordinate system, and filter the point cloud P that falls within the viewing cone corresponding to the magnified two-dimensional box F or P F ';

[0012] S33.In P F or P F ′ uses the deep learning model algorithm of trained point cloud data to perform three-dimensional target detection, and finally obtains the target category, three-dimensional bounding box and confidence.

[0013] S4. Perform non-maximum suppression (NMS) on the three-dimensional targets detected in all frustums to eliminate repeated detection of the same target, and output the final three-dimensional target detection result.

[0014] Furthermore, step 1 is as follows:

[0015] S11. Time synchronization of the LiDAR, visible light camera, and infrared camera is achieved through hard triggering or soft synchronization. The hard trigger triggers the camera exposure when the LiDAR rotates close to the field of view of the visible light camera or infrared camera. The soft synchronization records the timestamps of the LiDAR point cloud and image data and synchronizes them by finding the data frame with the closest timestamp.

[0016] S12. Calibrate the internal parameters of the visible light camera and the infrared camera to obtain the internal parameter matrix K of the visible light camera and the infrared camera V and K I ;

[0017] S13. Cross-calibrate the visible light camera and infrared camera with the lidar in turn to obtain the coordinate transformation matrix between the lidar coordinate system {L} and the visible light camera coordinate system {V} And the coordinate transformation matrix between the infrared camera coordinate system {I} The point cloud in the LiDAR coordinate system is projected onto the visible light image and infrared image respectively through the coordinate transformation matrix and the camera intrinsic parameter matrix to achieve registration between the point cloud and the two camera images.

[0018] Furthermore, the implementation process of step S13 is as follows:

[0019] S131. Pre-place several auxiliary calibration objects randomly so that the corner points of the calibration objects can be clearly identified in the visible light image, infrared image and 3D point cloud;

[0020] S132. Using the visible light camera, infrared camera and lidar to be calibrated, collect visible light images, infrared images and point cloud data of the calibration object;

[0021] S133. Select the pixels of the corner points of the auxiliary calibration object in the collected visible light image and infrared image to obtain the image coordinates of the corner points; then select the laser points corresponding to the corner points in the collected point cloud data to obtain their three-dimensional coordinates, and solve the coordinate transformation matrix for the visible light image and infrared image respectively according to the selected image coordinates and the corresponding three-dimensional laser point coordinates combined with the intrinsic parameter matrix of the camera. and

[0022] Furthermore, the auxiliary calibration object in step S131 is a heated flat plate.

[0023] Furthermore, step S2 is implemented as follows:

[0024] S21. Preprocessing the current visible light image or infrared image, normalizing it to a preset size, etc.;

[0025] S22. Input the preprocessed image into a pre-trained object detection network for feature extraction and bounding box prediction;

[0026] S23. Decode and perform non-maximum suppression post-processing on the network prediction results to obtain the target bounding box position and category information list corresponding to the original image.

[0027] Furthermore, the deep learning model algorithm of S33's point cloud data can adopt F-PointNet or F-ConvNet.

[0028] Compared with the prior art, the advantages of the present invention are:

[0029] 1) The present invention adopts a framework based on frustum 3D target detection, making full use of the existing mature and efficient image target detection technology. It only needs to perform 3D target detection in the frustum point cloud corresponding to the image detection result, which can not only obtain the accurate 3D position of the target, but also maintain a high overall computational efficiency.

[0030] 2) The present invention integrates visible light images and infrared images in the generation of two-dimensional targets, so that better two-dimensional detection results can be obtained both during the day and at night, and the number of missed three-dimensional targets caused by missed detection of single-modality images in complex environments such as poor lighting is greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of the implementation process of the all-day three-dimensional target detection method based on multi-sensor fusion.

[0032] Figure 2 It is a schematic diagram of the online processing flow of multi-sensor fusion 3D target detection. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and specific preferred examples, but the scope of protection of the present invention is not limited thereby.

[0034] A multi-sensor fusion all-day three-dimensional target detection method, comprising the following steps:

[0035] S1. Perform spatiotemporal registration of visible light camera, infrared camera and lidar;

[0036] S2. Perform target detection on the visible light image and infrared image using the trained deep network model to obtain a list of two-dimensional targets in the visible light image and infrared image, including the target category and two-dimensional bounding box;

[0037] S3. extract the corresponding 3D point cloud frustum for each 2D target bounding box detected in the visible light image and the infrared image, and perform 3D point cloud target detection in the frustum;

[0038] S4. Perform non-maximum suppression (NMS) on the 3D objects detected in all frustums to eliminate repeated detections of the same object and output the final 3D object detection results.

[0039] The specific steps for performing spatiotemporal registration of the visible light camera, infrared camera, and lidar in step S1 are as follows:

[0040] S11. Time synchronization of the LiDAR, visible light camera, and infrared camera is achieved through hard triggering or soft synchronization. The hard trigger triggers the camera exposure when the LiDAR rotates close to the camera's field of view. The soft synchronization records the timestamps of the LiDAR point cloud and image data and synchronizes them by finding the data frame with the closest timestamp.

[0041] S12. Use Zhang Zhengyou calibration method or other mature camera calibration methods to calibrate the intrinsic parameters of the visible light camera and infrared camera to obtain the intrinsic parameter matrix K of the visible light camera and infrared camera V and K I .

[0042] S13. Cross-calibrate the visible light camera and infrared camera with the lidar in turn to obtain the coordinate transformation matrix between the lidar coordinate system {L} and the visible light camera coordinate system {V} And the coordinate transformation matrix between the infrared camera coordinate system {I} This allows the point cloud in the lidar coordinate system to be projected onto the optical image and infrared image respectively according to the coordinate transformation matrix and the camera intrinsic parameter matrix, thereby realizing the registration of the point cloud and the two images.

[0043] The specific steps of step S13 are:

[0044] S131. Pre-place several auxiliary calibration objects (such as heated flat plates) randomly so that the corner points of the calibration objects can be clearly identified in the visible light image, infrared image, and 3D point cloud;

[0045] S132. Use the visible light camera, infrared camera, and lidar to be calibrated to collect visible light images, infrared images, and point cloud data of the calibration object;

[0046] S133. Select the pixels of the corner points of the auxiliary calibration object in the collected visible light image and infrared image to obtain the image coordinates of the corner points; then select the laser points corresponding to the corner points in the collected point cloud data to obtain their three-dimensional coordinates, and solve the coordinate transformation matrix for the visible light image and infrared image respectively according to the selected image coordinates and the corresponding three-dimensional laser point coordinates combined with the intrinsic parameter matrix of the camera. and

[0047] In step S2, target detection is performed on the visible light image and infrared image using the trained image target detection deep network model to obtain a two-dimensional target list in the visible light image and infrared image. Taking the visible light image as an example, the infrared image is similarly processed. The specific steps are:

[0048] S21. Perform necessary preprocessing on the current visible light image, such as normalizing it to a preset size;

[0049] S22. Input the preprocessed image into a pre-trained object detection network for feature extraction and bounding box prediction;

[0050] S23. Perform post-processing such as decoding and non-maximum suppression on the network prediction results to obtain a list of target bounding box positions and category information corresponding to the original image.

[0051] In step S3, the corresponding 3D point cloud frustum is extracted for each 2D target bounding box detected in the visible light image and the infrared image, and 3D point cloud target detection is performed in the frustum. Taking a 2D target detected in the visible light image as an example, the specific steps are as follows. The infrared image is similarly performed:

[0052] S31. Enlarging the two-dimensional bounding box of the target in the visible light image;

[0053] S32. Project the point cloud to the visible light camera coordinate system and filter the point cloud P that falls within the viewing cone corresponding to the magnified two-dimensional box F ;

[0054] S33.In P F In the paper, the trained F-PointNet (Qi, CR et al. Frustum PointNets for 3D Object Detection from RGB-D Data. [C] / / Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018: 918–927) or F-ConvNet (Wang, Z., & Jia, K. Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection. [C] / / Proceedings of the IEEE International Conference on Intelligent Robots and Systems. 2019: 1742–1749) algorithm is used to perform three-dimensional target detection, and finally the target category, three-dimensional bounding box and confidence are obtained.

[0055] Example 1

[0056] like Figure 1 As shown in the figure, this example integrates a visible light camera, an infrared camera, and a lidar to perform 3D target detection. The steps include:

[0057] S1. Perform spatiotemporal registration of visible light cameras, infrared cameras, and lidar.

[0058] S2. Perform target detection on visible light images and infrared images using the trained target detection deep network model, and obtain a two-dimensional target list in the visible light image and infrared image, including the target category and two-dimensional bounding box.

[0059] S3. Extract the corresponding 3D point cloud cone for each 2D target bounding box detected in the visible light image and the infrared image, and perform 3D point cloud target detection in the cone.

[0060] S4. Perform non-maximum suppression (NMS) processing on the three-dimensional targets detected in all frustums to eliminate repeated detection of the same target, and output the final three-dimensional target detection result.

[0061] This example uses a framework based on frustum 3D target detection, making full use of existing mature and efficient image target detection technology. It only needs to perform 3D target detection in the frustum point cloud corresponding to the image detection results, which can not only obtain the accurate 3D target pose, but also maintain high overall computational efficiency.

[0062] This example integrates visible light images and infrared images in 2D target generation, enabling better 2D detection results both day and night. This significantly reduces missed 3D target detections due to single-modality imagery in complex environments such as poor lighting.

[0063] In this example, the specific steps for performing spatiotemporal registration of the visible light camera, infrared camera, and lidar in step S1 are as follows:

[0064] S11. Time synchronization of the LiDAR, visible light camera, and infrared camera is achieved through hard triggering or soft synchronization. The hard triggering method triggers the camera exposure when the LiDAR rotates close to the camera's field of view. The soft synchronization method records the timestamps of the LiDAR point cloud and image data, and synchronizes them by finding the visible light image and infrared image with the closest timestamp for each frame of point cloud data.

[0065] S12. Use Zhang Zhengyou calibration method or other mature camera calibration methods to calibrate the intrinsic parameters of the visible light camera and infrared camera to obtain the intrinsic parameter matrix K of the visible light camera and infrared camera V and K I .

[0066] S13. Cross-calibrate the visible light camera and infrared camera with the lidar in turn to obtain the coordinate transformation matrix between the lidar coordinate system {L} and the visible light camera coordinate system {V} And the coordinate transformation matrix between the infrared camera coordinate system {I} This allows the point cloud in the lidar coordinate system to be projected onto the optical image and infrared image respectively according to the coordinate transformation matrix and the camera intrinsic parameter matrix, thereby realizing the registration of the point cloud and the two images.

[0067] In this example, the specific steps of step S13 are:

[0068] S131. Pre-place several auxiliary calibration objects (such as heated metal plates) randomly so that the corner points of the calibration objects can be clearly identified in the visible light image, infrared image, and 3D point cloud;

[0069] S132. Use the visible light camera, infrared camera, and lidar to be calibrated to collect visible light images, infrared images, and point cloud data of the calibration object;

[0070] S133. Select the pixels of the corner points of the auxiliary calibration object in the collected visible light image and infrared image to obtain the image coordinates of the corner points; then select the laser points corresponding to the corner points in the collected point cloud data to obtain their three-dimensional coordinates, and solve the coordinate transformation matrix for the visible light image and infrared image respectively according to the selected image coordinates and the corresponding three-dimensional laser point coordinates combined with the intrinsic parameter matrix of the camera. and

[0071] Taking a visible light image as an example, assuming that (u, v) and (x, y, z) are the coordinates of the corner point in the visible light image pixel coordinate system and the lidar coordinate system, respectively, according to the pinhole camera model, the coordinate transformation relationship is as follows:

[0072]

[0073] where K V is the intrinsic parameter matrix of the visible light camera obtained by step S12 calibration, is the transformation matrix between the LiDAR coordinate system and the visible light camera coordinate system. Equation (1) can be solved using the classic PnP (Perspective-n-Point) algorithm, which requires at least three corner point pairs. When the number of corner point pairs exceeds three, the optimal solution can be obtained using least squares.

[0074] In step S2, target detection is performed on the visible light image and infrared image using the trained image target detection deep network model to obtain a two-dimensional target list in the visible light image and infrared image. Taking the visible light image as an example, in this embodiment, YoloV3 is used as the target detection network, and the specific steps are as follows:

[0075] S21. Normalize the current visible light image to 416*416;

[0076] S22. Input the normalized image into the pre-trained YoloV3 network (Redmon, J., & Farhadi, A. YOLOv3: An Incremental Improvement.

[0077] https: / / arxiv.org / abs / 1804.02767) for feature extraction and bounding box prediction;

[0078] S23. Perform post-processing such as decoding and non-maximum suppression on the network prediction results to obtain a list of target bounding box positions and category information corresponding to the original image.

[0079] In this embodiment, in step S3, a corresponding 3D point cloud frustum is extracted for each 2D target bounding box detected in the visible light image and the infrared image, and 3D point cloud target detection is performed in the frustum. Taking a 2D target detected in the visible light image as an example, the specific steps are as follows:

[0080] S31. Enlarge the two-dimensional bounding box of the target in the visible light image, keep the center position of the bounding box unchanged, multiply the length and width of the bounding box by 1.2 respectively, calculate the vertices of the new bounding box, and truncate it to the image size if it exceeds the image boundary.

[0081] S32. Project the point cloud to the visible light camera coordinate system and filter the point cloud P that falls within the viewing cone corresponding to the magnified two-dimensional box F ;

[0082] S33.In P F The trained F-ConvNet (Wang, Z., & Jia, K. Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal. [C] / / Proceedings of the IEEE International Conference on Intelligent Robots and Systems. 2019: 1742–1749) algorithm is used for three-dimensional target detection, and the target category, three-dimensional bounding box and confidence level are finally obtained.

[0083] Although the present invention has been described through preferred embodiments, the present invention is not limited to the embodiments described herein but includes various changes and modifications that may be made without departing from the scope of the present invention.

Claims

1. A multi-sensor fusion all-day three-dimensional target detection method, characterized in that: The steps include: S1. Perform spatiotemporal registration of visible light camera, infrared camera and lidar; S2. Perform target detection on the visible light image and infrared image using the trained deep network model to obtain a list of two-dimensional targets in the visible light image and infrared image, including the target category and two-dimensional bounding box; S3. extract the corresponding 3D point cloud frustum for each 2D target bounding box detected in the visible light image and the infrared image, and perform 3D point cloud target detection in the frustum; S31. Enlarging the two-dimensional bounding box of the target in the visible light image or infrared image; S32. Project the point cloud to the visible light camera coordinate system or the infrared camera coordinate system, and filter the point cloud P that falls within the viewing cone corresponding to the magnified two-dimensional box F or P F '; S33.In P F or P F 'In this method, the deep learning model algorithm of the trained point cloud data is used to perform 3D target detection, and finally the target category, 3D bounding box and confidence level are obtained; S4. Perform non-maximum suppression on the three-dimensional targets detected in all frustums to eliminate repeated detection of the same target, and output the final three-dimensional target detection result.

2. The all-day three-dimensional target detection method based on multi-sensor fusion according to claim 1, characterized in that: The step 1 is specifically as follows: S11. Time synchronization of the lidar, visible light camera, and infrared camera is achieved through hard triggering or soft synchronization. The hard trigger triggers camera exposure when the lidar rotates close to the field of view of the visible light camera or infrared camera. Soft synchronization records the timestamps of the lidar point cloud and image data and synchronizes them by finding the data frame with the closest timestamp. S12. Calibrate the internal parameters of the visible light camera and the infrared camera to obtain the internal parameter matrix K of the visible light camera and the infrared camera V and K I ; S13. Cross-calibrate the visible light camera and infrared camera with the lidar in turn to obtain the coordinate transformation matrix between the lidar coordinate system {L} and the visible light camera coordinate system {V} And the coordinate transformation matrix between the infrared camera coordinate system {I} The point cloud in the lidar coordinate system is projected onto the visible light image and infrared image respectively through the coordinate transformation matrix and the camera intrinsic parameter matrix to achieve registration between the point cloud and the two camera images.

3. The all-day three-dimensional target detection method based on multi-sensor fusion according to claim 2, characterized in that: The implementation process of step S13 is as follows: S131. Pre-place several auxiliary calibration objects randomly so that the corner points of the calibration objects can be clearly identified in the visible light image, infrared image and 3D point cloud; S132. Using the visible light camera, infrared camera and lidar to be calibrated, collect visible light images, infrared images and point cloud data of the calibration object; S133. Select the pixels of the corner points of the auxiliary calibration object in the collected visible light image and infrared image to obtain the image coordinates of the corner points; then select the laser points corresponding to the corner points in the collected point cloud data to obtain their three-dimensional coordinates, and solve the coordinate transformation matrix for the visible light image and infrared image respectively according to the selected image coordinates and the corresponding three-dimensional laser point coordinates combined with the intrinsic parameter matrix of the camera. and 4. The all-day three-dimensional target detection method based on multi-sensor fusion according to claim 3, characterized in that: The auxiliary calibration object in step S131 is a heated metal flat plate.

5. The all-day three-dimensional target detection method based on multi-sensor fusion according to claim 1, characterized in that: The step S2 is implemented as follows: S21. Preprocessing the current visible light image or infrared image and normalizing it to a preset size; S22. Input the preprocessed image into a pre-trained object detection network for feature extraction and bounding box prediction; S23. Decode and perform non-maximum suppression post-processing on the network prediction results to obtain the target bounding box position and category information list corresponding to the original image.

6. The all-day three-dimensional target detection method based on multi-sensor fusion according to claim 1, characterized in that: The deep learning model algorithm of the point cloud data of S33 can adopt but is not limited to F-PointNet or F-ConvNet.

Images