Multi-mode 3D target detection method and system thereof, terminal and medium

Abstract

The invention provides a multi-modal 3D target detection method and a system thereof. The method comprises the following steps: respectively extracting features of an original image I and a corresponding LiDAR point cloud L; carrying out point and pixel feature fusion on the features of the original image I and the corresponding LiDAR point cloud L to form LiDAR point cloud features, taking the features of the original image I as image features, and respectively generating a 3D region proposal and a 2D region proposal; extracting features from the 3D area proposal and the 2D area proposal respectively and fusing, and generating a final 3D target detection result. Meanwhile, the invention provides a corresponding terminal and a medium. According to the method, target detection is completed by using a geometric constraint relationship and feature relevance between modals; a 3D target detection task is completed through feature fusion of a point-pixel level of a first stage and a region proposal level of a second stage; and a high-quality regional proposal is generated by using geometric constraints between the image and the LiDAR point cloud.

Description

technical field [0001] The present invention relates to a 3D target detection method, in particular to a deep network-based multi-modal 3D target detection method, system, terminal and medium. Background technique [0002] Object detection is an important direction in the field of computer vision, and has broad application prospects and market value. With the development of various sensor technologies and automotive technologies, the role of 3D object detection in the field of autonomous driving has gradually emerged. The most commonly used sensors for 3D object detection tasks in autonomous driving include cameras and LiDAR, and the corresponding data types are images and LiDAR point clouds, respectively. Due to the complementarity of information between the two modalities of image and LiDAR point cloud, and the gradually decreasing price of LiDAR, 3D target detection methods based on multimodality have gradually become the focus of research at home and abroad. [0003] E...

AI Technical Summary

Problems solved by technology

[0007] 1. The input of this method is an RGB-D image, which is not suitable for sparse point clouds in outdoor scenes

[0008] 2. In this method, the extraction of the candidate frame of the RGB image is decoupled from the branch of the 3D point cloud, and the classification of the 3D point cloud is also decoupled from the RGB image, so the complementary information between the image and the point cloud cannot be fully utilized.

[0010] 1. The feature fusion of point cloud data and RGB image input in this method only stays at the level of feature splicing, and does not consider the depth correlation between th...

Images