3D target detection method, device, equipment and storage medium

Abstract

The application discloses a 3D target detection method, device and equipment and a storage medium. The method comprises the following steps: acquiring a monocular 3D target queue obtained by performing target detection on a 3D space by using a camera, and laser 3D target queues and corresponding laser 3D information obtained by performing target detection on the 3D space by using a laser radar; projecting all laser 3D targets in the laser 3D target queue and all monocular 3D targets in the monocular 3D target queue onto an image respectively to obtain laser 3D projection frames and monocular 3D projection frames, establishing a matching relationship between the laser 3D projection frames and the monocular 3D projection frames, and combining 3D target information and information of the monocular 3D target queue to obtain a multi-modal feature map; performing calculation on the multi-modal feature map by using a convolution network, performing filtering detection on a result obtained after the calculation based on a confidence threshold, and obtaining a 3D target queue. The application can optimize the 3D target detection performance.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and in particular to a 3D target detection method, apparatus, device, and storage medium. Background Technology

[0002] 3D object detection and ranging play a crucial role in the field of autonomous driving. LiDAR, with its rich 3D information and high positioning accuracy, is frequently used as a tool for object detection in autonomous driving. However, using LiDAR as a single sensor for object detection often results in a high number of false positives and false negatives, due to the inherent sparsity of LiDAR and the insufficient performance of detection models.

[0003] Therefore, existing 3D object detection through multimodal fusion has become one of the mainstream solutions in the perception field of autonomous driving. Laser and image fusion algorithms are generally divided into data-level fusion, feature-level fusion, and target-level fusion, depending on the fusion stage. In target-level fusion schemes, most are based on logical judgment, that is, calculating the cost matrix through the association between 3D and 2D data to remove or retain targets. This approach usually requires the incorporation of human experience and is somewhat affected by manually selected parameters, resulting in lower generalization performance.

[0004] Among them, CLOCs are a deep learning-based target-level fusion algorithm that projects a 3D laser target onto an image and matches and fuses the 3D target with the 2D target based on the geometric consistency between the 3D projection box and the image detection box. However, geometric consistency often fails due to the perspective principle of the camera and the occlusion phenomenon of the target box, resulting in a decrease in the overlap between the 3D target box and the 2D box.

[0005] Therefore, existing technologies suffer from low performance in 3D target detection. Summary of the Invention

[0006] In view of this, embodiments of this application provide a 3D target detection method, apparatus, device, and storage medium, aiming to solve the technical problem of low 3D detection performance in autonomous driving.

[0007] This application provides a 3D target detection method, the method comprising:

[0008] Acquire a monocular 3D target queue after the camera performs target detection in 3D space, and a laser 3D target queue and corresponding laser 3D information after the lidar performs target detection in the 3D space. The laser 3D information includes a laser 3D probability distribution map and a laser 3D target information map.

[0009] All laser 3D targets in the laser 3D target queue and all monocular 3D targets in the monocular 3D target queue are projected onto the image to obtain laser 3D projection frames and monocular 3D projection frames. A matching relationship is established between the laser 3D projection frames and the monocular 3D projection frames. Combined with the 3D target information in the laser 3D target information map and the information of the monocular 3D target queue, a multimodal feature map is obtained.

[0010] The multimodal feature map is calculated using a convolutional network, and the calculated results are filtered and detected based on a confidence threshold to obtain a 3D target queue.

[0011] In one possible implementation of this application, the step of acquiring the laser 3D target queue and corresponding laser 3D information after the laser radar performs target detection in the 3D space includes a laser 3D probability distribution map and a laser 3D target information map, comprising:

[0012] Acquire laser point cloud data for target detection in 3D space using lidar;

[0013] Based on the CenterPoint framework, the VoxelNet network is used as the backbone network, and the CenterHead is used as the detection head to calculate the laser point cloud data, so as to obtain the laser 3D target queue and the corresponding laser 3D information, including the laser 3D probability distribution map and the laser 3D target information map.

[0014] The acquisition of the monocular 3D target queue after target detection in the 3D space by the camera includes:

[0015] A monocular 3D target queue is obtained based on the Smoke algorithm. Each monocular 3D target queue contains confidence and size coordinate information.

[0016] In one possible implementation of this application, the step of projecting all laser 3D targets in the laser 3D target queue and all monocular 3D targets in the monocular 3D target queue onto an image to obtain laser 3D projection frames and monocular 3D projection frames, establishing a matching relationship between the laser 3D projection frames and the monocular 3D projection frames, and combining the 3D target information in the laser 3D target information map and the information of the monocular 3D target queue to obtain a multimodal feature map, including:

[0017] The laser 3D target queue and the monocular 3D target queue are projected onto an image to obtain the laser 3D projection frame and the monocular 3D projection frame.

[0018] Calculate the intersection-union ratio between the laser 3D projection frame and the monocular 3D projection frame to form the first BEV feature map;

[0019] The second BEV feature map is obtained by forming the confidence level of the laser 3D target;

[0020] The confidence level of the monocular 3D target is obtained to form a third BEV feature map, and the confidence level of the monocular 3D target is determined by the information in the monocular 3D target queue.

[0021] Based on the 3D target information of the laser 3D target in the laser 3D target information map, the distance value of the laser 3D target is calculated and normalized to obtain the fourth BEV feature map.

[0022] A multimodal feature map is obtained by stitching together the four BEV feature maps.

[0023] In one possible implementation of this application, calculating the intersection-over-union ratio (IoU) between the laser 3D projection frame and the monocular 3D projection frame to form a first BEV feature map includes:

[0024] The laser 3D projection frames corresponding to the queue formed by the laser 3D targets are traversed, and the following steps are performed for each traversed laser 3D projection frame:

[0025] Based on the size information of the monocular 3D projection frame and the laser 3D projection frame, the intersection-over-union ratio between the laser 3D projection frame and each of the monocular 3D projection frames is calculated. The size information is obtained by calculating the projection information of the monocular 3D target and the laser 3D target.

[0026] The cross-union ratio with the largest value in the calculation results is selected as the target cross-union ratio, forming the first BEV feature map.

[0027] In one possible implementation of this application, obtaining the second BEV feature map formed by the confidence level of the laser 3D target includes:

[0028] The laser 3D probability distribution map obtained after the lidar performs target detection in the 3D space is used as the second BEV feature map.

[0029] In one possible implementation of this application, obtaining the confidence level of the monocular 3D target to form a third BEV feature map includes:

[0030] The monocular 3D target is projected onto the new BEV feature map to obtain the position information of each monocular 3D target on the new BEV feature map.

[0031] The confidence level of the monocular 3D target is assigned to the grid point corresponding to the position information on the new BEV feature map to form the third BEV feature map. The confidence level of the monocular 3D target is determined by the information in the monocular 3D target queue.

[0032] In one possible implementation of this application, the step of calculating the multimodal feature map based on a convolutional network and filtering the calculated result based on a confidence threshold to obtain a 3D target queue includes:

[0033] The multimodal feature map is subjected to dimensionality increase processing, and channel weights are assigned to features in the multimodal feature map through the SE attention mechanism;

[0034] The confidence level of the multimodal feature map is calculated based on the channel weights and the convolutional network.

[0035] The calculated results are filtered and detected based on the confidence threshold to obtain the 3D targets corresponding to the filtered confidence levels, thus forming a 3D target queue.

[0036] This application also provides a 3D target detection device, the device comprising:

[0037] The target queue generation module is used to acquire the monocular 3D target queue after the camera performs target detection in the 3D space, and the laser 3D target queue and corresponding laser 3D information after the lidar performs target detection in the 3D space. The laser 3D information includes a laser 3D probability distribution map and a laser 3D target information map.

[0038] The feature extraction module is used to obtain the laser 3D target in the laser 3D target queue and the monocular 3D target in the monocular 3D target queue, respectively projected onto the image to obtain the laser 3D projection frame and the monocular 3D projection frame, establish the matching relationship between the laser 3D projection frame and the monocular 3D projection frame, and combine the 3D target information in the laser 3D target information map and the information of the monocular 3D target queue to obtain a multimodal feature map;

[0039] The network prediction module is used to calculate the multimodal feature map based on the convolutional network, and filter and detect the calculated results based on the confidence threshold to obtain a 3D target queue.

[0040] In one possible implementation of this application, the target queue generation module further includes:

[0041] The first acquisition submodule is used to acquire laser point cloud data for target detection in 3D space by the lidar;

[0042] The first calculation submodule is used to calculate the laser point cloud data based on the CenterPoint framework using the VoxelNet network as the backbone network and the CenterHead as the detection head to obtain the laser 3D target queue and the corresponding laser 3D information, including the laser 3D probability distribution map and the laser 3D target information map.

[0043] And / or, the target queue generation module further includes:

[0044] The second calculation submodule is used to obtain a monocular 3D target queue based on the Smoke algorithm. Each monocular 3D target queue contains confidence and size coordinate information.

[0045] And / or, the feature extraction module includes:

[0046] The second acquisition submodule is used to acquire all laser 3D targets in the laser 3D target queue and all monocular 3D targets in the monocular 3D target queue projected onto the image to obtain laser 3D projection frames and monocular 3D projection frames.

[0047] The third calculation submodule is used to calculate the intersection-union ratio between the laser 3D projection frame and the monocular 3D projection frame to form the first BEV feature map.

[0048] The third acquisition submodule is used to acquire the second BEV feature map formed by the confidence of the laser 3D target;

[0049] The fourth acquisition submodule is used to acquire the confidence level of the monocular 3D target to form the third BEV feature map. The confidence level of the monocular 3D target is determined by the information in the monocular 3D target queue.

[0050] The fourth calculation submodule is used to calculate the distance value of the laser 3D target based on the 3D target information of the laser 3D target in the laser 3D target information map, and normalize it to obtain the fourth BEV feature map.

[0051] The feature extraction submodule is used to obtain a multimodal feature map by stitching together the four BEV feature maps.

[0052] And / or, the third computing submodule further includes:

[0053] The traversal unit is used to traverse the laser 3D projection frames corresponding to the queue formed by the laser 3D targets, and to perform the following steps for each laser 3D projection frame traversed:

[0054] The data calculation subunit is used to calculate the intersection-over-union ratio between the laser 3D projection frame and each of the monocular 3D projection frames based on the size information of the monocular 3D projection frame and the laser 3D projection frame. The size information is calculated through the projection information of the monocular 3D target and the laser 3D target.

[0055] The data selection sub-unit is used to select the cross-union ratio with the largest value in the calculation results as the target cross-union ratio, forming the first BEV feature map.

[0056] And / or, the third acquisition submodule further includes:

[0057] The feature map acquisition unit is used to take the laser 3D probability distribution map obtained by the lidar after performing target detection in the 3D space as the second BEV feature map.

[0058] And / or, the third acquisition submodule further includes:

[0059] The target projection unit is used to project the monocular 3D target onto a new BEV feature map to obtain the position information of each monocular 3D target on the new BEV feature map.

[0060] The assignment unit is used to assign the confidence level of the monocular 3D target to the grid point corresponding to the position information on the new BEV feature map to form a third BEV feature map. The confidence level of the monocular 3D target is determined by the information in the monocular 3D target queue.

[0061] And / or, the network prediction module further includes:

[0062] The preprocessing submodule is used to perform dimensionality upscaling on the multimodal feature map and assign channel weights to features in the multimodal feature map through the SE attention mechanism.

[0063] The network prediction submodule is used to calculate the confidence level of the multimodal feature map based on the channel weights and the convolutional network.

[0064] The filtering submodule is used to filter and detect the calculated results based on the confidence threshold, obtain the 3D targets corresponding to the filtered confidence levels, and form a 3D target queue.

[0065] This application also provides a 3D target detection device, which is a physical node device. The 3D target detection device includes: a memory, a processor, and a program of the 3D target detection method stored in the memory and executable on the processor. When the program of the 3D target detection method is executed by the processor, it can implement the steps of the 3D target detection method as described above.

[0066] To achieve the above objectives, a computer-readable storage medium is also provided, on which a 3D target detection program is stored, wherein when the 3D target detection program is executed by a processor, it implements the steps of any of the 3D target detection methods described above.

[0067] This application provides a 3D target detection method, apparatus, device, and storage medium. It acquires a monocular 3D target queue after target detection in 3D space by a camera, and a laser 3D target queue and corresponding laser 3D information after target detection in the 3D space by a lidar. The laser 3D information includes a laser 3D probability distribution map and a laser 3D target information map. It obtains laser 3D projection frames and monocular 3D projection frames by projecting all laser 3D targets in the laser 3D target queue and all monocular 3D targets in the monocular 3D target queue onto an image, respectively. It establishes a matching relationship between the laser 3D projection frames and the monocular 3D projection frames, and combines the 3D target information in the laser 3D target information map and the information of the monocular 3D target queue to obtain a multimodal feature map. It then calculates the multimodal feature map using a convolutional network, filters and detects the calculated results based on a confidence threshold, and obtains a 3D target queue. In other words, by acquiring the monocular 3D target queue after camera target detection in 3D space, and the laser 3D target queue and corresponding laser 3D information after LiDAR target detection in 3D space, a matching relationship is established between the laser 3D target projection bounding box and the monocular 3D target projection bounding box on the image, thus obtaining accurate multimodal features. Furthermore, by using a simple convolutional network to calculate the multimodal feature map, contextual information is effectively utilized for learning, improving the generalization performance of target detection. Specifically, in this application, the confidence of the 3D target is recalculated by post-fusion of the laser 3D target and the monocular 3D target, thereby optimizing the 3D target detection performance. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating the first embodiment of the 3D target detection method of this application;

[0069] Figure 2 This is a schematic diagram of the algorithm flow of the first embodiment of the 3D target detection method of this application;

[0070] Figure 3 This is a comparison of the overlap between the 2D detection frame and the monocular 3D projection frame and the laser 3D projection frame in the 3D target detection method of this application.

[0071] Figure 4 A schematic diagram of the network prediction module in the 3D target detection method of this application;

[0072] Figure 5This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application. Detailed Implementation

[0073] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0074] This application provides a 3D target detection method. In one embodiment of this 3D target detection method, it is applied to a 3D target detection device, as shown below. Figure 1 and Figure 2 The method includes:

[0075] Step S10: Obtain the monocular 3D target queue after the camera performs target detection in the 3D space, and the laser 3D target queue and corresponding laser 3D information after the lidar performs target detection in the 3D space. The laser 3D information includes a laser 3D probability distribution map and a laser 3D target information map.

[0076] Step S20: Project all laser 3D targets in the laser 3D target information image and all monocular 3D targets in the monocular 3D target queue onto the image respectively to obtain laser 3D projection frames and monocular 3D projection frames. Establish the matching relationship between the laser 3D projection frames and the monocular 3D projection frames. Combine the 3D target information in the laser 3D target information image and the information of the monocular 3D target queue to obtain a multimodal feature map.

[0077] Step S30: Calculate the multimodal feature map using the convolutional network, and filter and detect the calculated results based on a confidence threshold to obtain a 3D target queue.

[0078] This embodiment aims to improve the detection performance of 3D targets in autonomous driving.

[0079] As an example, in this application, targets in 3D space are detected using LiDAR to obtain a LiDAR 3D target queue, and targets in 3D space are detected using a vehicle camera to obtain a monocular 3D target queue. Target-level fusion is performed on the LiDAR 3D targets and the monocular 3D targets, and the confidence level of the 3D targets is recalculated to optimize the detection performance of 3D targets. Compared to existing technologies that use a geometric consistency mechanism between the LiDAR 3D projection frame and the image 2D detection frame to match and fuse 3D targets with 2D targets (ideally, the 2D detection frame of the image detection and the 3D projection frame of the LiDAR 3D detection coincide, but due to occlusion and camera perspective, in reality, the 2D detection frame and the 3D projection frame cannot completely coincide. Therefore, the geometric consistency usually fails due to the camera's perspective principle and the occlusion phenomenon of the target frame, resulting in low 3D target detection performance), the detection frames of monocular 3D targets and LiDAR 3D targets have a higher degree of overlap. In this application, a monocular 3D target queue after camera-based target detection in 3D space, and a laser 3D target queue after LiDAR-based target detection in 3D space, along with their corresponding laser 3D information and laser 3D target queue information, are obtained. A matching relationship is established between the laser 3D target projection bounding boxes on the image and the monocular 3D target projection bounding boxes on the image, enabling accurate multimodal features to be obtained. Furthermore, a simple convolutional network is used to calculate the multimodal feature maps, effectively utilizing contextual information for learning and improving the generalization performance of target detection. That is, in this application, by post-fusion of laser 3D targets and monocular 3D targets, the confidence of the 3D targets is recalculated, thereby optimizing the performance of 3D target detection.

[0080] As an example, in this application, the detection of laser 3D targets employs the CenterPoint framework with a VoxelNet network as the backbone and a CenterHead as the detection head to obtain laser 3D information. This information includes a laser 3D probability distribution map and a laser 3D target information map. Subsequently, the laser 3D probability distribution map is fused and updated using relevant data from monocular 3D targets to obtain an accurate 3D target queue. For instance, laser 3D information is calculated based on the laser point cloud data collected by the LiDAR using the CenterPoint framework. This laser 3D information includes a laser 3D probability distribution map and a 3D target size coordinate information map. The laser 3D probability distribution map is directly used to generate the second BEV feature map. All laser 3D target information represented in the laser 3D target size coordinate map is then consistently correlated with the features of monocular 3D targets to obtain accurate multimodal features.

[0081] As an example, in this application, during feature fusion between the 2D detection bounding box and the laser 3D projection bounding box of the image detection, the overlap between the 2D detection bounding box and the laser 3D projection bounding box is low due to occlusion and image perspective issues, resulting in the failure of the IoU (Intersection over Union) feature. Therefore, by projecting the laser 3D target and the monocular 3D target onto the image, a matching relationship is established between the laser 3D projection bounding box and the monocular 3D projection bounding box to obtain multimodal features. That is, the geometric consistency association between the image and the laser is established using the monocular 3D detection projection and the laser 3D projection. For example, the IoU between the laser 3D projection bounding box and the monocular 3D projection bounding box is calculated to form the first BEV feature map, the confidence of the monocular 3D target is obtained to form the third BEV feature map, and the distance value of the laser 3D target is calculated based on the 3D target information (coordinate information) of the laser 3D target in the laser 3D target information map and normalized to obtain the fourth BEV feature map. Feature extraction is performed using four layers of BEV feature maps to obtain multimodal feature maps.

[0082] As an example, in this application, a matching relationship is established using laser 3D projection frames and monocular 3D projection frames. Monocular 3D projection frames with a high degree of overlap with the laser 3D projection frames are matched and used to calculate the intersection-over-union ratio (IoU) between the laser 3D projection frames and the monocular 3D projection frames to obtain IoU features and establish a consistent association between the image and the laser. For example, each laser 3D projection frame corresponding to the queue formed by the laser 3D target is traversed, and the following steps are performed on the traversed laser 3D projection frames: based on the size information of the monocular 3D projection frames and the laser 3D projection frames, the intersection-over-union ratio between the laser 3D projection frame and each monocular 3D projection frame is calculated. The size information is calculated using the projection information of the monocular 3D target and the laser 3D target; the intersection-over-union ratio with the largest value in the calculation results is selected as the target intersection-over-union ratio, forming the first BEV feature map.

[0083] As an example, in this application, to establish a consistent association between the image and the laser through monocular 3D detection projection and laser 3D projection, the monocular 3D target is projected onto a new BEV feature map to obtain the confidence level of the monocular 3D target, which is then used to build a multimodal feature map. For instance, a queue of monocular 3D targets is projected onto the new BEV feature map to obtain the position information of each monocular 3D target on the new BEV feature map; the confidence level of the monocular 3D target is assigned to the grid point corresponding to the position information on the new BEV feature map to form a third BEV feature map.

[0084] As an example, this application uses a simple convolutional network (such as 2D convolution) to perform target-level fusion of multimodal feature maps, effectively utilizing the contextual information of the features and improving the generalization performance of 3D object detection. For instance, the multimodal feature maps are dimensionality-enhanced, and channel weights are assigned to the features in the multimodal feature maps using an SE attention mechanism. Based on the channel weights and the convolutional network, the confidence level of the multimodal feature maps is calculated. The calculated results are then filtered and detected based on a confidence threshold to obtain the 3D targets corresponding to the filtered confidence levels, forming a 3D target queue.

[0085] In this embodiment, the specific application scenario is:

[0086] 3D object detection and ranging play a crucial role in the field of autonomous driving. LiDAR, with its rich 3D information and high positioning accuracy, is often used as a tool for object detection in autonomous driving. However, using LiDAR as a single sensor for object detection usually results in a high number of false positives and false negatives. This is due to the inherent sparsity of LiDAR and the insufficient performance of the detection model.

[0087] For the reasons mentioned above, current 3D object detection methods employ geometric consistency between 3D projected bounding boxes and image detection boxes when performing target-level fusion of 3D targets detected by images and 3D targets detected by lasers. However, this geometric consistency often fails due to camera perspective and occlusion phenomena in the target bounding boxes, leading to reduced overlap between the 3D and 2D bounding boxes. Furthermore, in target-level fusion schemes, calculating the cost matrix through the correlation between 3D and 2D data requires incorporating human experience and is subject to lower generalization performance due to the influence of manually selected parameters. This results in low 3D detection performance in autonomous driving.

[0088] As an example, the 3D target detection method can be applied to a 3D target detection system, which includes a vehicle camera, a lidar, and a 3D target detection device.

[0089] As an example, a 3D target detection device can be built into a vehicle, or into other mobile terminals, or it can operate independently of vehicles and other mobile terminals.

[0090] As an example, a 3D target can be a different type of entity in 3D space, such as a car, a bus, or other obstacles, without any specific limitation.

[0091] As an example, 3D space refers to the spatial range in which a camera or lidar on a vehicle detects a target; it can be the environmental space surrounding the vehicle while it is in motion.

[0092] The specific steps are as follows:

[0093] Step S10: Obtain the monocular 3D target queue after the camera performs target detection in the 3D space, and the laser 3D target queue and corresponding laser 3D information after the lidar performs target detection in the 3D space. The laser 3D information includes a laser 3D probability distribution map and a laser 3D target information map.

[0094] As an example, a monocular 3D target refers to a target obtained from an image obtained through target detection in the camera's 3D space, and this target is in 3D form. Multiple monocular 3D targets form a monocular 3D target queue.

[0095] As an example, a laser 3D target refers to a 3D target obtained by detecting the surrounding environment of a vehicle using a lidar sensor and processing the laser point cloud data collected by the lidar.

[0096] As an example, a BEV map is a bird's-eye view, a map of the location of a 3D target obtained from the direction of the BEV view. The BEV map is divided into multiple grids, each grid corresponding to a 3D target.

[0097] As an example, a 3D probability distribution map (W×H×1) of laser light is obtained by LiDAR detection, denoted as S. lidar W and H represent the dimensions of the probability distribution map. The value of each point in this probability distribution map is the confidence score, which indicates the probability that a 3D object exists at that point.

[0098] As an example, the confidence level is one-dimensional information data, while the 3D target information in the laser 3D target information map is seven-dimensional information data, including the coordinate information, size information, and yaw angle information of the 3D target. The target information of the i-th laser 3D target is... express.

[0099] As an example, this involves acquiring a monocular 3D target after target detection in 3D space using a camera, and a laser 3D target after target detection in 3D space using a LiDAR scanner. These two pieces of information are used to detect the position, distance, and other information of the 3D target vehicle, providing a data foundation for autonomous driving.

[0100] Step S20: Project all laser 3D targets in the laser 3D target queue and all monocular 3D targets in the monocular 3D target queue onto the image respectively to obtain laser 3D projection frames and monocular 3D projection frames. Establish the matching relationship between the laser 3D projection frames and the monocular 3D projection frames. Combine the 3D target information in the laser 3D target information map and the information of the monocular 3D target queue to obtain a multimodal feature map.

[0101] As an example, multimodal features refer to multiple features used to reflect a 3D target. Each feature represents a single entity data point. By using feature data from different modalities, the accuracy of obtaining 3D targets can be improved.

[0102] As an example, a laser 3D projection frame refers to the 2D bounding box obtained by projecting a laser 3D target onto an image. A monocular 3D projection frame refers to the 2D bounding box obtained by projecting a monocular 3D target onto an image.

[0103] As an example, an image can also generate a 2D detection box through 2D detection. Ideally, the 2D detection box and the 3D projection box (the bounding box formed by the projection of the laser 3D target onto the image, i.e., the laser 3D projection box) should coincide. However, due to issues such as occlusion of the 3D target or camera perspective, in reality, the 2D detection box and the 3D projection box cannot completely coincide. Since the monocular 3D projection box and the 3D projection box belong to the same 3D space, the projection onto the image is more consistent.

[0104] Reference Figure 3 , Figure 3 The images show a comparison of the overlap between the 2D detection frame and the monocular 3D projection frame and the laser 3D projection frame. Figure 3 (a) and (b) show the overlap between the 2D detection frame and the laser 3D projection frame. Figure 3 (c) and (d) show the overlap between the monocular 3D projection frame and the laser 3D projection frame.

[0105] Due to occlusion and image perspective issues, the 2D detection bounding box has a low overlap with the 3D projection bounding box, causing the IoU feature to fail. In contrast, the monocular 3D projection bounding box has a high overlap with the laser 3D projection bounding box.

[0106] Therefore, by establishing a consistent correlation between the image and the laser using monocular 3D detection projection and laser 3D projection, and by establishing a matching relationship between the laser 3D projection frame and the monocular 3D projection frame, a more accurate multimodal feature map can be obtained. In other words, designing a matching mechanism between monocular 3D and laser 3D projection effectively solves the problem of geometric inconsistency between 3D and 2D, improving the accuracy of 3D target detection.

[0107] As an example, the process involves projecting all laser 3D targets in the laser 3D target queue and all monocular 3D targets in the monocular 3D target queue onto an image, obtaining laser 3D projection frames and monocular 3D projection frames, establishing a matching relationship between the laser 3D projection frames and the monocular 3D projection frames, and combining the 3D target information in the laser 3D target information map with the information of the monocular 3D target queue to obtain a multimodal feature map, including:

[0108] Step S21: Obtain the laser 3D projection frame and monocular 3D projection frame by projecting all laser 3D targets in the laser 3D target queue and all monocular 3D targets in the monocular 3D target queue onto the image.

[0109] Step S22: Calculate the intersection-union ratio between the laser 3D projection frame and the monocular 3D projection frame to form the first BEV feature map;

[0110] Step S23: Obtain the laser 3D target confidence score to form a second BEV feature map;

[0111] Step S24: Obtain the confidence level of the monocular 3D target to form the third BEV feature map. The confidence level of the monocular 3D target is determined by the information in the monocular 3D target queue.

[0112] Step S25: Based on the 3D target information of the laser 3D target in the laser 3D target information map, calculate the distance value of the laser 3D target, normalize it, and obtain the fourth BEV feature map.

[0113] Step S26: The four BEV feature maps are stitched together to obtain a multimodal feature map.

[0114] As an example, a laser 3D target and a monocular 3D target are projected onto an image to obtain a laser 3D projection frame and a monocular 3D projection frame, which refer to 2D bounding boxes projected onto the image.

[0115] As an example, the accuracy of location information in 3D object detection is typically measured using overlap or Intersection over Union (IoU). IoU is the quotient of the intersection value between the laser 3D projection frame and the monocular 3D projection frame, and the union value between the laser 3D projection frame and the monocular 3D projection frame; that is, the intersection value divided by the union value. For calculating the IoU of the 2D bounding boxes projected onto the image by the laser 3D projection frame and the monocular 3D projection frame, the intersection and union values ​​correspond to the areas of the intersection and union of the laser 3D projection frame and the monocular 3D projection frame, respectively.

[0116] That is, IoU = Area of ​​Intersection / Area of ​​Union, where Area of ​​Intersection is the area of ​​the intersection between the laser 3D projection frame and the monocular 3D projection frame projected onto the 2D frame on the image, and Area of ​​Union is the area of ​​the union between the laser 3D projection frame and the monocular 3D projection frame projected onto the 2D frame on the image.

[0117] The first BEV feature map is formed by calculating the intersection-union ratio between the laser 3D projection frame and the monocular 3D projection frame. The first BEV feature map contains the IoU feature information between the two laser 3D targets and the monocular 3D target.

[0118] As an example, a second BEV feature map is obtained based on the confidence level of the laser 3D target. That is, the confidence level distribution of the laser 3D target on the second BEV feature map. Confidence level refers to the probability that the laser 3D target appears in each grid cell of the second BEV image.

[0119] As an example, the confidence level of a monocular 3D target is used to form a third BEV feature map. The confidence level of the monocular 3D target is determined by information from the monocular 3D target queue. That is, the confidence distribution map of the monocular 3D target on the third BEV feature map. Confidence level refers to the probability that a monocular 3D target appears in each grid cell of the third BEV feature map.

[0120] As an example, based on the 3D target information of the laser 3D target, the distance d from the laser 3D target to the center of the vehicle is calculated. i That is, d i To generate a normalized distance map of the laser 3D target coordinates, calculate the distance between the center point of the laser 3D target and the center point of the vehicle.

[0121] For example, d i The calculation method is as follows: in Let be the Euclidean distance between the i-th laser 3D target and the vehicle. This refers to the maximum distance between the center point of the laser 3D target and the center point of the vehicle, thus forming a feature layer of distance values.

[0122] As an example, a multimodal feature map is obtained using multimodal features such as the IoU feature map (first BEV feature map), the laser 3D probability distribution map (second BEV feature map), the monocular 3D confidence distribution map (third BEV feature map), and the distance distribution map (fourth BEV feature map). That is, a 4-dimensional feature map (W×H×4) is designed using multimodal features:

[0123]

[0124] As an example, obtaining the second BEV feature map formed by the confidence level of the laser 3D target includes:

[0125] Step S231: The laser 3D probability distribution map obtained after the lidar performs target detection in the 3D space is used as the second BEV feature map.

[0126] As an example, the laser 3D probability distribution map detected by the laser 3D target based on the CenterPoint framework is directly used as the second BEV feature map.

[0127] As an example, obtaining the confidence level of the monocular 3D target to form a third BEV feature map includes:

[0128] Step S241: Project the monocular 3D target onto the new BEV feature map to obtain the position information of each monocular 3D target on the new BEV feature map;

[0129] Step S242: Assign the confidence level of the monocular 3D target to the grid point corresponding to the position information on the new BEV feature map to form the third BEV feature map.

[0130] As an example, the monocular 3D target queue is projected onto a new BEV feature map. The size range of the new BEV feature map is the same as that of the laser 3D detection, which is W×H. The size of each grid on the third BEV map is denoted as p. Assuming that the range of the X direction of laser detection is (-X, X), then p = 2X / W.

[0131] Based on the size coordinate information of the monocular 3D target, the center point of the monocular 3D target is projected onto the new BEV feature map to obtain the position information of the monocular 3D target on the new BEV feature map, that is, to find the grid point of the monocular 3D target on the new BEV feature map. The confidence score of the monocular 3D target is assigned to the grid point. The assignment method for other monocular 3D targets is basically the same, and will not be repeated here. That is, the confidence score of the 3D target is assigned to the grid point of the 3D target on the new BEV feature map. Thus, a confidence distribution map of the monocular 3D target is obtained, which is the third BEV feature map.

[0132] As an example, due to the small number of monocular 3D targets, the resulting monocular 3D confidence distribution map is too sparse. Therefore, it is necessary to apply Gaussian blur to the features of each grid point on the third BEV feature map. Gaussian blur (also called Gaussian smoothing) is a commonly used technique in image processing, mainly used to reduce image noise and detail. That is, each grid point feature is amplified according to the Gaussian radius, expanding the representation range of the monocular 3D target and better learning the features of the entire image. It should be noted that the larger the Gaussian radius, the more blurred the image; numerically speaking, the smoother the image. Therefore, the Gaussian radius is set according to the size of the monocular 3D target.

[0133] Step S30: Calculate the multimodal feature map using the convolutional network, and filter and detect the calculated results based on a confidence threshold to obtain a 3D target queue.

[0134] As an example, convolutional networks include neural networks such as 2D convolution and 3D convolution. Convolutional neural networks typically consist of convolutional layers, activation layers, and pooling layers, taking image data as input and outputting a specific feature space of the image.

[0135] As an example, a simple 2D convolutional group is used to predict the multimodal feature map, calculating a (W×H×1) feature, which is a new probability distribution map of the 3D target. This new probability distribution map is then used to modify the initial laser 3D probability distribution map obtained from laser 3D detection. The final 3D target queue is obtained through confidence thresholding and NMS filtering. In essence, a confidence threshold is obtained, and points in the probability distribution map that are greater than the confidence threshold are selected; the 3D targets corresponding to these points form the 3D target queue.

[0136] As an example, the step of calculating the multimodal feature map based on a convolutional network and filtering the calculated results based on a confidence threshold to obtain a 3D target queue includes:

[0137] Step S31: Perform dimensionality upscaling on the multimodal feature map and assign channel weights to the features in the multimodal feature map using the SE attention mechanism;

[0138] Step S32: Calculate the confidence level of the multimodal feature map based on the channel weights and the convolutional network;

[0139] Step S33: Filter the calculated results based on the confidence threshold to obtain the 3D targets corresponding to the filtered confidence levels, thus forming a 3D target queue.

[0140] As an example, refer to Figure 4 The feature map is dimensionality-enhanced to enrich the 4-dimensional multimodal feature map, such as increasing it to 16 dimensions. Different weights are assigned to different channels of the feature map using the SE feature attention mechanism, which is adaptively obtained during data computation. A series of 2D convolutions (W×H×1) are used to compute the feature map, including two 3×3 convolutions, followed by a sigmoid operation to obtain the final probability distribution map. This probability distribution map replaces the initial 3D detection distribution map, and the final 3D detection target queue is obtained by applying a confidence threshold and NMS filtering.

[0141] In this embodiment, the fusion of monocular 3D targets and laser 3D targets achieves a higher degree of matching in geometric consistency. Simple 2D convolutional layer operations are sufficient to improve detection performance. The 2D convolution calculates features, effectively utilizing the contextual information of the BEV feature space. Furthermore, an SE attention mechanism is employed to assign different weights to different channels of the features, enhancing the network's attention to different feature channels and making the learned features more effective and accurate. In other words, target-level fusion based on monocular 3D and laser 3D detection achieves improved 3D detection performance.

[0142] This application provides a 3D target detection method, apparatus, device, and storage medium. Compared with the low performance of current 3D detection in autonomous driving, this application acquires a monocular 3D target queue after camera target detection in 3D space, and a laser 3D target queue and corresponding laser 3D information after LiDAR target detection in 3D space. A matching relationship is established between the laser 3D target projection bounding box on the image and the monocular 3D target projection bounding box on the image, enabling accurate multimodal features to be obtained. Furthermore, a simple convolutional network is used to calculate the multimodal feature map, effectively utilizing contextual information for learning and improving the generalization performance of target detection. That is, in this application, the confidence of the 3D target is recalculated by post-fusion of laser 3D targets and monocular 3D targets, thereby optimizing the 3D target detection performance.

[0143] Based on the first embodiment of the 3D target detection method described above, a second embodiment of the 3D target detection method is proposed.

[0144] The acquisition of the laser 3D target queue and corresponding laser 3D information after the laser radar performs target detection in the 3D space includes a laser 3D probability distribution map and a laser 3D target information map, including:

[0145] Step A11: Obtain laser point cloud data for target detection in 3D space using lidar;

[0146] Step A12: Based on the CenterPoint framework, the VoxelNet network is used as the backbone network, and the CenterHead is used as the detection head to calculate the laser point cloud data to obtain the laser 3D target queue and the corresponding laser 3D information, including the laser 3D probability distribution map and the laser 3D target information map.

[0147] The laser point cloud data is calculated based on the CenterPoint model to obtain a laser 3D probability distribution map and a laser 3D target information map. The value corresponding to each 3D target in the laser 3D probability distribution map is the confidence level of the laser 3D target, and each 3D target in the laser 3D target information map corresponds to the 3D target information of the laser 3D target.

[0148] As an example, the confidence level is one-dimensional information data, and the 3D target information is seven-dimensional information data, including the coordinate information, size information, and yaw angle information of the 3D target. The target information of the i-th laser 3D target is... express.

[0149] As an example, CenterPoint is a center-based two-stage detection-tracking model. In the first stage, a keypoint detector (e.g., LiDAR) is used to detect the center of the 3D target, and regressions are performed on the 3D size, 3D orientation, and velocity of the detection box. In the second stage, a memory-based thinning module is designed to refine the detection box generated in the first stage using additional point features.

[0150] As an example, laser 3D target detection employs the CenterPoint framework. This framework characterizes the laser point cloud data from LiDAR target detection in 3D space, generating Voxel features, and then extracts the features from these Voxel features. For instance, the backbone of the CenterPoint framework uses VoxelNet as a 3D encoder to extract features, obtaining a feature map. This map is then directly output as a probability distribution map (W×H×1), denoted as S, through an anchor-free detection head. lidar The value corresponding to each point in the probability distribution diagram represents the first confidence level of the laser 3D target. The other branch of the CenterPoint framework regresses the size coordinates of the target bounding box, which is the 3D target information, denoted as... The expression is represented as follows: (x, y, z) represents the position information of the 3D target, (l, w, h) represents the length, width, and height information of the 3D target, and θ represents the yaw angle of the 3D target.

[0151] As an example, W and H represent the dimensions of the probability distribution map, where the value of each point is the confidence level of the laser 3D target. Each point in the probability distribution map is projected onto the BEV map, with each point corresponding to a grid position in the BEV, and each grid size denoted as p. Assuming the range of the laser detection in the X direction is (-X, X), then p = 2X / W.

[0152] As an example, the laser 3D target detection result includes a laser 3D probability distribution map and a laser 3D target information map. The probability distribution map and the 3D target information map have the same size. Therefore, there are a total of (W×H) laser 3D targets obtained through laser detection, forming a laser 3D target queue.

[0153] In this embodiment, the laser 3D target in 3D space is calculated using the CenterPoint framework, and target-level fusion is performed by combining the monocular 3D target to obtain an accurate 3D target queue.

[0154] As an example, the acquisition of the monocular 3D target queue after the camera performs target detection in the 3D space includes:

[0155] Step A13: Obtain a monocular 3D target queue based on the Smoke algorithm. Each monocular 3D target queue contains confidence level and size coordinate information.

[0156] The camera detects targets in the 3D space image, resulting in a monocular 3D target queue. For example, the Smoke network framework is used for image detection, and the NMS algorithm is used to obtain the final n monocular 3D targets, forming the monocular 3D target queue. The confidence score and size coordinates of each monocular 3D target are directly calculated using the Smoke network framework. The size coordinates of the j-th monocular 3D target are used... express.

[0157] Based on the first or second embodiment of the 3D target detection method described above, a third embodiment of the 3D target detection method is proposed.

[0158] The step of calculating the intersection-over-union ratio (IoU) between the laser 3D projection frame and the monocular 3D projection frame to form a first BEV map includes:

[0159] Step B11: Traverse each laser 3D projection frame corresponding to the queue formed by the laser 3D target, and perform the following steps on the traversed laser 3D projection frames:

[0160] Step B12: Based on the size information of the monocular 3D projection frame and the laser 3D projection frame, calculate the intersection-over-union ratio between the laser 3D projection frame and each of the monocular 3D projection frames. The size information is calculated using the projection information of the monocular 3D target and the laser 3D target.

[0161] Step B13: Select the cross-union ratio with the largest value in the calculation results as the target cross-union ratio to form the first BEV feature map.

[0162] As an example, the Intersection over Union (IoU) refers to the crossover ratio of the 2D bounding boxes generated by projecting the laser 3D projection frame and the monocular 3D projection frame onto the image. To establish the matching relationship between the monocular 3D projection frame and the laser 3D projection frame to obtain the 3D target, the IoU value is calculated between the laser 3D projection frame and all monocular 3D projection frames. The greater the overlap between the laser 3D projection frame and the monocular 3D projection frame, the larger the IoU value. Therefore, by taking the largest IoU value among the calculated IoU values ​​between the laser 3D projection frame and all monocular 3D projection frames, the first BEV feature map, i.e., the IoU feature map, is established.

[0163] As an example, for each laser 3D projection frame, the following steps are performed:

[0164] Based on the size information of the monocular 3D projection frame and the laser 3D projection frame, the cross-union ratio between the laser 3D projection frame and each monocular 3D projection frame is calculated. The size information is obtained by calculating the projection information of the monocular 3D target and the laser 3D target.

[0165] For example, a laser 3D projection frame is used as an example. The laser 3D projection frame has eight corner points, and its projection onto the image has eight projection coordinates. Based on the 3D target information of the laser 3D target, the coordinates of each corner point of the laser 3D target can be determined. By combining the intrinsic and extrinsic parameters of the image captured by the camera, the eight projection coordinates projected onto the image can be obtained. The four outermost projection points are selected to form an outer envelope 2D frame, denoted as the laser 3D projection frame. The specific implementation of a monocular 3D projection frame is basically the same and will not be repeated here.

[0166] Therefore, using the size information of the monocular 3D projection frame and the laser 3D projection frame, the crossover ratio (CRO) between the laser 3D projection frame and each monocular 3D projection frame is calculated, and the CRO with the largest value in the calculated results is selected as the target CRO. The first BEV feature map is formed using at least one target CRO, and the first BEV feature map contains the IoU feature information between the two laser 3D targets and the monocular 3D target.

[0167] In this embodiment, by establishing a consistent association between the image and the laser between the monocular 3D projection frame and the laser 3D projection frame, the problem of geometric consistency failing due to camera perspective and target frame occlusion is solved, thereby improving the accuracy of 3D target detection.

[0168] Reference Figure 5 , Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application.

[0169] like Figure 5As shown, the 3D target detection device may include: a processor 1001, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between the processor 1001 and the memory 1005.

[0170] Optionally, the 3D target detection device may also include a user interface, a network interface, a camera, RF (Radio Frequency) circuitry, sensors, a WiFi module, etc. The user interface may include a display screen and an input submodule such as a keyboard; optional user interfaces may also include standard wired or wireless interfaces. The network interface may include standard wired or wireless interfaces (such as a Wi-Fi interface).

[0171] Those skilled in the art will understand that Figure 5 The 3D target detection device structure shown does not constitute a limitation on the 3D target detection device. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0172] like Figure 5 As shown, the memory 1005, serving as a storage medium, may include an operating system, a network communication module, and a 3D target detection program. The operating system is a program that manages and controls the hardware and software resources of the 3D target detection device, supporting the operation of the 3D target detection program and other software and / or programs. The network communication module is used to enable communication between the various components within the memory 1005, as well as communication with other hardware and software in the 3D target detection system.

[0173] exist Figure 5 In the 3D target detection device shown, the processor 1001 is used to execute the 3D target detection program stored in the memory 1005 to implement the steps of the 3D target detection method described above.

[0174] The specific implementation of the 3D target detection device in this application is basically the same as the embodiments of the 3D target detection method described above, and will not be repeated here.

[0175] This application also provides a 3D target detection device, the device comprising:

[0176] The target queue generation module is used to acquire the monocular 3D target queue after the camera performs target detection in the 3D space, and the laser 3D target queue and corresponding laser 3D information after the lidar performs target detection in the 3D space. The laser 3D information includes a laser 3D probability distribution map and a laser 3D target information map.

[0177] The feature extraction module is used to obtain the laser 3D target in the laser 3D target queue and the monocular 3D target in the monocular 3D target queue, respectively projected onto the image to obtain the laser 3D projection frame and the monocular 3D projection frame, establish the matching relationship between the laser 3D projection frame and the monocular 3D projection frame, and combine the 3D target information in the laser 3D target information map and the information of the monocular 3D target queue to obtain a multimodal feature map;

[0178] The network prediction module is used to calculate the multimodal feature map based on the convolutional network, and filter and detect the calculated results based on the confidence threshold to obtain a 3D target queue.

[0179] In one possible implementation of this application, the target queue generation module further includes:

[0180] The first acquisition submodule is used to acquire laser point cloud data for target detection in 3D space by the lidar;

[0181] The first calculation submodule is used to calculate the laser point cloud data based on the CenterPoint framework using the VoxelNet network as the backbone network and the CenterHead as the detection head to obtain the laser 3D target queue and the corresponding laser 3D information, including the laser 3D probability distribution map and the laser 3D target information map.

[0182] And / or, the target queue generation module further includes:

[0183] The second calculation submodule is used to obtain a monocular 3D target queue based on the Smoke algorithm. Each monocular 3D target queue contains confidence and size coordinate information.

[0184] And / or, the feature extraction module includes:

[0185] The second acquisition submodule is used to acquire all laser 3D targets in the laser 3D target queue and all monocular 3D targets in the monocular 3D target queue projected onto the image to obtain laser 3D projection frames and monocular 3D projection frames.

[0186] The third calculation submodule is used to calculate the intersection-union ratio between the laser 3D projection frame and the monocular 3D projection frame to form the first BEV feature map.

[0187] The third acquisition submodule is used to acquire the second BEV feature map formed by the confidence of the laser 3D target;

[0188] The fourth acquisition submodule is used to acquire the confidence level of the monocular 3D target to form the third BEV feature map. The confidence level of the monocular 3D target is determined by the information in the monocular 3D target queue.

[0189] The fourth calculation submodule is used to calculate the distance value of the laser 3D target based on the 3D target information of the laser 3D target in the laser 3D target information map, and normalize it to obtain the fourth BEV feature map.

[0190] The feature extraction submodule is used to obtain a multimodal feature map by stitching together the four BEV feature maps.

[0191] And / or, the third computing submodule further includes:

[0192] The traversal unit is used to traverse the laser 3D projection frames corresponding to the queue formed by the laser 3D targets, and to perform the following steps for each laser 3D projection frame traversed:

[0193] The data calculation subunit is used to calculate the intersection-over-union ratio between the laser 3D projection frame and each of the monocular 3D projection frames based on the size information of the monocular 3D projection frame and the laser 3D projection frame. The size information is calculated through the projection information of the monocular 3D target and the laser 3D target.

[0194] The data selection sub-unit is used to select the cross-union ratio with the largest value in the calculation results as the target cross-union ratio, forming the first BEV feature map.

[0195] And / or, the third acquisition submodule further includes:

[0196] The feature map acquisition unit is used to take the laser 3D probability distribution map obtained by the lidar after performing target detection in the 3D space as the second BEV feature map.

[0197] And / or, the third acquisition submodule further includes:

[0198] The target projection unit is used to project the monocular 3D target onto a new BEV feature map to obtain the position information of each monocular 3D target on the new BEV feature map.

[0199] The assignment unit is used to assign the confidence level of the monocular 3D target to the grid point corresponding to the position information on the new BEV feature map to form a third BEV feature map. The confidence level of the monocular 3D target is determined by the information in the monocular 3D target queue.

[0200] And / or, the network prediction module further includes:

[0201] The preprocessing submodule is used to perform dimensionality upscaling on the multimodal feature map and assign channel weights to features in the multimodal feature map through the SE attention mechanism.

[0202] The network prediction submodule is used to calculate the confidence level of the multimodal feature map based on the channel weights and the convolutional network.

[0203] The filtering submodule is used to filter and detect the calculated results based on the confidence threshold, obtain the 3D targets corresponding to the filtered confidence levels, and form a 3D target queue.

[0204] The specific implementation of the 3D target detection device of this application is basically the same as the embodiments of the above-described 3D target detection method, and will not be repeated here.

[0205] This application provides a computer-readable storage medium that stores one or more programs, which can be executed by one or more processors to implement the steps of the 3D target detection method described in any of the above claims.

[0206] The specific implementation of the storage medium in this application is basically the same as the embodiments of the above-described 3D target detection method, and will not be repeated here.

[0207] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the 3D target detection method described above.

[0208] The specific implementation of the computer program product of this application is basically the same as the embodiments of the above-described 3D target detection method, and will not be repeated here.

[0209] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0210] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0211] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of a software plus hardware platform, or by hardware, but in many cases the former is a better implementation. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0212] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A 3D target detection method, characterized in that, The method includes: Acquire a monocular 3D target queue after the camera performs target detection in 3D space, and a laser 3D target queue and corresponding laser 3D information after the lidar performs target detection in the 3D space. The laser 3D information includes a laser 3D probability distribution map and a laser 3D target information map. All laser 3D targets in the laser 3D target queue and all monocular 3D targets in the monocular 3D target queue are projected onto the image to obtain laser 3D projection frames and monocular 3D projection frames. A matching relationship is established between the laser 3D projection frames and the monocular 3D projection frames. Combined with the 3D target information in the laser 3D target information map and the information of the monocular 3D target queue, a multimodal feature map is obtained. The step of obtaining the laser 3D target information from the laser 3D target queue and the monocular 3D target information from the monocular 3D target queue by projecting them onto an image, obtaining laser 3D projection frames and monocular 3D projection frames, establishing a matching relationship between the laser 3D projection frames and monocular 3D projection frames, and combining the 3D target information from the laser 3D target information map and the information from the monocular 3D target queue to obtain a multimodal feature map includes: The laser 3D target queue and the monocular 3D target queue are projected onto an image to obtain the laser 3D projection frame and the monocular 3D projection frame. Calculate the intersection-union ratio between the laser 3D projection frame and the monocular 3D projection frame to form the first BEV feature map; The second BEV feature map is obtained by forming the confidence level of the laser 3D target; The confidence level of the monocular 3D target is obtained to form a third BEV feature map, and the confidence level of the monocular 3D target is determined by the information in the monocular 3D target queue. Based on the 3D target information of the laser 3D target in the laser 3D target information map, the distance value of the laser 3D target is calculated and normalized to obtain the fourth BEV feature map. By stitching together the four BEV feature maps, a multimodal feature map is obtained; The multimodal feature map is calculated using a convolutional network, and the calculated results are filtered and detected based on a confidence threshold to obtain a 3D target queue.

2. The 3D target detection method as described in claim 1, characterized in that, Acquire the laser 3D target queue and corresponding laser 3D information after the laser radar performs target detection in the 3D space, including a laser 3D probability distribution map and a laser 3D target information map, including: Acquire laser point cloud data for target detection in 3D space using lidar; Based on the CenterPoint framework, the VoxelNet network is used as the backbone network, and the CenterHead is used as the detection head to calculate the laser point cloud data to obtain the laser 3D target queue and the corresponding laser 3D information, including the laser 3D probability distribution map and the laser 3D target information map. The acquisition of the monocular 3D target queue after target detection in the 3D space by the camera includes: A monocular 3D target queue is obtained based on the Smoke algorithm. Each monocular 3D target queue contains confidence and size coordinate information.

3. The 3D target detection method as described in claim 1, characterized in that, The calculation of the intersection-union ratio between the laser 3D projection frame and the monocular 3D projection frame to form the first BEV feature map includes: The laser 3D projection frames corresponding to the queue formed by the laser 3D targets are traversed, and the following steps are performed for each traversed laser 3D projection frame: Based on the size information of the monocular 3D projection frame and the laser 3D projection frame, the intersection-over-union ratio between the laser 3D projection frame and each of the monocular 3D projection frames is calculated. The size information is obtained by calculating the projection information of the monocular 3D target and the laser 3D target. The cross-union ratio with the largest value in the calculation results is selected as the target cross-union ratio, forming the first BEV feature map.

4. The 3D target detection method as described in claim 1, characterized in that, The second BEV feature map formed by obtaining the confidence level of the laser 3D target includes: The laser 3D probability distribution map obtained after the lidar performs target detection in the 3D space is used as the second BEV feature map.

5. The 3D target detection method as described in claim 1, characterized in that, The process of obtaining the confidence level of the monocular 3D target to form the third BEV feature map includes: The monocular 3D target is projected onto the new BEV feature map to obtain the position information of each monocular 3D target on the new BEV feature map. The confidence level of the monocular 3D target is assigned to the grid point corresponding to the position information on the new BEV feature map to form a third BEV feature map. The confidence level of the monocular 3D target is determined by the information in the monocular 3D target queue.

6. The 3D target detection method as described in claim 1, characterized in that, The step of calculating the multimodal feature map using a convolutional network and filtering the calculated results based on a confidence threshold to obtain a 3D target queue includes: The multimodal feature map is subjected to dimensionality increase processing, and channel weights are assigned to features in the multimodal feature map through the SE attention mechanism; The confidence level of the multimodal feature map is calculated based on the channel weights and the convolutional network. The calculated results are filtered and detected based on the confidence threshold to obtain the 3D targets corresponding to the filtered confidence levels, thus forming a 3D target queue.

7. A 3D target detection device, characterized in that, The device includes: The target queue generation module is used to acquire the monocular 3D target queue after the camera performs target detection in the 3D space, and the laser 3D target queue and corresponding laser 3D information after the lidar performs target detection in the 3D space. The laser 3D information includes a laser 3D probability distribution map and a laser 3D target information map. The feature extraction module is used to obtain the laser 3D target in the laser 3D target queue and the monocular 3D target in the monocular 3D target queue, respectively projected onto the image to obtain the laser 3D projection frame and the monocular 3D projection frame, establish the matching relationship between the laser 3D projection frame and the monocular 3D projection frame, and combine the 3D target information in the laser 3D target information map and the information of the monocular 3D target queue to obtain a multimodal feature map; The network prediction module is used to calculate the multimodal feature map based on the convolutional network, and filter and detect the calculated results based on the confidence threshold to obtain a 3D target queue. The feature extraction module further includes: The second acquisition submodule is used to acquire all laser 3D targets in the laser 3D target queue and all monocular 3D targets in the monocular 3D target queue projected onto the image to obtain laser 3D projection frames and monocular 3D projection frames. The third calculation submodule is used to calculate the intersection-union ratio between the laser 3D projection frame and the monocular 3D projection frame to form the first BEV feature map. The third acquisition submodule is used to acquire the second BEV feature map formed by the confidence of the laser 3D target; The fourth acquisition submodule is used to acquire the confidence level of the monocular 3D target to form the third BEV feature map. The confidence level of the monocular 3D target is determined by the information in the monocular 3D target queue. The fourth calculation submodule is used to calculate the distance value of the laser 3D target based on the 3D target information of the laser 3D target in the laser 3D target information map, and normalize it to obtain the fourth BEV feature map. The feature extraction submodule is used to obtain a multimodal feature map by stitching together the four BEV feature maps.

8. The 3D target detection device as described in claim 7, characterized in that, The target queue generation module also includes: The first acquisition submodule is used to acquire laser point cloud data for target detection in 3D space by the lidar; The first calculation submodule is used to calculate the laser point cloud data based on the CenterPoint framework using the VoxelNet network as the backbone network and the CenterHead as the detection head to obtain the laser 3D target queue and the corresponding laser 3D information, including the laser 3D probability distribution map and the laser 3D target information map. And / or, the target queue generation module further includes: The second calculation submodule is used to obtain a monocular 3D target queue based on the Smoke algorithm. Each monocular 3D target queue contains confidence and size coordinate information. And / or, the third computing submodule further includes: The traversal unit is used to traverse the laser 3D projection frames corresponding to the queue formed by the laser 3D targets, and to perform the following steps for each laser 3D projection frame traversed: The data calculation subunit is used to calculate the intersection-over-union ratio between the laser 3D projection frame and each of the monocular 3D projection frames based on the size information of the monocular 3D projection frame and the laser 3D projection frame. The size information is calculated through the projection information of the monocular 3D target and the laser 3D target. The data selection subunit is used to select the cross-union ratio with the largest value in the calculation results as the target cross-union ratio, forming the first BEV feature map; And / or, the third acquisition submodule further includes: The feature map acquisition unit is used to take the laser 3D probability distribution map obtained by the lidar after performing target detection in the 3D space as the second BEV feature map. And / or, the third acquisition submodule further includes: The target projection unit is used to project the monocular 3D target onto a new BEV feature map to obtain the position information of each monocular 3D target on the new BEV feature map. The assignment unit is used to assign the confidence of the monocular 3D target to the grid point corresponding to the position information on the new BEV feature map to form a third BEV feature map. The confidence of the monocular 3D target is determined by the information in the monocular 3D target queue. And / or, the network prediction module further includes: The preprocessing submodule is used to perform dimensionality upscaling on the multimodal feature map and assign channel weights to features in the multimodal feature map through the SE attention mechanism. The network prediction submodule is used to calculate the confidence level of the multimodal feature map based on the channel weights and the convolutional network. The filtering submodule is used to filter and detect the calculated results based on the confidence threshold, obtain the 3D targets corresponding to the filtered confidence levels, and form a 3D target queue.

9. A 3D target detection device, characterized in that, The 3D target detection device includes a memory, a processor, and a 3D target detection program stored in the memory and executable on the processor. When the processor executes the 3D target detection program, it implements the steps of the 3D target detection method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a 3D target detection program, which, when executed by a processor, implements the steps of the 3D target detection method as described in any one of claims 1 to 6.

Images