A multi-modal three-dimensional object detection method and system based on laser radar guidance

Abstract

The application discloses a kind of multi-modal three-dimensional target detection method and system based on laser radar guidance.The laser radar point cloud and multi-view camera image of the same time are acquired, the multi-level depth enhancement strategy of laser radar guidance is introduced in the camera bird's eye view generation stage, the multi-level depth information including geometric depth feature and semantic depth feature is constructed, and the reliability of camera depth estimation is improved by adaptive fusion;In the feature fusion stage, a dynamic scale attention mechanism is used, and the fusion weight of camera and laser radar bird's eye view features is dynamically adjusted according to the target scale to take into account the feature advantages of targets of different scales;At the same time, in the time sequence aggregation stage, the multi-frame features are aligned and weighted aggregated by combining target motion information, and the motion ghosting is suppressed and the occlusion problem is alleviated.The application significantly improves the accuracy and stability of three-dimensional target detection in complex dynamic environment, and is suitable for automatic driving, intelligent robot and other environment perception applications.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent perception technology, and in particular to a multimodal 3D target detection method and system based on lidar-guided multi-level depth enhancement and dynamic fusion. Background Technology

[0002] 3D object detection, as a core technology in autonomous driving, intelligent robotics, and intelligent transportation systems, provides crucial support for environmental perception, path planning, and decision-making control by accurately locating and identifying various targets in the environment in three dimensions. With the rapid development of autonomous driving technology, higher demands are being placed on the accuracy, robustness, and real-time performance of 3D object detection.

[0003] Existing 3D target detection methods are mainly divided into two categories: single-sensor-based methods and multi-sensor fusion-based methods. The former includes visual methods that use only camera images or geometric methods that use only LiDAR point clouds; the latter attempts to integrate the advantages of multiple sensors in order to obtain more comprehensive environmental perception capabilities.

[0004] Camera sensors offer advantages such as high resolution, rich semantic information, and low cost, effectively recognizing objects with rich textures and distinctive appearance features. However, the two-dimensional images acquired by cameras lack accurate depth information. In complex environments such as low light, strong backlight, and adverse weather conditions, depth estimation results are easily affected, leading to a significant decrease in 3D positioning accuracy. In contrast, LiDAR provides accurate spatial geometry and distance information, unaffected by lighting conditions. However, its point cloud data is typically sparse, especially at long distances, and its semantic representation is limited, making it difficult to distinguish targets with similar geometric shapes but different categories.

[0005] To overcome the limitations of single sensors, existing technologies generally employ multimodal fusion strategies using cameras and LiDAR, and introduce Bird's Eye View (BEV) representation to achieve unified spatial modeling. In BEV space, data acquired by different sensors are projected onto a unified top-down coordinate system, facilitating feature fusion and target detection. However, existing multimodal fusion methods still face the following key challenges:

[0006] First, in the process of converting multi-view camera images into a bird's-eye view space, the accuracy of depth estimation becomes a bottleneck affecting overall performance. Inaccurate depth information not only weakens the camera's advantage in dealing with textured targets, especially in challenging environments such as low light and strong backlight, leading to a decrease in detection recall, but also causes spatial misalignment between camera bird's-eye view features and LiDAR bird's-eye view features, thereby reducing the subsequent fusion effect.

[0007] Secondly, due to the fundamental differences in the perception mechanisms of cameras and LiDAR, the bird's-eye view features generated by the two exhibit significant relative sensitivity differentiation across different target scales. Camera bird's-eye view features are advantageous in regions with high semantic density (such as small targets at a distance), while LiDAR bird's-eye view features perform better in regions sensitive to geometric accuracy (such as large targets at close range). Existing methods mostly employ direct stitching or simple weighted fusion approaches, which cannot adapt to the feature requirements of targets at different scales and are prone to mutual interference between cross-modal features, causing the respective advantageous features to be assimilated, especially resulting in poor performance in the detection of small targets at a distance and in complex occlusion scenarios.

[0008] Furthermore, while the introduction of temporal information helps alleviate target occlusion, in dynamic scenes, simply overlaying features from multiple frames of bird's-eye view can easily lead to spatiotemporal aliasing. Continuous displacement of moving targets can blur bird's-eye view features, and residual historical features can easily generate false responses, creating motion tailing, which seriously affects the stability and reliability of the detection system.

[0009] Although existing studies have attempted to improve the multimodal fusion effect through methods such as depth supervision and cross-modal alignment, most of these methods have failed to fully consider the multi-level characteristics of depth estimation, the dynamic changes of target scale, and the temporal consistency of target motion, making it difficult to achieve high-precision and high-stability 3D target detection in complex dynamic scenes.

[0010] Therefore, there is an urgent need for a multimodal 3D target detection method that can improve the reliability of camera depth, maintain the advantages of multi-scale target features, and adapt to dynamic scenes, so as to meet the perception needs of autonomous driving and intelligent robots in real and complex environments. Summary of the Invention

[0011] This invention addresses the shortcomings of existing technologies by providing a multimodal three-dimensional target detection method and system based on lidar guidance.

[0012] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0013] A multimodal 3D target detection method based on lidar guidance includes the following steps:

[0014] Step S1: Acquire multi-view camera image data and lidar point cloud data at the same time, wherein the multi-view camera is arranged around the carrier to obtain two-dimensional image information of the environment, and the lidar is used to acquire three-dimensional spatial geometric information of the environment.

[0015] Step S2: Construct multi-level depth information based on the LiDAR point cloud data, and use the multi-level depth information to guide the depth of camera image features, mapping the camera image features to the bird's-eye view space to generate camera bird's-eye view features; wherein, the multi-level depth information includes geometric depth features obtained by LiDAR point cloud projection and semantic depth features extracted from camera images, and combines them through an adaptive fusion method to enhance the depth continuity of the foreground target area and suppress background depth noise;

[0016] Step S3: Perform voxelization encoding and height-direction compression processing on the lidar point cloud data to generate lidar bird's-eye view features, so that they are located in the same bird's-eye view coordinate space as the camera bird's-eye view features.

[0017] Step S4: Based on the scale information of the target in the bird's-eye view space, perform deformable attention adaptive fusion of the camera bird's-eye view features and the lidar bird's-eye view features based on target scale perception, and dynamically adjust the fusion weights of the two modal features to maintain the fine-grained semantic information of small-scale targets and enhance the overall geometric structure expression of large-scale targets.

[0018] Step S5: Based on the fused bird's-eye view features at different times, spatially align the historical bird's-eye view features by combining the target's motion information, and use a deformable attention mechanism to weighted aggregate the features from multiple times to reduce feature misalignment and residue caused by target motion and occlusion.

[0019] Step S6: Perform 3D target detection based on the aggregated fused bird's-eye view features, and output the target's category information, spatial location, and 3D bounding box parameters.

[0020] Furthermore, the specific process of constructing multi-level depth information in step S2 includes:

[0021] A sparse depth map is generated by projecting the lidar point cloud, and a dense depth map is obtained by using a maximum block dilation strategy.

[0022] A multi-layer depth pyramid is constructed by multiple downsampling operations, including geometric depth features with edge enhancement and densification enhancement at different spatial resolutions.

[0023] Extract multi-level semantic features from camera images to generate deep semantic features;

[0024] A global-local attention fusion module is used to adaptively fuse sparse depth, dense depth, and multi-scale edge features to enhance the robustness of depth estimation.

[0025] Furthermore, the densification process of the maximum block expansion strategy is expressed as follows:

[0026]

[0027] in, This represents the depth map after densification. This represents a sparse depth map obtained by projecting a point cloud from a lidar radar. Represents image coordinates, Indicates Centered Local neighborhood, To expand the window size, This indicates the operation of retrieving the maximum value. Representing the neighborhood Local pixel coordinates within.

[0028] Furthermore, the adaptive fusion method employs a channel attention mechanism based on feature weights, normalizing the attention weights through the Sigmoid function to optimize the combination of deep features.

[0029] Furthermore, the dynamic scale attention adaptive fusion in step S4 employs a deformable attention mechanism, specifically including:

[0030] Channel alignment is performed on camera bird's-eye view features and LiDAR bird's-eye view features to generate shared query features;

[0031] Furthermore, the uniform size of the bird's-eye view features is ,in This represents the number of feature channels.

[0032] The feature fusion mechanism dynamically samples and fuses features across different scale regions using a multi-head deformable attention mechanism, and its computational representation is as follows:

[0033]

[0034] in To query features, Modal features, For attention weights, For learnable offsets, This represents the number of sampling points in the deformable attention mechanism. The linear transformation weight matrix represents the values. Indicates a reference point;

[0035] The final fusion features are generated by weighted summation to adaptively adjust the fusion weights of targets at different scales.

[0036] Furthermore, the time-series aggregation method in step S5 includes:

[0037] Build length is The historical feature queue is used to predict the two-dimensional velocity information of the target instance in the current frame. The length of the historical feature queue is a positive integer, preferably 2 to 5 frames. The two-dimensional velocity is predicted by introducing a velocity regression branch in the target detection head. The velocity is represented by the lateral velocity and longitudinal velocity in the bird's-eye view coordinate system.

[0038] Let the set of target instances detected in the current frame be... Each instance Including center coordinates and prediction speed Based on velocity, historical features are spatially aligned, and the offset of the bird's-eye view instance is calculated as follows: ,in For the first The speed of the target Indicates the historical frame time interval. Indicates a time index;

[0039] The dynamic scale attention mechanism aggregates features from multiple time points and outputs the aggregated bird's-eye view features.

[0040] Furthermore, the method is applicable to environmental perception scenarios for autonomous driving or intelligent robots, and is used to detect multiple categories of targets, including vehicles, pedestrians, and obstacles.

[0041] This invention also discloses a multimodal three-dimensional target detection system based on lidar guidance, used to perform the above-described method, comprising:

[0042] The data acquisition unit is used to acquire multi-view camera image data and lidar point cloud data at the same time.

[0043] The depth guidance unit constructs multi-level depth information based on LiDAR point cloud data and uses it to guide the depth of camera image features.

[0044] The feature generation unit generates camera bird's-eye view features and LiDAR bird's-eye view features, respectively;

[0045] The scale-adaptive fusion unit adaptively fuses the bird's-eye view features of the two modalities based on the target scale information.

[0046] The temporal aggregation unit performs spatial alignment and weighted aggregation of historical bird's-eye view features based on target motion information;

[0047] The target detection unit performs 3D target detection based on the features of the final fused bird's-eye view.

[0048] Furthermore, the system is integrated into an in-vehicle or robotic platform to process multimodal perception data in dynamic scenarios in real time.

[0049] Compared with the prior art, the advantages of the present invention are as follows:

[0050] 1. By employing a multi-level depth enhancement strategy guided by LiDAR, this invention effectively overcomes the problem of depth estimation noise in cameras under complex lighting conditions. This method enables cameras to maintain accurate depth estimation capabilities even in challenging environments such as low illumination and strong backlighting, significantly improving the geometric accuracy of camera bird's-eye view features and reducing detection failures caused by depth noise.

[0051] 2. The BEV dynamic scale attention fusion mechanism proposed in this invention can automatically adjust the fusion weights of LiDAR and camera features according to the characteristics of targets at different scales. This mechanism particularly improves the detection performance of small targets at different scales at long distances (such as buses and bicycles), significantly reduces the false negative rate, and performs exceptionally well in complex scenarios where large foreground targets occlude small targets.

[0052] 3. By using motion-guided temporal feature alignment and aggregation, this invention effectively solves the feature misalignment problem caused by target motion in dynamic scenes. This method significantly improves the continuity of the target trajectory, reduces motion tailing, and enhances the stability and reliability of the system in high-speed scenes and complex traffic environments.

[0053] 4. Comprehensive experimental verification shows that the method of this invention outperforms existing advanced methods in multiple evaluation indicators. In detection tasks involving various traffic participants (including vehicles, pedestrians, bicycles, etc.), this invention demonstrates a comprehensive performance improvement, especially under challenging scenarios with complex lighting and occlusion effects.

[0054] 5. This invention, through a multi-level depth enhancement and dynamic fusion mechanism, demonstrates excellent adaptability in complex scenarios such as nighttime, low-light conditions, long distances, and severe occlusion. Visualization results confirm that this method can reliably detect targets that traditional methods easily miss, especially demonstrating greater reliability in identifying small-scale targets and partially occluded targets.

[0055] 6. This invention fully considers practical deployment needs, maintaining high performance while also ensuring computational efficiency. This method is applicable to various 3D perception application scenarios such as autonomous vehicles and intelligent robots, providing more accurate and stable foundational support for environmental perception, path planning, and decision control, and has significant engineering application value. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a general framework diagram of the multimodal three-dimensional target detection method in this embodiment of the invention;

[0058] Figure 2 This is a schematic diagram of the depth guidance strategy structure based on multi-layer enhancement of laser point cloud in an embodiment of the present invention;

[0059] Figure 3 This is a structural diagram of enhanced deep feature extraction in an embodiment of the present invention;

[0060] Figure 4 This is a structural diagram of the multi-scale feature adaptive fusion module in an embodiment of the present invention;

[0061] Figure 5 This is a diagram of the dynamic scale attention fusion network architecture in an embodiment of the present invention;

[0062] Figure 6 This is a framework diagram of the motion-guided temporal aggregation method in an embodiment of the present invention;

[0063] Figure 7 This is a visual comparison of the method of this invention and the baseline method;

[0064] Figure 8 A comparison of camera depth-guided visualization results in this embodiment of the invention;

[0065] Figure 9 Visualization comparison of BEV features after fusion with baseline. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] This embodiment provides a multimodal 3D target detection method based on LiDAR guidance. The overall process includes three stages: dual-stream bird's-eye view feature generation, scale-adaptive feature fusion, and temporal information aggregation. First, features are extracted from both LiDAR point cloud data and multi-view camera image data, mapping the two modalities to a unified bird's-eye view space. Specifically, the camera branch incorporates multi-level depth information constructed from the LiDAR point cloud during bird's-eye view feature generation to guide camera depth estimation, reducing the impact of depth noise under low-light or weak-texture conditions, thus obtaining more accurate camera bird's-eye view features. The LiDAR branch generates LiDAR bird's-eye view features through voxelization encoding and high compression, and enhances the representation capability for distant small targets through multi-scale processing. Subsequently, within the bird's-eye view space, dynamic scale-adaptive fusion of the camera and LiDAR bird's-eye view features is performed according to target scale differences, allowing targets of different scales to fully utilize the advantages of their respective modalities. Finally, based on the motion information of the target, the fused bird's-eye view features at different times are spatially aligned, and the features of multiple frames are temporally aggregated. This process not only suppresses motion blur but also makes full use of historical information, thereby improving the accuracy and stability of 3D target detection in occluded and dynamic scenes.

[0068] like Figure 1 As shown, in this embodiment, the multimodal 3D target detection method based on lidar guidance may include at least the following steps:

[0069] Step S1: Acquire image data from multi-view cameras and point cloud data from LiDAR. The multi-view cameras include six cameras arranged around the vehicle, covering a 360° field of view around the vehicle, and each camera acquires RGB images; the LiDAR is an onboard 3D LiDAR used to acquire 3D point cloud data of the environment. Let the point cloud acquired at a certain moment be represented as: ,in These represent the spatial coordinates of the point cloud in the lidar coordinate system.

[0070] Step S2, as follows Figure 2 As shown, multi-level depth information is constructed based on the point cloud data, and camera features are extracted by combining the image data. The maximum block dilation strategy is used to densify the depth map and construct a multi-level depth pyramid. The technical effect is that it can simultaneously preserve the geometric details of small near-field targets and large far-field targets, significantly improving the completeness and multi-scale adaptability of depth estimation.

[0071] Step S21: For the numerous void regions present in the sparse depth map, this embodiment employs a maximum block dilation strategy for densification.

[0072]

[0073] in Indicates Centered Local neighborhood.

[0074] Based on this, a three-layer depth pyramid is constructed through multiple downsampling operations:

[0075]

[0076] The multi-level depth features are used to simultaneously describe the depth distribution of small near-field targets and far-field targets.

[0077] Step S22: Extract multi-level semantic features from the camera image and generate deep semantic features: , where H is the height dimension of the feature, W is the width dimension, and C is the number of channels.

[0078] Step S23, as follows Figure 3 As shown, the sparse depth of the point cloud Dense depth Multi-feature global-local attention with multi-scale edge features G Module fusion enhances deep multi-feature global-local attention ( The fusion module, with its detailed structure as follows: Figure 4 As shown, this module uses a channel attention mechanism to recalibrate the fused features. The module consists of two key parts: a local attention block and a global attention block.

[0079] Local attention blocks are used by Convolutional compression and activation operations are used to model the dependencies between channels. The specific workflow is as follows:

[0080]

[0081] The global attention module compresses the feature map into global features through global average pooling, and then... Convolution performs channel compression and restoration. The specific process is as follows:

[0082]

[0083] Local attention features and global attention features The weights are added together and then normalized to attention weights using the Sigmoid activation function.

[0084]

[0085] in This is the Sigmoid activation function, used to normalize attention features into weights.

[0086] The enhanced depth features are used to constrain the depth distribution of camera features, completing the mapping from image space to bird's-eye view space, and generating camera bird's-eye view features. .

[0087] Step S24: Perform depth guidance on the camera features to generate camera bird's-eye view features. The multi-level depth information consists of geometric depth features with different spatial resolutions and multi-level semantic depth features, which are fused in an adaptive manner to enhance the depth continuity of the foreground target and suppress background depth noise.

[0088] Step S25: Voxelization encoding and high compression are performed on the point cloud data to generate LiDAR bird's-eye view features. In this embodiment, both the camera bird's-eye view features and the lidar bird's-eye view features are unified to a single size:

[0089]

[0090] Step S3, as follows Figure 5 As shown, based on the target scale information, BEV dynamic scale attention fusion is performed on the camera bird's-eye view features and the lidar bird's-eye view features.

[0091] Step S31: First, perform channel alignment on the two modal features and generate shared query features:

[0092]

[0093] Step S32, then a deformable attention mechanism is introduced to dynamically sample and fuse features in regions of different scales:

[0094]

[0095] Parallel computation through multi-head attention mechanism:

[0096]

[0097] Step S33: Finally, the fused bird's-eye view features are generated. The fusion weights of camera features and lidar features are dynamically adjusted for targets of different scales to maintain the fine-grained semantic information of small-scale targets and the overall geometric structure of large-scale targets.

[0098]

[0099] Step S4: Based on the fused bird's-eye view features from different times, temporal alignment and aggregation are performed by combining target motion information. Spatial alignment of moving targets reduces feature misalignment and feature loss caused by target motion. The overall block diagram is as follows: Figure 6 .

[0100] Step S41, construct a length of Historical feature queue: The length of the historical feature queue is a positive integer, preferably 2 to 5 frames, and can be set according to computing resources and timing stability requirements in specific implementations. Predict the two-dimensional velocity of the target in the current frame: The two-dimensional velocity is predicted by introducing a velocity regression branch in the target detection head, and the velocity is represented by lateral and longitudinal velocities in a bird's-eye view coordinate system.

[0101] Step S42, based on the set of target instances detected in the current frame as follows: Each instance Including center coordinates and prediction speed Perform historical feature alignment:

[0102] Step S43, finally completes multi-time feature aggregation through dynamic scale attention:

[0103]

[0104] Step S5: Perform 3D target detection based on the features of the aggregated fused bird's-eye view, and output the target's category information, spatial location, and 3D bounding box parameters.

[0105] To verify the detection performance and stability of the proposed multimodal 3D target detection method based on bird's-eye view in complex traffic scenarios, this embodiment uses a publicly available autonomous driving multimodal dataset to verify the method.

[0106] The experimental dataset used in this embodiment is the nuScenes dataset. This dataset is a large-scale autonomous driving perception benchmark dataset, containing image data acquired by multi-view cameras and point cloud data acquired by LiDAR. It is divided into multiple driving scenarios, some of which are used for model training, and others for model validation and testing. The dataset achieves spatial alignment of multimodal data in a unified coordinate system through joint calibration of multi-view cameras and LiDAR, making it suitable for validating multimodal 3D object detection methods.

[0107] In terms of object detection evaluation, the nuScenes dataset defines a unified evaluation metric system, including mean accuracy (mAP) and nuScenes detection score (NDS). mAP is an metric obtained by averaging the detection accuracy of multiple object categories across multiple bird's-eye view center distance thresholds, used to measure the overall accuracy of the detection results. NDS is a comprehensive evaluation metric obtained by weighting multiple error metrics such as target position, size, orientation, velocity, and attributes based on mAP, used to reflect the overall performance of the 3D object detection method.

[0108] In this embodiment, the multimodal 3D target detection method based on bird's-eye view proposed in this invention is applied to the above dataset to detect multi-class targets in road scenes, and the detection results are compared and analyzed with a baseline model trained on the same dataset. Specific experimental results are shown in Table 1.

[0109]

[0110] Among them, Car, Truck, Bus, and Pedestrian represent the detection accuracy for different traffic participant categories.

[0111] Furthermore, to further verify the detection performance of the method of the present invention in complex scenarios, this embodiment presents a visual comparative analysis of the detection results of the method of the present invention and the baseline method BEVFusion, such as... Figure 7 As shown in the figure. The comparison results show that, compared with the baseline method, the present invention can obtain more stable and accurate 3D target detection results in complex traffic scenarios, and can simultaneously detect targets of different categories, scales and distances. Especially under conditions of distant small targets, severely occluded targets and low-light environments, the method of the present invention can still maintain a high target detection rate and detection accuracy, demonstrating the robustness and practicality of the present invention in complex dynamic scenarios.

[0112] The experimental results above demonstrate that the method of this invention, by introducing a multi-level depth guidance mechanism, an adaptive scale bird's-eye view feature fusion mechanism, and a temporal feature aggregation method based on motion information, can more fully integrate multimodal information from cameras and LiDAR in complex traffic scenarios, effectively improving the overall accuracy and stability of 3D target detection. This verifies the effectiveness and practicality of the method in real-world autonomous driving scenarios.

[0113] Furthermore, in order to analyze the contribution of each key technical module of the present invention to the detection performance, this embodiment conducted ablation experiments on the multi-level depth enhancement module, the dynamic scale attention fusion module, and the motion-guided temporal aggregation module.

[0114]

[0115] Experimental results show that introducing a multi-level depth enhancement module into the baseline model improves the model's NDS and mAP by 0.7% and 0.9%, respectively, indicating that LiDAR-guided camera depth estimation effectively improves the geometric consistency of camera bird's-eye view features. Further introducing a dynamic scale attention fusion module results in an additional 0.6% improvement in both mAP and NDS, validating the effectiveness of the scale-adaptive fusion mechanism in improving the detection performance of both distant small and large-scale targets. When combined with a motion-guided temporal aggregation module, the model's mAP and NDS further improve by 0.5% and 0.4%, respectively, effectively mitigating feature misalignment issues caused by target motion and occlusion. The synergistic effect of these three modules enables the model to maintain higher detection stability and continuity in dynamic and complex scenes.

[0116] Furthermore, through visualization of the test results Figure 8 , Figure 9 Comparative analysis reveals that in complex scenarios such as nighttime, low-light conditions, long distances, and severe occlusion, the method of this invention can detect targets missed by baseline methods, especially demonstrating greater stability in identifying small-scale targets and partially occluded targets. This indicates that the present invention, through multi-level depth enhancement, scale-adaptive fusion, and temporal feature alignment, effectively improves the robustness and practicality of multimodal 3D target detection in real-world road environments.

[0117] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0118] In another embodiment, a multimodal 3D target detection system guided by lidar is provided, which corresponds one-to-one with the modal 3D target detection methods described in the above embodiments. The system includes:

[0119] The data acquisition unit is used to acquire multi-view camera image data and lidar point cloud data at the same time.

[0120] The depth guidance unit constructs multi-level depth information based on LiDAR point cloud data and uses it to guide the depth of camera image features.

[0121] The feature generation unit generates camera bird's-eye view features and LiDAR bird's-eye view features, respectively;

[0122] The scale-adaptive fusion unit adaptively fuses the bird's-eye view features of the two modalities based on the target scale information.

[0123] The temporal aggregation unit performs spatial alignment and weighted aggregation of historical bird's-eye view features based on target motion information;

[0124] The target detection unit performs 3D target detection based on the features of the final fused bird's-eye view.

[0125] The system can be integrated into vehicle or robot platforms to process multimodal perception data in dynamic scenarios in real time.

[0126] For specific limitations regarding the multimodal 3D target detection system, please refer to the limitations of the multimodal 3D target detection method above, which will not be repeated here. Each module in the above system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0127] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0129] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multimodal three-dimensional target detection method based on lidar guidance, characterized in that... Includes the following steps: Step S1: Acquire multi-view camera image data and lidar point cloud data at the same time, wherein the multi-view camera is arranged around the carrier to obtain two-dimensional image information of the environment, and the lidar is used to acquire three-dimensional spatial geometric information of the environment. Step S2: Construct multi-level depth information based on the LiDAR point cloud data, and use the multi-level depth information to guide the depth of camera image features, mapping the camera image features to the bird's-eye view space to generate camera bird's-eye view features; wherein, the multi-level depth information includes geometric depth features obtained by LiDAR point cloud projection and semantic depth features extracted from camera images, and combines them through an adaptive fusion method to enhance the depth continuity of the foreground target area and suppress background depth noise; Step S3: Perform voxelization encoding and height-direction compression processing on the lidar point cloud data to generate lidar bird's-eye view features, so that they are located in the same bird's-eye view coordinate space as the camera bird's-eye view features. Step S4: Based on the scale information of the target in the bird's-eye view space, perform deformable attention adaptive fusion of the camera bird's-eye view features and the lidar bird's-eye view features based on target scale perception, and dynamically adjust the fusion weights of the two modal features to maintain the fine-grained semantic information of small-scale targets and enhance the overall geometric structure expression of large-scale targets. Step S5: Based on the fused bird's-eye view features at different times, spatially align the historical bird's-eye view features by combining the target's motion information, and use a deformable attention mechanism to weighted aggregate the features from multiple times to reduce feature misalignment and feature loss caused by target motion and occlusion. Step S6: Perform 3D target detection based on the aggregated fused bird's-eye view features, and output the target's category information, spatial location, and 3D bounding box parameters.

2. The method according to claim 1, characterized in that, The specific process of constructing multi-level depth information in step S2 includes: A sparse depth map is generated by projecting the lidar point cloud, and a dense depth map is obtained by using a maximum block dilation strategy. A multi-layer depth pyramid is constructed by multiple downsampling operations, including geometric depth features with edge enhancement and densification enhancement at different spatial resolutions. Extract multi-level semantic features from camera images to generate deep semantic features; A global-local attention fusion module is used to adaptively fuse sparse depth, dense depth, and multi-scale edge features to enhance the robustness of depth estimation.

3. The method according to claim 1, characterized in that: The densification process of the maximum block expansion strategy is expressed as follows: ， in, This represents the depth map after densification. This represents a sparse depth map obtained by projecting a point cloud from a lidar radar. Represents image coordinates, Indicates Centered Local neighborhood, To expand the window size, This indicates the operation of retrieving the maximum value. Representing the neighborhood Local pixel coordinates within.

4. The method according to claim 1, characterized in that, The adaptive fusion method adopts a channel attention mechanism based on feature weights, and normalizes the attention weights through the Sigmoid function to optimize the combination of deep features.

5. The method according to claim 1, characterized in that, The dynamic scale attention adaptive fusion in step S4 employs a deformable attention mechanism, specifically including: Channel alignment is performed on camera bird's-eye view features and LiDAR bird's-eye view features to generate shared query features; The feature fusion mechanism dynamically samples and fuses features across different scale regions using a multi-head deformable attention mechanism, and its computational representation is as follows: ， in To query features, Modal features, For attention weights, For learnable offsets, This represents the number of sampling points in the deformable attention mechanism. The linear transformation weight matrix represents the values. Indicates a reference point; The final fusion features are generated by weighted summation to adaptively adjust the fusion weights of targets at different scales.

6. The method according to claim 1, characterized in that, The time-series aggregation method in step S5 includes: Build length is The historical feature queue decouples each target instance from the background and predicts the two-dimensional velocity information of each target instance in the current frame; wherein the length of the historical feature queue is a positive integer; A velocity regression branch is introduced for velocity prediction, and historical features are spatially aligned based on velocity. The offset is calculated as follows: ,in For the first The speed of the target Indicates time interval, Indicates a time index; The dynamic scale attention mechanism aggregates features from multiple time points and outputs the aggregated bird's-eye view features.

7. The method according to claim 1, characterized in that, The method is applicable to environmental perception scenarios for autonomous driving or intelligent robots, and is used to detect multiple types of targets, including vehicles, pedestrians, and obstacles.

8. The method according to claim 1, characterized in that, The uniform size of the bird's-eye view features is ,in This represents the number of feature channels.

9. A multimodal three-dimensional target detection system based on lidar guidance, used to execute the method according to any one of claims 1 to 8, characterized in that... include: The data acquisition unit is used to acquire multi-view camera image data and lidar point cloud data at the same time. The depth guidance unit constructs multi-level depth information based on LiDAR point cloud data and uses it to guide the depth of camera image features. The feature generation unit generates camera bird's-eye view features and LiDAR bird's-eye view features, respectively; The scale-adaptive fusion unit adaptively fuses the bird's-eye view features of the two modalities based on the target scale information. The temporal aggregation unit performs spatial alignment and weighted aggregation of historical bird's-eye view features based on target motion information; The target detection unit performs 3D target detection based on the features of the final fused bird's-eye view.

10. The system according to claim 9, characterized in that, The system is integrated into a vehicle or robot platform to process multimodal perception data in dynamic scenarios in real time.

Images