Three-dimensional target detection method based on three-dimensional point cloud and multi-view image fusion

Abstract

The present application relates to the field of automatic driving and computer vision technology, and proposes a three-dimensional target detection method based on three-dimensional point cloud and multi-view image fusion, specifically, the present application constructs a hybrid feature interaction mechanism by fusing lidar and camera data, and combines a hierarchical loss optimization strategy, which significantly improves the detection accuracy and robustness in complex urban scenes.In methodology, the present application makes full use of the feature information of the camera branch and the lidar branch, and provides an effective query initialization means.In addition, the present application designs a QFD through an adjustable weight coefficient, and innovatively dynamically fuses historical query features and initial queries.The present application effectively solves the feature degradation problem of the traditional cascade decoder through cross-layer fusion of multi-modal features.Experiments prove that the present method has reached advanced performance on the nuScenes benchmark dataset.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving and computer vision technology, and more specifically, to a method for three-dimensional target detection based on the fusion of three-dimensional point clouds and multi-view images. Background Technology

[0002] 3D object detection is one of the key technologies for achieving vehicle environmental perception in autonomous driving. It involves identifying and locating other vehicles, pedestrians, bicycles, and other surrounding objects from sensor data, and estimating their 3D position, size, and orientation. This information is crucial for autonomous vehicles to plan safe routes and make decisions. Autonomous driving systems typically rely on multiple sensors to collect environmental data, including LiDAR, cameras, and radar. LiDAR provides high-precision 3D point cloud data, accurately capturing the shape and distance of surrounding objects. Cameras provide rich visual information, such as color, texture, and scene layout, which helps identify the category and state of objects. LiDAR measures the distance, speed, and orientation of objects, making it particularly suitable for adverse weather conditions.

[0003] Currently, 3D object detection methods can be divided into two main categories: single-sensor methods, which typically rely on a single modality, such as LiDAR or a camera alone. For example, LiDAR-based methods like PointPillars and VoxelNet directly process point clouds for 3D detection. Camera-based methods, such as MV3D and MonoGRNet, utilize camera images to estimate the position of 3D objects through geometric transformations. Multi-sensor fusion methods, such as FusionNet and AVOD, integrate LiDAR and cameras to combine their complementary advantages, thereby improving detection accuracy and robustness for more comprehensive environmental perception. However, these methods still face several major challenges: (1) Difficulty and inefficiency in multimodal information fusion Traditional multimodal fusion methods (such as early or late fusion) struggle to fully leverage the complementary advantages between LiDAR's precise geometric information and the camera's rich semantic information. Simple feature stitching or result-level fusion may introduce noise and fail to achieve deep, adaptive interaction at the feature level, resulting in poor fusion performance.

[0004] (2) The query initialization efficiency and effect are not good in large-scale scenarios. In large-scale 3D object detection based on Transformer, using random initialization of object queries can lead to slow model convergence, unstable training, susceptibility to local optima, and high training costs. This is because the distribution of random queries differs greatly from that of real targets in a vast space, requiring a long learning process for the model to explore the possible regions where targets may appear.

[0005] (3) Insufficient robustness to inherent defects in sensors When relying on images for 3D detection, limitations exist due to the lack and inaccuracy of depth information, as well as distortions and errors arising from projecting 3D space onto 2D images. While LiDAR modalities offer precise geometric information, they are inferior to images in object classification and semantic understanding.

[0006] (4) Feature degradation or forgetting during decoder iteration In the multi-layer iteration process of cascaded decoders, as the network deepens, the initial strong information from the sensors may be gradually diluted or forgotten, causing the optimization process to deviate from the correct direction, which is the so-called "feature degradation" problem.

[0007] (5) Accuracy and robustness challenges in complex urban scenarios Autonomous driving in urban scenarios is characterized by a wide variety of target types, high density, severe occlusion, and large variations in lighting and weather, which places extremely high demands on the accuracy and robustness of detection models.

[0008] In summary, there is an urgent need to propose a novel 3D target detection method to address the problems of difficulty in multimodal information fusion, low query initialization efficiency, feature degradation, and insufficient detection accuracy in complex scenarios in the existing technologies. Summary of the Invention

[0009] This invention proposes a three-dimensional target detection method based on the fusion of three-dimensional point clouds and multi-view images. By fusing the multimodal features of lidar point clouds and multi-view images, and introducing a weighted update mechanism based on heatmap object query initialization and query fusion decoder, the accuracy and robustness of three-dimensional target detection are significantly improved.

[0010] To achieve the above objectives, this invention proposes a 3D target detection method based on the fusion of 3D point clouds and multi-view images, comprising: Acquire 3D point cloud data collected by lidar and 2D image data collected by multi-view cameras; Feature extraction is performed on the point cloud data and image data through the lidar branch and camera branch respectively to obtain lidar BEV features and image BEV features; Heatmaps are generated based on the BEV features of the LiDAR and the BEV features of the image, and object queries are initialized based on the heatmap scores. The initialized object query is input into the query fusion decoder. In each layer of the decoder, the query output of the previous layer is fused with the initial query, and the query is updated through a multi-head self-attention mechanism. The updated query is classified and three-dimensional bounding box regression is performed by the probe to output three-dimensional target detection results.

[0011] Furthermore, the process of extracting features from the point cloud data and image data through the LiDAR branch and camera branch respectively includes: For the camera branch, a 3D sampling grid is formed based on the input image features. All sampling positions within the 3D sampling grid are projected onto the image using known camera parameters. Voxel features are obtained through bilinear sampling and cross-view summation. The voxel features are then compressed by cascading along the channel dimension. Convolutional layers are used to encode the features to obtain the image BEV features, and a heatmap is generated using ResNet-18. For the LiDAR branch, the input point cloud is processed by VoxelNet to extract 3D voxel features, and the output is the LiDAR BEV feature.

[0012] Furthermore, the initialization of object queries based on heatmap scores includes: A heatmap is generated based on the BEV characteristics of the LiDAR, and the top K positions with the highest centrality scores are selected as the initial query positions. The BEV features at the corresponding locations are concatenated with the heatmap scores to form the initial query features.

[0013] Furthermore, the query fusion decoder consists of multiple cascaded multimodal prediction interaction layers. Before input, each decoder layer fuses the query output of the previous layer with the initial query through a weighted fusion module. The fusion formula is as follows:

[0014] in, δ This is an adjustable weighting coefficient used to balance the contributions of initial queries and historical queries. q init This is the initial query, derived from the heatmap initialization. `l` is the index of the current layer, where `l∈{1,2,...,L}`. q prev This is the output of the query from the upper level.

[0015] Furthermore, the query fusion decoder includes a storage unit for caching the query output of the previous layer so that it can be fused and updated in the next layer decoder.

[0016] Furthermore, the method also includes constructing a loss function, including heatmap loss, classification loss, and 3D bounding box regression loss, with the total loss function being the sum of the three, and matching the prediction with the real target using the Hungarian algorithm.

[0017] Furthermore, the heatmap loss is calculated using the Gaussian focal loss function, the classification loss is calculated using FocalLoss, and the regression loss is calculated using L1Loss.

[0018] Furthermore, the method employs a two-stage training strategy during training, using the AdamW optimizer and a single-cycle learning rate strategy, and freezes pre-trained weights in the image branch to save computational resources.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a multimodal 3D object detection framework for autonomous driving scenarios. By fusing LiDAR and camera data to construct a hybrid feature interaction mechanism and combining it with a hierarchical loss optimization strategy, the detection accuracy and robustness in complex urban scenes are significantly improved. Methodologically, it fully utilizes the feature information from the camera and LiDAR branches, providing an effective query initialization method. The designed QFD innovatively and dynamically fuses historical query features and the initial query through adjustable weight coefficients. Through cross-layer fusion of multimodal features, the feature degradation problem of traditional cascaded decoders is effectively solved. Experiments demonstrate that the proposed method achieves state-of-the-art performance on the nuScenes benchmark dataset. Attached Figure Description

[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a diagram illustrating the overall architecture of the 3D target detection method based on 3D point cloud and multi-view image fusion of the present invention. Figure 2 This is a schematic diagram illustrating the principle and structure of the query fusion decoder in this embodiment of the invention; Figure 3 This is a schematic diagram of the Renault Zoe vehicle sensor distribution in the nuScenes dataset in this embodiment of the invention; Figure 4 This is a visualization of the model proposed in this invention on the nuScenes validation set. Detailed Implementation

[0021] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] The overall framework of the 3D target detection method based on multimodal fusion of 3D point cloud and 2D multi-view camera images proposed in this embodiment is as follows: Figure 1 As shown. Unlike all existing technologies, the method proposed in this embodiment learns two representations of the 3D LiDAR and 2D image modalities respectively, and performs multimodal interaction through model encoding and decoding. The framework consists of four parts: a LiDAR branch, a camera branch, a fusion decoder, and a probe.

[0023] The sparse framework for multimodal 3D object detection initializes features extracted from the LiDAR and camera branches as heatmaps, and inputs the resulting initial query into the QFD (Query Fusion Decoder). The query is then fused and updated in each decoder layer of the QFD module to ultimately generate predictions.

[0024] Heatmap-based query initialization In object detection, especially in Transformer-based architectures, object queries are a set of learnable embedding vectors that serve as prior information for the model. These vectors are used to extract relevant object information from feature maps to predict attributes such as the object's location and category in the image. Random object query initialization involves setting initial values ​​at the start of training, such as uniform or normal distributions. During model training, these randomly initialized object queries are continuously adjusted and optimized, and the model learns the correspondence between these queries and objects in the image. However, random object query initialization is unsuitable for large-scale scenarios like autonomous driving. First, it has a slow convergence speed because the initial values ​​differ significantly from the true object feature distribution, requiring numerous iterations. Second, it has unstable performance; different random initializations can cause the model to converge to different local optima, leading to performance fluctuations. Third, it is prone to getting trapped in local optima; improper initial value selection can trap the model, making it difficult to find the global optimum. Fourth, it increases training costs, requiring more training epochs and computational resources to achieve satisfactory performance. Preliminary integration and processing of feature information before initialization can guide query initialization to focus on regions with a high probability of object occurrence. Detr4d proposes the concept of heatmap-based query initialization. Before initializing object queries, the image is first processed to obtain a heatmap that represents the objectivity of the bird's-eye view, generated by grid-based feature sampling.

[0025] In this embodiment, when processing information from both image and LiDAR modalities, the data is first divided into a LiDAR branch and an image branch. Features are extracted through a backbone network, and preliminary interaction of the information is performed before generating the heatmap. This leverages the advantages of both modalities. For example, semantic information from the image is used to assist LiDAR point cloud target classification, while depth information from the LiDAR is used to enhance the 3D localization of targets in the image, facilitating subsequent filtering and fusion. Following the Deep Interaction approach, the two mode-specific scene representations extracted independently from the LiDAR and image backbones are input to the encoding layer for information interaction. The outputs remain two independent modalities, preserving the mode-specific features of both.

[0026] This embodiment proposes using a heatmap-based query initialization process in both the image branch and the liDAR branch to guide query initialization, focusing on areas with a high probability of object occurrence. For the image branch, based on the features of the input six-view image, a 3D sampling grid corresponding to the scene range is first formed. Then, using the known camera parameters, all sampling locations within the grid are projected onto the image to obtain Gp. Voxel features Fv are then obtained through bilinear sampling and cross-view summation. Fv is then compressed by cascading along the channel dimensions, and finally, convolutional layers are used to encode the features to obtain a flattened BEV representation. Finally, a heatmap M is generated using ResNet-18.

[0027] (1) σ is the sigmoid activation function, and Fbev is obtained by performing the above sampling, projection, cross-view summation, and cascade compression on the six-view image.

[0028] For the LiDAR branch, the input point cloud is processed by VoxelNet to extract 3D voxel features, and the output is a bird's-eye view (BEV) feature map (Flidar∈RH×W×C). Similarly, a heatmap N is generated from the BEV features. This heatmap is then fused with the features extracted by ResNet, so that the features carry information about the target location. In this embodiment, the top K objects with the highest centrality scores are retained to represent the corresponding instances, and the positions of these K objects are used as the initial positions for object queries.

[0029] (2) (3) (4) The initial query features are formed by concatenating the BEV features and heatmap scores at the corresponding locations. Using this method, both the LiDAR branch and the camera branch can independently initialize the query. In 3D object detection, initializing the query using the LiDAR branch heatmap is superior to initializing it using the camera branch heatmap. Using the LiDAR branch can reduce nonlinear distortion introduced during camera projection and depth errors from reprojecting the 3D mesh onto the 2D image plane, thereby improving model accuracy.

[0030] Understandably, this embodiment proposes a dedicated Query Fusion Decoder (QFD) that allows for continuous and deep interaction between features from both LiDAR and image modalities during the decoding phase through a hierarchical iterative approach. It doesn't simply merge features; instead, it enables object queries to simultaneously extract information from both modalities (image semantics and LiDAR geometry) at each layer, achieving dynamic and refined fusion. Furthermore, this embodiment introduces heatmap-based query initialization. Heatmaps representing the probabilities of target occurrence are generated in both the LiDAR and image branches, and the region with the highest probability is selected to initialize the object query. This essentially provides the model with powerful "prior knowledge," focusing the search scope on high-value regions and significantly improving training efficiency and model performance.

[0031] Query the weighted fusion mechanism and update strategy of the fusion decoder The query fusion decoder is the core component of this research's multimodal 3D object detection framework. Its core design philosophy lies in achieving progressive refinement of object querying and bounding box prediction through layered iterative multimodal prediction interactions. This overcomes the limitations of single-modal feature representation, fully leveraging the complementary value of LiDAR's geometric localization advantages and image semantic recognition capabilities, providing crucial support for high-precision 3D object detection. The decoder employs a multi-layered cascaded structure, consisting of multiple functionally consistent multimodal prediction interaction layers stacked together. Through iterative optimization of target feature representation and spatial localization, it ensures the accuracy and robustness of the detection results.

[0032] In the hierarchical iteration process of the decoder, the core logic is to enhance object query and bounding box prediction by utilizing the information from the two LiDAR modalities and the RoI features of the corresponding image regions.

[0033] The basic idea of ​​the query fusion decoder is to re-fuse the semantic information of each object query. Since the geometric information of the LiDAR is dominant when the query is initialized, when the query is updated in this module to improve the quality, it should also relatively update the original geometric information provided by the LiDAR while fusing more semantic information provided by the images.

[0034] like Figure 2As shown, this embodiment proposes a novel hybrid method for updating query features, similar to DETR4d's decoder directly performing query-image interaction, transforming 3D object detection into an ensemble prediction method. To facilitate inter-layer updates, this embodiment creates a storage unit to cache past object queries and query features, storing the object query output from the last Transformer layer of the previous layer. Before each iteration enters the next layer's decoder, the query output from the previous layer and the initial query are input into a weighted combination module. By adjusting the weights of the query output from the previous layer and the initial query—weights that also reflect the accuracy of radar branch initialization information and image branch initialization information—a fused query is obtained, which is then input into the multi-head self-attention layer of the next layer's decoder. The weighted combination module uses a weighted summation principle, assigning different weights to different queries before summing them.

[0035] (8) (9) (10) Qinit is the initial query, derived from the heatmap initialization. l is the index of the current layer, l∈{1,2,...,L}. Qprev is the query output from the previous layer, which is also the decoder's output at layer l. The closer δ is to 1, the more it reinforces the initial query, meaning it relies on the reliability of the sensor heatmap. The closer δ is to 0, the more it favors historical iteration results, meaning it relies on the decoder's optimization capabilities. In summary, this embodiment designs a weighted fusion update strategy. Before each layer of the decoder begins, the query output from the previous layer is weighted and combined with the initial query (using learnable weight coefficients δ). This mechanism allows the model to dynamically decide whether to rely more on the initial sensor information or the optimization results learned by the network during iteration, thereby effectively preserving valuable information from the data source and preventing feature degradation.

[0036] Loss function construction To provide supervision for the predicted heatmap, this embodiment generates the true heatmap Mgt by drawing a Gaussian distribution on the annotation bounding boxes with a fixed radius r. Then, the heatmap loss with Gaussian focal loss is calculated: (11) The loss function for each decoder layer needs to combine multiple losses from classification and 3D bounding box regression, and align the predicted values ​​with the ground truth using Hungarian matching. Object classification is optimized using focal loss, while 3D bounding box regression is optimized using L1 loss. The overall loss function is the sum of the classification loss, bounding box loss, and heatmap loss.

[0037] (12) Where: Lcls(l) is the classification loss of the l-th layer, Lbbox(l) is the bounding box regression loss of the l-th layer, and λ(l) is the layer weight coefficient. Training strategy The nuScenes dataset, developed by the Motional team, is a multimodal open-source dataset for autonomous driving, providing stereo perception data for high-density urban road scenes. This challenging 3D object detection dataset offers point clouds acquired using a 32-line LiDAR with 360-degree surround view and six images from multi-view cameras. Data alignment is achieved through a unified timestamp and spatial coordinate system. nuScenes captures approximately 1000 annotated driving scenes in 324 urban environments, with 1.4 million annotated 3D bounding boxes for objects from 10 different categories, providing a comprehensive dataset for algorithm development and evaluation. This experiment used official evaluation metrics, including mean average accuracy (mAP) and scene detection score (NDS). Data acquisition was conducted using two Renault Zoe vehicles with identical sensor layouts, driven in Boston and Singapore. For the placement of the LiDAR and image sensors, please refer to [reference needed]. Figure 3 .

[0038] There are six cameras (CAMs), located at the front, front right, front left, back, back right, and back left. There is one lidar (DAR) sensor, located on the roof.

[0039] Our implementation is based on the MMDetection3D framework. For the camera branch, we use a ResNet50 network as the backbone and initialize it with a MaskR-CNN instance segmentation network pre-trained on nuImage. We scale the input image to a resolution of 800×448 and freeze the weights of the image branch during training to save training time. For the LiDAR branch, this embodiment uses VoxelNet with voxel sizes of (0.075m, 0.075m, 0.2m). The detection range is set to XY axis [-54m, 54m] and Z axis [-5m, 3m]. For the decoder part, this embodiment uses 6 cascaded decoder layers. We set the number of queries to 200 for training and testing. In the experiments, PyTorch 1.9.0 and CUDA 11.1 were used, employing a two-stage training scheme called Transfusion. The AdamW optimizer and a single-cycle learning rate strategy with a learning rate of 0.0001 were used. The network was trained on four NVIDIA 3090 GPUs with a batch size of 2 for 10 epochs. During training, the Hungarian algorithm was used to match predicted and ground truth bounding boxes. Focal loss and L1 loss were also employed. ] They are used for classification and 3D bounding box regression, respectively.

[0040] Experimental verification Experiment 1: Overall Comparison This method was compared with other methods, including camera-based BEVFormer, LiDAR-based PointPillars, SECOND, and CenterPoint, and LiDAR-camera based CFF, mmFusion, UVTR, CMT, TransFusion, AutoAlign, AutoAlignV2, and FUTR3D. The results are shown in Table 1.

[0041] Table 1

[0042] As shown in Table 1, on the validation dataset, the method of this invention significantly improves the NDS metric compared to BEVFormer, which only uses image data. Specifically, there is a relative improvement of 19.9%. Compared to PointPillars, SECOND, and CenterPoint, the method of this invention achieves relative improvements of 10.3%, 8.6%, and 6.2% in the NDS metric, respectively. Compared to LiDAR-camera based methods such as CFF, mmFUSION, UVTR, AutoAlign, FUTR3D, CMT, TransFusion, and AutoAlignV2, the method of this invention shows relative improvements of 2.4%, 1.9%, 1.4%, 0.5%, 3.3%, 0.8%, 0.3%, and 0.4% in the NDS metric on the validation dataset, respectively. Furthermore, the method of this invention improves the mAP metric to varying degrees. Compared to CFF, mmFUSION, UVTR, AutoAlign, FUTR3D, CMT, TransFusion, and AutoAlignV2 using the same data, the method of this invention shows an improvement in mAP of 3.6%, 2.7%, 2.7%, 1.5%, 3.6%, 0.2%, 0.6%, and 1.0%, respectively.

[0043] Figure 4 This is a visualization of the model's validation results on the nuScenes validation set. The left image shows the Bev view of the 3D detection results. Different colored detection boxes represent different categories. The white area represents the point cloud acquired by the LiDAR sensor; the right image shows the detection results projected from six viewpoints for each sub-map. Figure 4 Experimental results on the nuScenes dataset demonstrate that the method of this invention effectively integrates information from two modalities through interactive fusion of Lidar and Camera data, thereby improving the performance of 3D object detection.

[0044] Experiment 2: Multimodal Fusion Decoder Layer Ablation Experiment The effectiveness of the predictive interaction design is evaluated by comparing the multimodal predictive interaction decoder layers in this embodiment. As shown in Table 2, increasing the number of decoder layers to 6 can continuously improve performance.

[0045] Table 2

[0046] Experiment 3: Ablation Experiment of Weighted Fusion Mechanism and Update Strategy of Decoder Ablation studies were conducted on the nuScenes validation set, as shown in Table 3, demonstrating the effectiveness of the proposed design. In this experiment, the model was tested by initializing queries using both the LIDAR branch and the image branch. Compared to initializing queries using the image branch heatmap, initializing queries using the LIDAR branch heatmap improved the detection model's NDS by 4.4% and mAP by 3.9%, proving the effectiveness of the LIDAR branch heatmap initialization strategy.

[0047] Experiment 3 results show that using the heatmap from the LiDAR branch for query initialization is far superior to using the image branch. This effectively avoids depth errors and nonlinear distortions caused by camera projection, leverages the inherent advantages of LiDAR in spatial localization, and provides a more reliable foundation for the entire detection process. In the fusion decoder, the initial query is dominated by the geometric information of LiDAR, and the semantic information of the image is gradually incorporated during updates. This strategy makes the system more robust to the limitations of single sensors.

[0048] Table 3

[0049] Experiment 5: Ablation Experiment of Weight Setting Module As shown in Table 4, this experiment conducted ablation experiments on the query update strategy for each layer and adjusted... The values ​​are used to adjust the weights of the radar branch heatmap initialization query and the query obtained from the previous layer, which can reveal... When the value is 0.5, NDS and mAP perform best.

[0050] Table 4

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A 3D target detection method based on fusion of 3D point clouds and multi-view images, characterized in that, include: Acquire 3D point cloud data collected by lidar and 2D image data collected by multi-view cameras; Feature extraction is performed on the point cloud data and image data through the lidar branch and camera branch respectively to obtain lidar BEV features and image BEV features; Heatmaps are generated based on the BEV features of the LiDAR and the BEV features of the image, and object queries are initialized based on the heatmap scores. The initialized object query is input into the query fusion decoder. In each layer of the decoder, the query output of the previous layer is fused with the initial query, and the query is updated through a multi-head self-attention mechanism. The updated query is classified and three-dimensional bounding box regression is performed by the probe to output three-dimensional target detection results.

2. The method according to claim 1, characterized in that, The process of extracting features from the point cloud data and image data through the LiDAR branch and camera branch respectively includes: For the camera branch, a 3D sampling grid is formed based on the input image features. All sampling positions within the 3D sampling grid are projected onto the image using known camera parameters. Voxel features are obtained through bilinear sampling and cross-view summation. The voxel features are then compressed by cascading along the channel dimension. Convolutional layers are used to encode the features to obtain the image BEV features, and a heatmap is generated using ResNet-18. For the LiDAR branch, the input point cloud is processed by VoxelNet to extract 3D voxel features, and the output is the LiDAR BEV feature.

3. The method according to claim 1, characterized in that, The initialization of object queries based on heatmap scores includes: A heatmap is generated based on the BEV characteristics of the LiDAR, and the top K positions with the highest centrality scores are selected as the initial query positions. The BEV features at the corresponding locations are concatenated with the heatmap scores to form the initial query features.

4. The method according to claim 1, characterized in that, The query fusion decoder consists of multiple cascaded multimodal prediction interaction layers. Before input, each layer's decoder uses a weighted fusion module to fuse the query output of the previous layer with the initial query. The fusion formula is as follows: ， in, δ This is an adjustable weighting coefficient used to balance the contributions of initial queries and historical queries. q init This is the initial query, derived from the heatmap initialization. `l` is the index of the current layer, where `l∈{1,2,...,L}`. q prev This is the output of the query from the upper level.

5. The method according to claim 4, characterized in that, The query fusion decoder includes a storage unit for caching the query output of the previous layer so that it can be fused and updated in the next layer decoder.

6. The method according to claim 1, characterized in that, The method also includes constructing a loss function, which includes heatmap loss, classification loss, and 3D bounding box regression loss. The total loss function is the sum of the three, and the prediction is matched with the real target using the Hungarian algorithm.

7. The method according to claim 6, characterized in that, The heatmap loss is calculated using the Gaussian focal loss function, the classification loss is calculated using FocalLoss, and the regression loss is calculated using L1Loss.

8. The method according to claim 1, characterized in that, The method employs a two-stage training strategy during training, using the AdamW optimizer and a single-cycle learning rate strategy, and freezes pre-trained weights in the image branch to save computational resources.

Images