A Multimodal 3D Object Detection Method Based on Panoramic Depth Fusion and Temporal Dynamic Refinement

Abstract

This invention provides a multimodal 3D target detection method based on panoramic depth fusion and temporal dynamic refinement, belonging to the field of computer vision technology. The method is executed on a target detection system deployed with a 4D millimeter-wave radar-camera fusion 3D target detection neural network model. The neural network model includes an image feature encoder, a radar feature encoder, a 2D instance segmentation branch, a 3D detection head, a panoramic depth fusion (PDF) module, a deformable gated temporal fusion (DGTF) module, and an instance-guided dynamic refinement (IGDR) module. During the training phase, it is optimized based on a loss function system using multiple supervised signals. The PDF module reduces depth map structure anomalies caused by sparse radar points, overlap, and small objects. The DGTF module achieves accurate alignment of historical features without any external pose input and adaptively handles occlusion and dynamic objects. The IGDR module recovers lost small target features and improves the system's generalization ability for long-tailed scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and relates to the establishment of a three-dimensional perception neural network model and method using computer vision and deep learning technology. Specifically, it relates to a 3D target detection method based on panoramic depth fusion and temporal dynamic refinement of 4D millimeter-wave radar and camera fusion. Background Technology

[0002] 3D perception is fundamental to intelligent systems such as autonomous driving, roadside sensing systems, drones, and intelligent robots. To achieve this, current research has been actively exploring sensor solutions that surpass traditional LiDAR-camera configurations. Among the many emerging solutions, 4D millimeter-wave radar has attracted widespread attention due to its all-weather reliability, long-range sensing capabilities, and significantly lower cost compared to LiDAR. However, the inherent sparsity and noise characteristics of 4D millimeter-wave radar point cloud data make it difficult to independently support high-precision 3D detection. Therefore, 4D millimeter-wave radar-camera multimodal fusion methods have become the mainstream choice in practical applications. However, developing high-performance neural network models for 4D millimeter-wave radar-camera 3D object detection still faces many challenges.

[0003] The existing technology has the following three main limitations:

[0004] First, the primary challenge of current 4D millimeter-wave radar-camera fusion frameworks lies in the limited accuracy of their absolute depth estimation modules, leading to inaccurate 3D localization. Existing radar-camera fusion frameworks rely on accurate, dense depth estimation during view transformations. However, these methods only apply absolute depth supervision to the foreground points, resulting in sparse depth supervision. Therefore, they cannot achieve high-quality panoramic depth estimation, which can improve foreground localization accuracy while reducing background noise. Furthermore, recent research on relative depth estimation has demonstrated strong generalization capabilities. How to effectively utilize these capabilities to achieve accurate panoramic absolute depth estimation is a topic worthy of in-depth exploration.

[0005] Second, there is the issue of temporal fusion when vehicle pose is missing. Temporal input provides crucial historical 3D information about occluded objects, which is essential for 3D object detection tasks. In real-world scenarios, especially in rural areas, missing or inaccurate vehicle poses due to GPS signal loss are common. Mainstream 4D millimeter-wave radar datasets (such as TJ4DRadSet and VoD) also lack vehicle poses. Existing research attempts to incorporate temporal information by simply stacking historical bird's-eye view (BEV) features along the channel dimension, but this improvement is very limited.

[0006] Third, recall and precision are low for small objects (such as pedestrians and cyclists in the distance). When small objects appear in an image, their surfaces may not reflect any radar point cloud. For such challenging situations, it is necessary to utilize visual prior information for recognition. Some works attempt to extract instance candidate boxes from two-dimensional features, but they all rely on Transformer-based architectures and treat the extracted instances only as query vectors, which makes them incompatible with other widely used object detection frameworks (such as CenterPoint). Summary of the Invention

[0007] To address the aforementioned limitations, this invention provides a 3D target detection method based on 4D millimeter-wave radar-camera fusion with panoramic depth fusion and temporal dynamic refinement, for high-performance 3D object detection. To address the first challenge, this invention proposes a panoramic depth fusion module. Specifically, to eliminate geometric contamination, this module abandons sparse metric regression and introduces a pairwise ordinal ranking loss with dynamic tolerance, enabling the model to learn high-fidelity, sharp-edge, and structurally continuous geometric information. To address the second challenge, this invention designs a deformable gated temporal fusion module. This module explicitly aligns non-rigid motion through deformable convolution and smooths temporal noise through the gating mechanism of gated recurrent units (GRUs), achieving stable feature aggregation and avoiding the instability of implicit loops. To address the third challenge, this invention proposes an instance-guided dynamic optimization module. This module utilizes 2D instance-guided semantic prototypes to purify features and jointly optimizes them with the detection head to correct semantic errors in projection and restore clear features of distant or partially occluded objects.

[0008] The technical solution of this invention is as follows:

[0009] A multimodal 3D target detection method based on panoramic depth fusion and temporal dynamic refinement is disclosed. This method is executed on a target detection system equipped with a 4D millimeter-wave radar-camera fusion 3D target detection neural network model. The target detection system includes a camera and a 4D millimeter-wave radar sensor. The neural network model comprises an image feature encoder, a radar feature encoder, a 2D instance segmentation branch, a 3D detection head, a panoramic depth fusion (PDF) module, a deformable gated temporal fusion (DGTF) module, and an instance-guided dynamic refinement (IGDR) module. During the training phase, the neural network model is optimized based on a loss function system using multiple supervised signals. The specific steps of this method include:

[0010] 1) Multimodal data input and basic feature extraction;

[0011] It receives camera image data and 4D millimeter-wave radar point cloud data at the current moment, extracts image features and radar BEV features using image and radar feature encoders respectively, and outputs the 2D instance features and 2D instance mask of the current frame in parallel by the 2D instance segmentation branch.

[0012] 2) Construct a panoramic depth fusion (PDF) module and perform panoramic depth fusion processing to generate high-precision panoramic depth;

[0013] A depth estimation network is constructed, which takes the extracted image features and radar BEV features as input, combines the radar's geometric cues and image semantics to predict the depth probability distribution, and calculates the expected high-fidelity panoramic dense depth map based on the distribution. Based on this, the image features are projected onto the 3D bird's-eye view space and fused with the radar BEV features to generate the panoramic fused BEV features of the current frame.

[0014] 3) Construct a deformable gated temporal fusion (DGTF) module to perform pose-independent temporal feature aggregation and temporal fusion processing;

[0015] Read the historical frame temporal hidden state features stored in the temporal memory bank, input the current frame panoramic fusion BEV features and the historical frame temporal hidden state features into the DGTF module. The module first performs motion alignment based on feature differences, and then fuses historical memory and updates temporal information through a gating mechanism to output temporally enhanced panoramic fusion BEV features.

[0016] 4) Construct an instance-bootstrapping dynamic refinement (IGDR) module to perform instance-bootstrapping feature refinement processing;

[0017] The temporal enhanced panoramic fusion BEV feature output in step 3) is used as the feature to be calibrated, and the 2D instance feature and 2D instance mask output in step 1) are used as the guiding feature input to the IGDR module. The IGDR module uses the 2D instance feature to generate a pure semantic prototype, performs feature enhancement and denoising on the object region in the temporal enhanced panoramic fusion BEV feature, and outputs the final purified panoramic fusion BEV feature.

[0018] 5) Target detection decoding output: Input the purified panoramic fusion BEV features obtained in step 4) into the 3D detection head, decode, calculate and output the 3D target detection boxes (position, size, rotation angle), category labels and velocity information of all targets.

[0019] Furthermore, step 1) multimodal data input and basic feature extraction specifically includes:

[0020] 11) Acquire camera image data around the target detection system in real time, preprocess the images (including cropping size, normalization, etc.); input the preprocessed camera image data into the image feature encoder, extract the backbone of the encoder network and fuse it at multiple scales to output a multi-scale image feature map containing rich texture and semantic information.

[0021] 12) Synchronously acquire point cloud data of the 4D millimeter-wave radar of the target detection system, including information such as spatial coordinates, radial velocity and reflection intensity; input the point cloud data into the radar feature encoder, and output radar BEV features containing spatial geometry and motion information;

[0022] 13) The above image features are processed in parallel using the 2D instance segmentation branch. Region proposal is used to generate feature alignment operations, and the 2D instance-level feature vectors of the foreground objects in the current scene and the corresponding 2D instance masks are output. This provides a noise-free, high-confidence semantic guidance source with clear small objects for the subsequent instance-guided dynamic refinement module.

[0023] Furthermore, step 2) of the construction and execution process of the panoramic depth fusion module specifically includes:

[0024] 21) Receive the multimodal feature data output in step 1), specifically including encoded image semantic features and radar geometric features;

[0025] 22) Construct a depth estimation network using an attention-based feature interaction architecture. This network is configured to use radar features as the geometric query index and image features as the semantic key / value pair. By combining the sparse geometric cues provided by radar features with the dense semantic context provided by image features, the network directly predicts the depth probability distribution in the view frustum space through cross-modal interaction. It should be noted that the network module in this step is explicitly constrained by the panoramic depth fusion loss system (including probability distribution supervision loss, global refinement supervision loss, and relative loss) during the training phase. Thanks to this, the network has edge sensitivity to depth abrupt changes during inference, effectively suppressing the edge blurring phenomenon of traditional depth estimation.

[0026] 23) The depth probability distribution is multiplied by the image features, and the Z-axis is compressed by convolution before being concatenated with the original radar features to obtain semantically enhanced radar BEV features. The image 2D instance mask, depth probability distribution, and semantic features are used as inputs to the view transformation operation, which uses a 2D to 3D projection algorithm to project the image features from the perspective view space to the 3D bird's-eye view space to generate localized enhanced image BEV features. Subsequently, the image BEV features and radar BEV features are fused at the feature level to output geometrically enhanced panoramic fusion BEV features for the current frame.

[0027] Furthermore, in step 3), the construction and execution steps of the Deformable Gated Timing Fusion (DGTF) module specifically include:

[0028] 31) Read the hidden state features of historical frames in the previous time-series memory;

[0029] 32) Construct a motion-aware alignment branch. This branch is configured to not rely on external pose information. It uses deformable convolution to analyze the feature differences of the temporal hidden state features between the current frame and the historical frames, predicts the spatial displacement deviation of feature points between time sequences, and performs spatial distortion correction and registration on the historical features based on the displacement deviation to compensate for the non-rigid misalignment of features caused by motion.

[0030] 33) Construct a gated temporal update branch and introduce an adaptive temporal feature fusion mechanism. This mechanism is configured to evaluate the effectiveness of the current observed features and the historical aligned features, dynamically calculate the forgetting ratio of historical information and the updating ratio of current information, and generate panoramic fusion BEV features with enhanced temporal consistency through weighted fusion, which effectively ensures the stability of the results in step 32).

[0031] Furthermore, in step 4), the specific steps for constructing and executing the instance-guided dynamic refinement (IGDR) module include:

[0032] 41) Construct an instance semantic prototype generation unit;

[0033] The system receives instance feature vectors and 2D instance masks extracted from the 2D instance segmentation branch. First, it performs global average pooling on each instance feature vector to extract the instance semantic prototype. Then, it uses perspective projection to project the 2D instance mask onto the BEV plane to obtain an instance spatial distribution map. Finally, based on the position index of the distribution map, it broadcasts and maps the instance semantic prototype to the corresponding position in the BEV space to construct a clean instance semantic feature map with the same size as the panoramic fusion BEV feature map.

[0034] 42) Construct a prototype-guided dynamic calibration unit;

[0035] Using the clean instance semantic feature map generated in step 41) as a conditional guide, the affine transformation parameters for each pixel are dynamically predicted through a convolutional layer. The parameters include a scaling factor map and a bias factor map. The affine transformation parameters are used to perform pixel-by-pixel feature recalibration on the input panoramic fusion BEV features to be calibrated. By applying the affine transformation, the feature response amplitude of the foreground region is enhanced and the background noise is suppressed, thereby generating calibrated panoramic fusion BEV features.

[0036] 43) Construct a foreground gating fusion unit;

[0037] A foreground gating map is generated based on the instance spatial distribution map in step 41). The calibrated panoramic fusion BEV features output in step 42) are weighted and fused with the original panoramic fusion BEV features to be calibrated using the foreground gating map. Specifically, the calibrated panoramic fusion BEV features are fused preferentially in the instance region indicated by the gating map, while the original panoramic fusion BEV features are retained in the background region, thereby outputting the final purified features that have corrected feature divergence and enhanced small target response.

[0038] Furthermore, in step 5), the target detection and decoding output includes the following process:

[0039] 51) Receive the purified panoramic fusion BEV features after foreground gating fusion in step 4) and construct a 3D target detection head as the output layer of the system; the 3D detection head is configured to perform target decoding operation on the input purified panoramic fusion BEV features, and parse the target's category confidence information and three-dimensional spatial geometric attribute information including motion state from the feature map in parallel; finally, integrate the category confidence information and three-dimensional spatial geometric attribute information to generate and output a high-precision 3D target detection box.

[0040] Furthermore, to enable the neural network model to possess the aforementioned high-precision feature extraction and target detection capabilities, the neural network model is optimized during the training phase based on a loss function system using multiple supervision signals. The specific loss function system is as follows:

[0041] For the aforementioned panoramic depth fusion (PDF) module, a sparse-dense complementary depth estimation loss is designed. ,include:

[0042] (1) Probability distribution supervised loss ( The target distribution is constructed using sparse, high-precision radar points, and the absolute depth accuracy of the foreground region is monitored by minimizing the divergence between the predicted distribution and the target distribution.

[0043] (2) Global detailed monitoring of losses ( It combines absolute depth constraints from sparse 4D millimeter-wave radar point cloud data with dense relative depth constraints generated by a depth estimation supervision model; used to constrain the prediction results of the depth estimation network.

[0044] Specifically, a pre-trained depth estimation supervised model is used to process camera image data to generate dense depth pseudo-real values ​​with complete scene geometry; the depth estimation supervised model is configured to extract relative depth relationships in the image;

[0045] When calculating the global refinement supervision loss, on the one hand, the difference in probability distribution between the depth probability distribution of the depth estimation network and the projection of sparse 4D millimeter-wave radar point cloud data is calculated to ensure the measurement accuracy of the prediction results; on the other hand, the scale-invariant depth error (or relative depth structure error) between the panoramic dense depth map and the dense depth pseudo-true value is calculated, so as to use the geometric structure information in the dense depth pseudo-true value to guide the prediction results to recover complete scene details, so that the depth estimation results can recover the complete scene geometry guided by a large amount of data with superior generalization ability while having measurement accuracy.

[0046] (3) Relative loss ( To address this, a pairwise ordinal ranking loss based on dynamic thresholds is designed, specifically including edge sharpening and global relative position strategies: morphological dilation is performed on the 2D instance mask, and pairwise pixel sampling is performed between the morphologically dilated region at the object edge (the background boundary that does not belong to the 2D instance mask after morphological dilation) and the object interior (foreground core), forcing the network to learn the depth abrupt changes at the object contour, thereby eliminating edge blurring in depth estimation; pairwise pixel sampling is performed on the entire image to better understand the relative positional relationships between large objects and the overall image, as well as between different parts of the image.

[0047] The total loss function of the panoramic depth estimation fusion module is: , and Weighted summation;

[0048] For the 3D detection head module, a basic detection loss is constructed. The loss specifically includes: a classification loss (e.g., Focal Loss) used to supervise the prediction of 3D object categories, and a regression loss (e.g., L1 Loss) used to supervise the localization accuracy of 3D object detection boxes.

[0049] For the 2D instance segmentation branch and the panoramic depth fusion (PDF) module, a 2D instance segmentation and auxiliary perception loss is designed. To maintain the semantic consistency of the feature extraction network during training and to assist the multimodal fusion module in fast convergence, this includes:

[0050] (1) 2D task loss applied to the 2D instance segmentation branch, including RPN network loss, RoI Head classification and regression loss and 2D instance mask prediction loss, to maintain the semantic purity of image features;

[0051] (2) The joint supervision signal of the auxiliary perception loss acting on the multimodal fusion module, including the BEV segmentation loss and the auxiliary loss in the encoding stage of 4D millimeter-wave radar point cloud data, is used to constrain the foreground distribution of the fused panoramic fusion BEV features.

[0052] The final total loss function used for backpropagation during the training phase is the weighted sum of all the above losses of the unfrozen modules during the current training phase, which is used to constrain the final output of the system and the auxiliary semantic branches.

[0053] Furthermore, the neural network model employs a two-stage progressive training method, including:

[0054] Phase 1: Pre-training of basic spatial perception features:

[0055] The parameters of the detection head, deformable gated temporal fusion (DGTF) module, and instance-guided dynamic refinement (IGDR) module in the neural network model are frozen. The remaining parts are subjected to perceptual training to extract single-frame BEV multimodal fusion features. The 2D instance segmentation branch is supervised by 2D annotation. The panoramic depth fusion (PDF) module is pre-trained for depth estimation to enable it to have accurate 3D geometric projection capabilities.

[0056] The second stage involves end-to-end spatiotemporal fine-tuning:

[0057] Load the pre-trained weights for the first stage, activate the Deformable Gated Temporal Fusion (DGTF) module and the Instance Guided Dynamic Refinement (IGDR) module, start the alignment of temporal and dynamic refinement fine-tuning, and start the 3D detection head for end-to-end model training.

[0058] Compared with the prior art, the present invention has the following significant advantages:

[0059] 1. This invention addresses the issues of fuzzy depth estimation and inaccurate edges in multimodal fusion, thereby improving localization accuracy. The panoramic depth fusion (PDF) module proposed in this invention effectively reduces the anomalies in the depth probability distribution structure caused by sparse radar points, overlaps, and small objects by introducing probability distribution supervision, global refinement supervision, and relative supervision. This results in more geometrically accurate panoramic fusion BEV features, significantly improving the localization accuracy of 3D target detection.

[0060] 2. This invention eliminates the reliance on high-precision vehicle pose for temporal fusion, enhancing system robustness. Most existing technologies rely on precise pose matrices to stitch together features from multiple frames; however, detection performance deteriorates drastically if the signal is lost or the vehicle experiences severe vibrations. The Deformable Gated Temporal Fusion (DGTF) module proposed in this invention automatically predicts motion offsets through feature-level motion-aware alignment branches, achieving accurate alignment of historical features without any external pose input. Combined with a gating update mechanism, it can adaptively handle occlusion and dynamic objects, enabling the system to maintain stable detection performance even in confined environments such as tunnels and urban canyons.

[0061] 3. This invention effectively solves the problems of feature loss and noise contamination during the BEV projection process for small targets and overlapping objects, thus improving recall. Due to depth estimation errors, small targets at a distance (such as cyclists and pedestrians) often exhibit feature divergence or disappearance when projected from the image to the BEV space, and pedestrians standing close together may be identified as a single object. The Instance-Guided Dynamic Refinement (IGDR) module proposed in this invention utilizes clean 2D instance features as semantic prototypes to perform pixel-level redrawing and enhancement of panoramic fusion BEV features through dynamic refinement. This not only restores the lost features of small targets but also significantly improves the system's generalization ability to long-tailed scenes through a training strategy that simulates inference paths.

[0062] 4. High-performance end-to-end detection is achieved, reaching a leading level on public datasets. This invention organically integrates the above modules to form a robust end-to-end detection model framework. Experimental data shows that on public datasets such as View-of-Delft (VoD) and TJ4DRadSet, the method of this invention significantly outperforms existing baseline models (SGDet3D) and other mainstream methods in terms of mean accuracy (mAP). Attached Figure Description

[0063] Figure 1 This is a diagram illustrating the overall architecture and workflow of the method of this invention.

[0064] Figure 2 This is a schematic diagram of the structure and workflow of the panoramic depth fusion (PDF) module in this invention.

[0065] Figure 3 This is a schematic diagram of the structure and workflow of the Deformable Gated Timing Fusion (DGTF) module in the invention.

[0066] Figure 4 This is a schematic diagram of the structure and workflow of the Instance-Guided Dynamic Refinement (IGDR) module in this invention.

[0067] Figure 5 This is a qualitative comparison diagram between the present invention and existing baseline methods. Detailed Implementation

[0068] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0069] This invention proposes a 3D target detection method based on panoramic depth fusion and temporal dynamic refinement of 4D millimeter-wave radar and camera fusion. Addressing the three core pain points of existing 4D millimeter-wave radar-camera 3D target detection neural network models, namely inaccurate depth estimation, temporally dependent pose, and loss of small target features in radar-vision fusion, this invention designs three innovative modules: panoramic depth fusion (PDF), deformable gated temporal fusion (DGTF), and instance-guided dynamic refinement (IGDR).

[0070] This invention discloses a 4D millimeter-wave radar and camera fusion 3D target detection method based on panoramic depth fusion and temporal dynamic refinement. This method is executed on a target detection system equipped with a 4D millimeter-wave radar-camera fusion 3D target detection neural network model. The target detection system includes a camera and a 4D millimeter-wave radar sensor. The neural network model includes an image feature encoder, a radar feature encoder, a 2D instance segmentation branch, a 3D detection head, a panoramic depth fusion (PDF) module, a deformable gated temporal fusion (DGTF) module, and an instance-guided dynamic refinement (IGDR) module. During the training phase, the method is optimized based on a loss function system with multiple supervision signals. The steps of this invention include:

[0071] S1. Multimodal data input and basic perception: Receive camera images and 4D millimeter-wave radar point cloud data, and extract image semantic features and radar geometric features through image encoder and radar encoder respectively; at the same time, the 2D instance segmentation branch works in parallel to directly extract the instance feature vector of the foreground target and the 2D instance mask from the image.

[0072] S2. Panoramic Depth Fusion Execution: The PDF module receives multimodal features and infers the depth probability distribution; it uses this distribution to perform a view transformation operation, projecting the image features onto the 3D bird's-eye view space and fusing them with radar features to generate the current frame fused features.

[0073] S3. Input the current frame fusion features and historical hidden states into the DGTF module, first perform motion alignment based on feature content (without reading pose data), then fuse historical memory and update temporal information through a gating mechanism, and output temporally enhanced panoramic fusion BEV features.

[0074] S4. Input the temporally enhanced panoramic fusion BEV features, the 2D instance features and 2D instance mask obtained in step S1 as guiding features into the IGDR module. The IGDR module uses the 2D instance features to generate a clean semantic prototype, and performs feature enhancement and denoising on the object regions (especially small targets or blurred regions) in the panoramic fusion BEV features, and outputs the final purified panoramic fusion BEV features.

[0075] S5. Input the purified panoramic fusion BEV features obtained in step 4) into the 3D detection head, calculate and output the 3D target detection boxes (position, size, rotation angle), category labels and velocity information of all targets, that is, output the 3D target detection results.

[0076] In this embodiment, the target detection system is an intelligent driving perception system. This intelligent driving perception system is equipped with sensors such as cameras and 4D millimeter-wave radar. It simultaneously collects images from a monocular camera and point cloud data from the 4D millimeter-wave radar as inputs to a neural network model, and outputs three-dimensional target detection results.

[0077] This embodiment also provides a specific implementation method for pre-building and training a neural network model, combined with a 3D target detection method based on panoramic depth fusion and temporal dynamic refinement using 4D millimeter-wave radar and camera fusion, specifically including the following steps:

[0078] S1. Pre-select or construct the basic modules for multimodal feature extraction and basic perceptual coding, which specifically includes the following sub-steps:

[0079] S1.1 Image Feature Encoder: In this embodiment, ResNet-50 is selected as the backbone network, followed by a Feature Pyramid Network (FPN). Input camera image data is preprocessed and its resolution is adjusted to 896*1408 (for the VoD dataset) or 480*640 (for the TJ4DRadSet dataset). The image is processed by ResNet to extract features in four stages, which are then fused by the FPN to output a feature map with a downsampling factor of 8. Its channel dimension is set to C=256. Its purpose is to provide the system with high-resolution visual semantic context, which is the foundation for subsequent accurate object classification and depth estimation.

[0080] S1.2 Radar Feature Encoder: The RadarPillarNet module is selected as the radar feature extractor. Input is 4D millimeter-wave radar point cloud data, including... Its five-dimensional features are horizontal coordinate, vertical coordinate, height coordinate, radar cross section (RCS), and Doppler velocity. The point cloud space is divided into voxels of size 0.16m. PointNet is used to encode the points within the voxels, and the VES is projected onto the bird's-eye view plane through a scattering operation to output the radar BEV features. Its function is to utilize the long-range detection and velocity measurement capabilities of radar to provide the system with accurate geometric and motion priors unaffected by lighting conditions;

[0081] S1.3 2D Instance Segmentation Branch: A lightweight instance segmentation head is constructed in parallel after the image FPN, using a Mask R-CNN architecture that includes a Region Proposal Network (RPN) and RoIAlign layers. The output is a 2D instance feature vector of the foreground objects in the image. and pixel-level binary 2D instance mask Its function is to directly separate clean foreground target information from the original image, independent of the BEV projection process, and provide a noise-free, high-confidence semantic guidance source with clear small objects for the subsequent instance-guided dynamic refinement module.

[0082] S1.4 The detection network described in this example uses the CenterPoint detection head; after extracting features through shared convolutional layers, the features are input in parallel to the classification prediction branch and the attribute regression branch; the classification branch is used to predict the probability of the center point position of various targets in the bird's-eye view and the category confidence, while the attribute regression branch is used to regress the three-dimensional geometric dimensions, spatial position offset, orientation angle and motion velocity vector of the target. Finally, the outputs of the two branches are combined to decode and obtain a high-precision 3D target detection box.

[0083] S1.5 Depth Estimation Model: The Metric3D model is selected. This model is only used in the offline training phase to perform inference on the training set images. The generated dense depth pseudo-real values ​​are used as pseudo-real values ​​for auxiliary supervision.

[0084] S1.6 Data Acquisition: During driving, the intelligent driving system uses various sensors to synchronously acquire environmental data in real time. Specifically, this includes images from a monocular camera covering the forward field of view of the intelligent driving system. and sparse point cloud data output by 4D millimeter-wave radar. (Include ).

[0085] S1.7 Basic Feature Extraction: Input the single-frame image I into the ResNet-50 encoder constructed in step S1.1 to extract multi-scale image features. After fusion by FPN, output image features with uniform resolution. .

[0086] 4D millimeter-wave radar point cloud data The Radar PillarNet constructed from input S1.2 is processed by voxelization and point feature encoding to output radar BEV features containing geometric and motion information. .

[0087] S1.8 2D instance prior extraction:

[0088] In parallel, image features Input the 2D instance segmentation branch of S1.3. Generate proposal boxes via RPN and extract features using RoIAlign, outputting instance feature vectors of all foreground objects within the current frame's forward field of view. And pixel-level 2D instance masks. This process is performed entirely in 2D perspective space, independent of subsequent view transformation operations, thus obtaining... These are pure semantic features that are not contaminated by monocular depth estimation errors.

[0089] S2. The process of panoramic depth fusion (PDF) and view transformation is as follows: Figure 2 As shown, this module aims to generate high-quality depth distributions. In this embodiment, the specific construction of the PDF module and the design of the loss function are as follows:

[0090] S2.1 Receive the multimodal feature data output in step S1, specifically including encoded image semantic features and radar geometric features;

[0091] S2.2 Constructing a depth estimation network and inferring the depth probability distribution: A depth estimation network is constructed using a feature interaction architecture based on an attention mechanism. Specifically, the image semantic features and radar geometric features output from step S1 are received. For the projection position of a radar point on the image, a 5*5 local window is selected around it. Using a neighborhood cross attention mechanism, the radar geometric features are used as a query, and the image semantic features are used as a key / value pair for interaction, directly predicting the depth probability distribution in the view frustum space. (That is, the number of depth surfaces is set to a probability tensor of D=80). The model parameters of the depth estimation network are jointly trained and optimized based on multiple supervised loss functions (including probability distribution supervised loss, global refinement supervised loss and relative loss); thanks to the constraint of the relative ranking loss on the object boundary, the depth probability distribution output by this step has significant gradient cutoff characteristics at the object contour (i.e. sharp edges) and has high-fidelity measurement accuracy.

[0092] S2.3 The depth probability distribution generated by the above method is multiplied by the image features, and after convolution compression of the Z-axis, it is concatenated with the original radar features to obtain semantically enhanced radar BEV features; the image 2D instance mask, depth probability distribution, and semantic features are combined. As input to the view transformation operation, the view transformation operation employs a 2D to 3D projection algorithm (such as Lift, Splat, Shoot (LSS) algorithm) to obtain the localized and enhanced image BEV features (details are as follows in 2.4). These two BEV features are then fused into a preliminary panoramic fused BEV feature through cross-modal fusion, i.e., channel stitching (Concatenation + Conv) or a deformable attention method based on transformers. Subsequently, neighborhood cross-attention is performed (using the obtained fused BEV from the 2D instance mask). Enhanced depth estimation (The query is performed in the middle) to obtain the final multimodal fusion BEV (RC-BEV) features, thereby completing a deeper cross-modal interaction;

[0093] S2.4 Specific details of the above view transformation operation (LSS): Constructing an object-oriented 3D feature space. First, the image semantic features... With depth probability distribution An outer product expansion is performed to generate the basic view frustum features; subsequently, a 2D instance mask of the image is introduced. The pixel-domain weighted filtering of this feature volume is calculated using the following formula: ;in, This refers to the enhanced visual cone feature. This represents the Hadamard product (i.e., element-level multiplication). This represents the Kronecker product. Subsequently, using camera intrinsic and extrinsic parameters, this feature volume is projected (Splat) and pooled along the vertical direction into a 3D bird's-eye view space, ultimately generating a geometry-aware image BEV feature. .

[0094] S3. Deformable Gated Timing Fusion (DGTF) module, such as Figure 3 As shown, its construction and execution process;

[0095] This module is used to aggregate timing information without pose input.

[0096] S3.1 State Reading: Read the hidden state features of the historical frames stored in the previous time step from the system memory. .

[0097] S3.2 Constructing a motion-aware alignment branch: Deformable convolution (DCNv2) is employed. This branch is configured to be independent of external pose information. By analyzing the feature differences between the temporal hidden state features of the current frame and historical frames, it predicts the spatial displacement deviation of feature points between time sequences. Based on this displacement deviation, it performs spatial distortion correction and registration on historical features to compensate for the non-rigid misalignment of features caused by motion (i.e., decoupled spatial alignment).

[0098] After obtaining the current frame RC-BEV features from step S2.3, first, the features of the current frame (denoted as time t) are... (This refers to the RC-BEV mentioned earlier) and the hidden state of the previous frame. By concatenating the data, the sampling offset is predicted using two layers of convolution. And the modulation mask m, the formula is expressed as:

[0099] ,

[0100] Among them, Conv offset ( ) represents the convolutional layer used to predict the offset, and Concat() represents the concatenation operation along the feature channel dimension; the alignment formula is expressed as: DCN() represents deformable convolution operation; This step involves spatially aligned historical frames with temporal hidden state features; it does not rely on vehicle pose data and is entirely based on feature matching.

[0101] S3.3 Constructs a gated temporal update branch, employing a combination of a convolutional neural network (CNN) and a gated recurrent unit (GRU).

[0102] The hybrid neural network structure is ConvGRU, which introduces an adaptive temporal feature fusion mechanism. This mechanism is configured to evaluate the effectiveness of the current observation features and the historical alignment features, dynamically calculate the forgetting ratio of historical information and the updating ratio of current information, and generate temporally enhanced panoramic fusion BEV features through weighted fusion, which effectively ensures the stability of the results in step 3.2).

[0103] Calculate the update gate at time t. and reset door :

[0104] ,

[0105] Where σ represents the Sigmoid activation function, Conv z ( ) and Conv r ( ) represent the convolutional layers used to generate the update gate and reset gate, respectively; Concat() represents the feature concatenation operation; and Conv() represents the convolution operation.

[0106] The state update formula is expressed as:

[0107] ,

[0108] Where tanh represents the hyperbolic tangent activation function;

[0109] S4. Instance-guided Dynamic Refinement (IGDR) module, such as Figure 4 As shown, its construction and execution process;

[0110] S4.1 Instance Semantic Prototype Generation Unit Construction:

[0111] The input is the instance feature vector extracted from the 2D instance segmentation branch in step S1.3. (Where N is the number of instances and C is the number of channels) and the corresponding 2D instance mask. .

[0112] First, global average pooling is performed on the feature vector of each instance to extract a semantic prototype with high semantic generality, denoted as . .

[0113] Subsequently, using perspective projection geometry, the 2D instance mask is... Projecting onto the BEV plane yields the BEV mask. .based on The position index will Broadcast to the corresponding spatial location of the BEV feature map to generate a clean instance semantic feature map with the same size as the panoramic fused BEV feature map. C, H, and W represent the number of channels in the feature map, the height of the panoramic fusion BEV feature map, and the width, respectively.

[0114] S4.2 Prototype-guided construction of dynamic calibration cells:

[0115] This unit utilizes As a guiding condition, pixel-level affine transformation parameters are dynamically generated. Specifically, two independent 1x1 convolutional layers are used. Processing is performed to predict scaling factor maps separately. and bias factor plot :

[0116] ,

[0117] Using the above parameters to fuse the input panoramic BEV features Pixel-by-pixel feature recalibration is performed to enhance foreground (especially small targets) features and suppress background noise. The calibration formula is as follows:

[0118] ,

[0119] in, The calibrated panoramic fusion BEV features;

[0120] S4.3 Foreground Gated Fusion Unit Construction:

[0121] To avoid misleading the background region during the calibration process, a foreground gating mechanism is introduced. A foreground gating map is generated using a BEV mask. (That is, the instance area is 1 and the background area is 0).

[0122] Final purification characteristics BEV features fused from calibrated panoramic views BEV features integrated with the original panoramic view The weighted fusion yields:

[0123] ,

[0124] This step can significantly amplify the feature response of small targets at a distance (such as cyclists) and correct feature divergence caused by monocular depth estimation errors.

[0125] S5. Object Detection Decoding:

[0126] S5.1 Purification characteristics after foreground-gated fusion Input the CenterPoint detection head selected in this invention. Detection.

[0127] The head employs a combination of heatmap regression and attribute prediction branches. Specifically, the detection head uses a keypoint heatmap branch to predict the center confidence of the target category under Gaussian Focal Loss supervision, while simultaneously using an attribute regression branch to accurately regress the target's center offset, absolute height, logarithmic size, sine and cosine values ​​of rotation angles, and velocity vector under L1Loss supervision. During the inference phase, 3*3 max pooling is used to extract heatmap peaks, and a circular non-maximum suppression algorithm optimized for BEV scenarios is employed to remove overlapping boxes. Finally, a list of 3D target detection boxes is decoded and output, including the target's 3D position (x, y, z), size (l, w, h), and orientation. ,speed and category confidence, where l, w, and h represent the length, width, and height of the target, respectively. This indicates the target's yaw angle, i.e., the target's orientation on the bird's-eye view plane.

[0128] The following are the training strategies, loss functions, and hyperparameter settings for training the neural network model in this embodiment:

[0129] (1) Optimizer Configuration: In this embodiment, the AdamW optimizer is used to update the model parameters. The initial learning rate is set to... The weight decay coefficient is set to 0.01, and the global batch size is set to 4.

[0130] (2) Learning rate scheduling: The learning rate is dynamically adjusted using a cosine annealing strategy, and no restart is set to ensure the stability of model convergence.

[0131] (3) Loss function configuration: The final optimized total loss function of the system is defined as the weighted average of the following multi-task losses:

[0132] (3.1) 3D target basic detection loss ( ): For the final output, it includes FocalLoss for class classification, Smooth-L1 Loss for 3D object detection box refinement, and orientation classification loss;

[0133] (3.2) Panoramic depth fusion loss ( ):

[0134] (3.2.1) Probability distribution supervised loss ( ): sparse radar truth points Transform into a Gaussian distribution centered thereon Calculate the KL divergence between the predicted distribution and the target distribution:

[0135] ,

[0136] in, The depth probability distribution predicted by S2.2; Let KL divergence be a metric. It is the set of all pixels that have a valid truth value; It is the total number of pixels with truth monitoring signals;

[0137] (3.2.2) Globally refined monitoring loss ( ): Combining the L1 loss of sparse radar points with the scale-invariant logarithmic loss (SILog) of Metric3D pseudo-true values:

[0138] ,

[0139] in, , These are weighting coefficients; Yes The panoramic dense depth map obtained by performing expected calculations;

[0140] It is a sparse radar measurement depth; These are pseudo-true values ​​generated by the Metric3D model; It is a scale

[0141] Scale-Invariant Logarithmic Loss (SIL), a loss function commonly used in depth estimation, is related to...

[0142] Note the relative depth relationship rather than the absolute numerical difference.

[0143] (3.2.3) Relative loss ( ): Using the 2D instance mask generated by S1.3, a morphological dilation operation is performed on the object's edges. Pairwise sampling of pixels (i, j) is performed in the dilation ring region (the background boundary that is not part of the object's 2D instance mask) and inside the object (the foreground core). ,calculate: ,in, Let be the ground truth depth values ​​for pixels i and j; Let i be the predicted depth for pixels i and j; This is a depth sign function that indicates which pixel is farther, i or j. It is 1 if i is farther than j, and -1 otherwise.

[0144] The total loss function of the panoramic depth estimation fusion module is: , and The weighted summation; expressed as:

[0145] In this embodiment, the preferred configuration of the above-mentioned loss coefficient is as follows:

[0146] .

[0147] (3.3) The weighted loss of 2D instance segmentation loss and auxiliary perception loss ( ): For 2D instance segmentation loss, it includes RPN network loss and RoI Head classification (such as Focal Loss), regression loss (such as L1 / Smooth-L1 Loss) and 2D instance mask prediction loss, which are used to ensure the purity of semantic guidance; the auxiliary perception loss includes BEV segmentation loss used to constrain the distribution of fused feature foreground and auxiliary loss in the 4D millimeter-wave radar point cloud data encoding stage.

[0148] The total loss function used for backpropagation during the training phase is a weighted sum of all the losses mentioned above for modules that are not frozen during the current training phase. It is used to constrain the final output of the system and the auxiliary semantic branches.

[0149] It should be noted that the basic detection loss, 2D instance segmentation, and auxiliary perception loss mentioned in this embodiment are merely illustrative examples. Those skilled in the art may use other loss functions with the same constraint functions to replace them according to actual scenario requirements, and such replacements are all within the scope of protection of this application.

[0150] (4) Training steps: The training process adopts a two-stage progressive strategy;

[0151] Phase 1: Pre-training of basic spatial perception features (15 Epochs);

[0152] This stage focuses on optimizing the system's spatial feature extraction and auxiliary geometric perception capabilities, temporarily omitting Long Temporal Feature Aggregation (DGTF), IGDR dynamic refinement, and the detection head module. The emphasis is on pre-training the panoramic depth fusion module (PDF) for depth estimation, enabling it to possess accurate 3D geometric projection capabilities; 2D labeled supervised instance segmentation is used. This stage ensures that the features input to subsequent fusion modules are of high geometric and semantic quality, avoiding fusion noise caused by feature misalignment.

[0153] The parameters of the detection head, DGTF, and IGDR modules in the above neural network model are frozen, and the remaining parts are subjected to perceptual training to enhance the single-frame feature extraction capability. The total loss function in this stage is: and The weighted summation result;

[0154] Phase 2: Spatiotemporal end-to-end training (15 Epochs);

[0155] All parameters in the neural network model pre-trained in the first stage are unfrozen, and then alignment and dynamic fine-tuning begin. End-to-end model training is then initiated using the 3D detector head. The total loss used for backpropagation in this training stage is... and The weighted summation result;

[0156] To verify the effectiveness of this invention, rigorous testing was conducted on the TJ4DRadSet 4D millimeter-wave radar and image-based autonomous driving dataset. The system built based on this embodiment achieved significantly better performance than existing technologies in 3D object detection tasks. A qualitative comparison of this invention with existing baseline methods is shown in the figure below. Figure 5 As shown, the left side of each of the four scene sub-images represents the baseline model SGDet3D, while the right side represents the R4Det method of this invention. Objects not detected by the baseline model are circled in orange, while those successfully detected by this method are highlighted. Specifically, the mAP of this embodiment... 3D The mean three-dimensional accuracy reached 47.29%, mAP BEVThe mean accuracy (mAP) from a bird's-eye view reached 54.07%. Compared with the baseline model SGDet3D, this invention achieves higher mAP. 3D It improved by +5.47% in mAP. BEV The accuracy improved by +6.91%. In more specific categories, for cyclists, who are extremely difficult to detect due to their small radar cross-section, the accuracy improvement of this invention reached as high as +7.91%, fully demonstrating the recall capability of the IGDR module for small targets. On another dataset, View-of-Delft (VoD), this embodiment achieved an mAP of 66.69%, also reaching the state-of-the-art level. Figure 5 This is the visualization result of this embodiment.

[0157] The core of this invention is to construct a 4D millimeter-wave radar-camera fusion 3D object detection method based on panoramic depth fusion and temporal dynamic refinement, aiming to achieve high-precision, high-efficiency, and high-robustness 3D object detection. This method constructs a 4D millimeter-wave radar-camera 3D object detection model compatible with the Bird's-Eye View (BEV) paradigm, and designs three plug-and-play functional modules based on this model. These modules include: a panoramic depth fusion module for fusing multi-source perception information to generate high-quality panoramic absolute depth estimates; a deformable gated temporal fusion module for effectively fusing multiple temporal features without requiring vehicle pose information, thereby enhancing the temporal consistency of target perception; and an instance-guided dynamic optimization module for adaptive feature optimization processing for small-sized or sparse targets. Through the synergistic effect of these modules, this invention can significantly improve the accuracy, real-time performance, and robustness of 3D object detection in complex scenes. Experiments show that this invention achieves state-of-the-art performance on both the VoD and TJ4DRadSet datasets, verifying its effectiveness and generalization ability.

[0158] Another embodiment of the present invention provides a vehicle that integrates the above-mentioned target detection system including a 4D millimeter-wave radar-camera fusion 3D target detection neural network model.

[0159] Another embodiment of the present invention provides a computer device including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0160] Another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0161] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A multimodal 3D target detection method based on panoramic depth fusion and temporal dynamic refinement, wherein the method is executed on a target detection system equipped with a 4D millimeter-wave radar-camera fusion 3D target detection neural network model, the target detection system being equipped with a camera and a 4D millimeter-wave radar sensor, characterized in that, The neural network model includes an image feature encoder, a radar feature encoder, a 2D instance segmentation branch, a 3D detection head, a panoramic depth fusion module, a deformable gated temporal fusion module, and an instance-guided dynamic refinement module. The neural network model is optimized during the training phase based on a loss function system with multiple supervision signals. The specific steps of this method include: 1) Multimodal data input and basic feature extraction; It receives camera image data and 4D millimeter-wave radar point cloud data at the current moment, extracts image features and radar BEV features using image and radar feature encoders respectively, and outputs the 2D instance features and 2D instance mask of the current frame in parallel by the 2D instance segmentation branch. 2) Construct a panoramic depth fusion module and perform panoramic depth fusion processing to generate panoramic depth; A depth estimation network is constructed, which takes the extracted image features and radar BEV features as input, combines the radar's geometric cues and image semantics to predict the depth probability distribution, calculates the expected panoramic dense depth map based on the distribution, and projects the image features onto the 3D bird's-eye view space accordingly, and fuses them with the radar BEV features to generate the panoramic fused BEV features of the current frame. 3) Construct a deformable gated temporal fusion module to perform pose-independent temporal feature aggregation and temporal fusion processing; Read the historical frame temporal hidden state features stored in the temporal memory bank, input the current frame panoramic fusion BEV features and the historical frame temporal hidden state features into the deformable gating temporal fusion module. The module first performs motion alignment based on feature differences, and then fuses historical memory and updates temporal information through a gating mechanism to output temporal enhanced panoramic fusion BEV features. 4) Construct a dynamic refinement module for instance-guided programming and perform feature refinement processing for instance-guided programming; The temporal enhanced panoramic fusion BEV feature output in step 3) is used as the feature to be calibrated, and the 2D instance feature and 2D instance mask output in step 1) are used as the guiding feature input to the instance-guided dynamic refinement module. The instance-guided dynamic refinement module uses the 2D instance feature to generate a pure semantic prototype, performs feature enhancement and denoising on the object region in the temporal enhanced panoramic fusion BEV feature, and outputs a purified panoramic fusion BEV feature. 5) Target detection decoding output: Input the purified panoramic fusion BEV features obtained in step 4) into the 3D detection head, decode, calculate and output the 3D target detection boxes, category labels and velocity information of all targets.

2. The method as described in claim 1, characterized in that, Step 1) Multimodal data input and basic feature extraction specifically includes: 11) Acquire image data from cameras around the target detection system in real time, preprocess the images; input the preprocessed image data into the image feature encoder, extract the backbone of the encoder network and fuse it at multiple scales to output a multi-scale image feature map containing texture and semantic information. 12) Synchronously acquire point cloud data of the 4D millimeter-wave radar of the target detection system, including spatial coordinates, radial velocity and reflection intensity information; input the point cloud data into the radar feature encoder, and output radar BEV features containing spatial geometry and motion information; 13) The above image features are processed in parallel using the 2D instance segmentation branch. Region proposal is used to generate feature alignment operations, and the 2D instance-level feature vectors of the foreground targets in the current scene and the corresponding 2D instance masks are output, providing semantic guidance sources for the subsequent instance-guided dynamic refinement module.

3. The method as described in claim 1, characterized in that, The construction and execution process of the panoramic depth fusion module in step 2) specifically includes: 21) Receive the multimodal feature data output in step 1), specifically including encoded image semantic features and radar geometric features; 22) Construct a depth estimation network and adopt an attention-based feature interaction architecture. The network is configured to use radar features as geometric query indexes and image features as semantic key-value libraries. By combining the sparse geometric cues provided by radar features and the dense semantic context provided by image features, the network directly predicts the depth probability distribution in the view frustum space through cross-modal interaction. During the training phase, the network is explicitly constrained by the panoramic depth fusion loss system, which includes probability distribution supervision loss, global refinement supervision loss, and relative loss. 23) The depth probability distribution is multiplied by the image features, and the Z-axis is compressed by convolution before being concatenated with the original radar features to obtain semantically enhanced radar BEV features. The image 2D instance mask, depth probability distribution, and semantic features are used as inputs to the view transformation operation, which uses a 2D to 3D projection algorithm to project the image features from the perspective view space to the 3D bird's-eye view space to generate localized enhanced image BEV features. Subsequently, the image BEV features and radar BEV features are fused at the feature level to output geometrically enhanced panoramic fusion BEV features for the current frame.

4. The method as described in claim 1, characterized in that, In step 3), the construction and execution steps of the deformable gating timing fusion module specifically include: 31) Read the hidden state features of historical frames in the previous time-series memory; 32) Construct a motion-aware alignment branch. This branch is configured to not rely on external pose information. It uses deformable convolution to analyze the feature differences between the current frame and historical frames, predicts the spatial displacement deviation of feature points in time, and performs spatial distortion correction and registration on historical features based on the displacement deviation to compensate for the non-rigid misalignment of features caused by motion. 33) Construct a gated temporal update branch and introduce an adaptive temporal feature fusion mechanism. This mechanism is configured to evaluate the effectiveness of the current observed features and the historical aligned features, dynamically calculate the forgetting ratio of historical information and the updating ratio of current information, and generate temporally enhanced panoramic fusion BEV features through weighted fusion.

5. The method as described in claim 1, characterized in that, In step 4), the construction and execution steps of the instance-guided dynamic refinement module specifically include: 41) Construct an instance semantic prototype generation unit; The system receives instance feature vectors and 2D instance masks extracted from the 2D instance segmentation branch. First, it performs global average pooling on each instance feature vector to extract the instance semantic prototype. Then, it uses perspective projection to project the 2D instance mask onto the BEV plane to obtain an instance spatial distribution map. Finally, based on the position index of the distribution map, it broadcasts and maps the instance semantic prototype to the corresponding position in the BEV space to construct a clean instance semantic feature map with the same size as the panoramic fusion BEV feature map. 42) Construct a prototype-guided dynamic calibration unit; Using the clean instance semantic feature map generated in step 41) as a conditional guide, the affine transformation parameters for each pixel are dynamically predicted through a convolutional layer. The parameters include a scaling factor map and a bias factor map. The affine transformation parameters are used to perform pixel-by-pixel feature recalibration on the input panoramic fusion BEV features to be calibrated. The affine transformation is applied to enhance the feature response amplitude of the foreground region and suppress background noise, thereby generating calibrated panoramic fusion BEV features. 43) Construct a foreground gating fusion unit; A foreground gating map is generated based on the instance spatial distribution map in step 41). The calibrated panoramic fusion BEV features output in step 42) are weighted and fused with the original panoramic fusion BEV features to be calibrated using the foreground gating map. Specifically, the calibrated panoramic fusion BEV features are fused preferentially in the instance region indicated by the gating map, while the original panoramic fusion BEV features are retained in the background region, and the purified panoramic fusion BEV features are output.

6. The method as described in claim 1, characterized in that, In step 5), the target detection and decoding output includes the following process: 51) Receive the purified panoramic fusion BEV features after foreground gating fusion in step 4), and construct a 3D target detection head as the output layer of the system; the 3D detection head is configured to perform target decoding operation on the input purified panoramic fusion BEV features, and parse the target's category confidence information and three-dimensional spatial geometric attribute information including motion state in parallel from the feature map; finally, the category confidence information and three-dimensional spatial geometric attribute information are combined to generate and output a 3D target detection box.

7. The method as described in claim 1, characterized in that, The neural network model is optimized during the training phase based on a loss function system using multiple supervision signals. The specific loss function system is as follows: For the aforementioned panoramic depth fusion module, a sparse-dense complementary depth estimation loss is designed. ,include: (1) Probability distribution supervised loss The target distribution is constructed using sparse, high-precision radar points. By minimizing the divergence between the predicted distribution and the target distribution, the absolute depth accuracy of the foreground region is monitored. (2) Detailed global monitoring of losses It combines absolute depth constraints from sparse 4D millimeter-wave radar point cloud data with dense relative depth constraints generated by a depth estimation supervision model; used to constrain the prediction results of the depth estimation network. Specifically, a pre-trained depth estimation supervised model is used to process camera image data to generate dense depth pseudo-real values; the depth estimation supervised model is configured to extract relative depth relationships in the image; When calculating the global refinement supervision loss, on the one hand, the difference between the probability distribution of the depth estimation network and the probability distribution of the 4D millimeter-wave radar point cloud data projection is calculated; on the other hand, the scale-invariant depth error between the panoramic dense depth map and the dense depth pseudo-true value is calculated. (3) Relative loss We designed a pairwise ordinal ranking loss based on dynamic thresholds, specifically including edge sharpening strategies and global relative position strategies: morphological dilation of the 2D instance mask is performed, and pairwise pixel sampling is performed between the dilated region at the object edge (i.e., the background boundary that does not belong to the object's 2D instance mask) and the object's interior (i.e., the foreground core), forcing the network to learn the depth abrupt change relationship at the object's contour; pairwise pixel sampling is performed on the entire image to understand the relative positional relationships between large objects and the image as a whole, as well as between different parts of the image. Total loss of panoramic depth estimation fusion module for , and Weighted summation; For the 3D detection head module, a basic detection loss is constructed. The loss specifically includes: a classification loss used to supervise the prediction of 3D target categories, and a regression loss used to supervise the localization accuracy of 3D target detection boxes; For the 2D instance segmentation branch and the panoramic depth fusion module, a 2D instance segmentation and auxiliary perception loss is designed. Specifically, it includes: (1) 2D task loss applied to the 2D instance segmentation branch, including RPN network loss, RoI Head classification and regression loss and 2D instance mask prediction loss, to maintain the semantic purity of image features; (2) The auxiliary perception loss applied to the multimodal fusion module, including BEV segmentation loss and auxiliary loss in the encoding stage of 4D millimeter-wave radar point cloud data, is used to constrain the foreground distribution of the fused panoramic BEV features. The total loss function used for backpropagation during the training phase is the weighted sum of all the above losses for the unfrozen modules during the current training phase.

8. The method as described in claim 7, characterized in that, The neural network model employs a two-stage progressive training process, including: The first stage is pre-training of basic spatial perception features; The parameters of the detection head, deformable gated temporal fusion module, and instance-guided dynamic refinement module in the neural network model are frozen, and the remaining parts are subjected to perceptual training. The single-frame panoramic fusion BEV features are extracted, and the 2D instance segmentation branch is supervised by 2D annotation to pre-train the panoramic depth fusion module for depth estimation. Phase Two: Spatiotemporal End-to-End Fine-tuning; Load the pre-trained weights of the first stage, activate the deformable gated temporal fusion module and the instance-guided dynamic refinement module, start the alignment of temporal sequence and dynamic refinement fine-tuning, and start the 3D detection head for end-to-end model training.

9. A vehicle, characterized in that, The vehicle integrates the target detection system as described in claim 1.

10. A computer device, characterized in that, The method includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method as described in claim 1.

Images