Multi-modal time sequence fusion 3D target detection method based on instantiation sparse representation
By instantiating a sparse representation framework and using explicit-implicit co-motion modeling, and adaptively fusing cross-modal and cross-temporal information, the problems of high computational complexity and error amplification in existing technologies are solved, thereby improving the robustness and efficiency of 3D target detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-22
- Publication Date
- 2026-04-07
AI Technical Summary
Existing multimodal temporal fusion methods fail to fully exploit the potential for modal complementarity across spatiotemporal dimensions, resulting in amplified initial fusion errors and high computational complexity. Dense BEV feature representations are misaligned, affecting the robustness and efficiency of 3D target detection.
By constructing an instance-based sparse representation framework, using LiDAR and camera branches to independently extract features, generating complementary enhanced instances, and combining explicit-implicit co-motion modeling and a lightweight cross-attention mechanism, cross-modal and cross-temporal information is adaptively fused to achieve instance-level sparse representation.
It significantly reduces computational complexity, improves the accuracy and efficiency of 3D target detection, enhances the ability to capture the motion state of dynamic targets, strengthens multimodal complementary characteristics, and exhibits excellent detection performance.
Smart Images
Figure CN121811014A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of object detection in autonomous driving perception, and specifically relates to a multimodal temporal fusion 3D object detection method based on instantiated sparse representation. Background Technology
[0002] In today's rapidly developing intelligent transportation and smart mobility, autonomous driving has become a crucial frontier driving the widespread application of artificial intelligence and robotics. As a core component of autonomous driving, the perception system needs to accurately perceive and understand surrounding targets in complex and ever-changing environments. To ensure driving safety and reliability, autonomous driving perception systems typically rely on the collaborative work of multiple complementary sensors, such as LiDAR (Light Detection and Range), cameras, and millimeter-wave radar. Cameras capture dense two-dimensional images rich in semantics and texture, while LiDAR provides accurate but sparse three-dimensional point cloud information. The complementary advantages of both make LiDAR-camera the mainstream configuration for multimodal heterogeneous sensor fusion. To overcome the limitations of single-modality sensors, such as the sparsity of LiDAR and the lack of depth information in cameras, multimodal fusion detectors improve perception robustness by integrating sensor advantages, leading to a series of excellent research works. Among these, feature fusion through bird's-eye-view (BEV) space has become the mainstream paradigm. Simultaneously, understanding dynamic scenes requires modeling temporal information to capture the target's motion state, which has also spurred the development of temporal fusion methods. Therefore, multimodal temporal fusion, which combines multimodal perception with the temporal dimension, has become an emerging research direction for improving 3D object detection capabilities.
[0003] Existing multimodal temporal fusion methods generally separate multimodal feature alignment and temporal information aggregation into independent stages, forming a sequential process of "modal fusion first, then temporal modeling" or vice versa. This decoupling process, to some extent, disrupts the synergistic relationship between modal complementarity and temporal continuity. For example, BEVFusion4D adopts a strategy of completing multimodal spatial fusion first and then temporal aggregation, while FusionFormer separates multimodal encoding and temporal encoding into independent modules. Although these methods demonstrate good performance in benchmark tests, their design fails to fully exploit the potential of modal complementarity across spatiotemporal dimensions, affecting the efficiency of interaction between motion states and modal characteristics. Furthermore, such methods typically rely on fixed BEV features as intermediate representations. When there are deviations in camera depth estimation, the initial fusion error is usually amplified during temporal propagation and is difficult to correct. On the other hand, there is a misalignment between dense BEV features and instance-level temporal fusion in the representation space, and the computational cost of global attention mechanisms on dense features significantly restricts efficiency.
[0004] To address the aforementioned challenges, this patent proposes a multimodal temporal fusion 3D target detection method based on instanced sparse representation (TempoCross). By unifying multimodal features and temporal states into an instance-level sparse representation space, it adaptively fuses key information from different modalities and time periods, achieving more accurate and efficient multimodal temporal fusion 3D target detection.
[0005] TempoCross constructs a unified instantiation spatiotemporal representation framework. The LiDAR and camera branches first extract native modal features through independent backbone networks and generate complementary cross-modal enhanced instances to initialize high-quality queries. Each branch is equipped with a dynamic memory queue that continuously caches historical frame instance states, including position, motion attributes, and modality embeddings, aligning them to the current frame through a semi-explicit-implicit collaborative motion compensation mechanism. Finally, in the fusion detector, a lightweight cross-attention mechanism is applied to current frame instance candidates from different branches and their associated historical queue information to adaptively aggregate the most relevant cross-modal and spatiotemporal information within the instance space.
[0006] The above design abandons the paradigm of complex fusion on dense BEV features, effectively avoiding the propagation and amplification problem of initial depth estimation error, while significantly reducing computational complexity. By constraining the fusion operation to a sparse instance space highly correlated with the target, TempoCross can more accurately capture the motion state changes and multimodal complementary characteristics of dynamic targets, thereby improving the robustness and efficiency of 3D target detection. Summary of the Invention
[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a multimodal temporal fusion 3D target detection method based on instanced sparse representation. This method unifies multimodal features and temporal states into an instance-level sparse representation space, adaptively fusing key information to achieve more accurate and efficient detection. The multimodal temporal fusion 3D target detection method based on instanced sparse representation includes the following steps:
[0008] Step 1: Extract native modal features through independent backbone networks of LiDAR and camera branches, and generate complementary cross-modal augmented instances to initialize the query;
[0009] Step 2: Propose a hybrid motion modeling mechanism that combines explicit rigid body motion transformation with implicit learnable deformable residuals to perform motion compensation and alignment on historical frame instances.
[0010] Step 3: Perform time-aware enhancement on the current frame instance candidates and the motion-compensated historical frame information;
[0011] Step 4: Adaptively fuse cross-modal and cross-temporal instance information through a lightweight cross-attention mechanism to generate 3D object detection results.
[0012] Step 1 describes extracting native modal features through independent backbone networks of LiDAR and camera branches, and generating complementary cross-modal augmented instances to initialize the query;
[0013] The specific steps are as follows:
[0014] Step 1-1: For the camera branch, extract multi-view, multi-scale image features through the image backbone network to obtain a two-dimensional image feature set. in Let V represent the m-th scale feature, and let V, C, H, W represent the number of multi-viewpoints, feature dimension, image height, and image width, respectively.
[0015] Steps 1-2: For the LiDAR branch, extract and generate point cloud features in the bird's-eye-view (BEV) space using the LiDAR backbone network. Where X, Y, and C represent the grid and feature dimensions of the BEV space, respectively;
[0016] Steps 1-3: Through the cross-modal augmentation instance generation mechanism, the point cloud is projected onto the image coordinate system to generate a sparse depth image, which is then convolved with the image features and fused to form a visual augmentation instance. At the same time, the BEV space grid points are back-projected onto the image coordinate system to sample semantic features, which are then added to the LiDAR features and generated through deformable attention interaction to initialize a high-quality query.
[0017] Step 2 describes the proposed hybrid motion modeling mechanism, which combines explicit rigid body motion transformation with implicit learnable deformable residuals to perform motion compensation and alignment on historical frame instances.
[0018] The specific steps are as follows:
[0019] Step 2-1: Maintain a dynamic memory queue to continuously cache the instance states of the most recent historical frames, including position, motion attributes and modal embeddings, and retain the most confident instance objects for each frame;
[0020] Step 2-2: For position encoding, a semi-explicit modeling method is used, based on the 3D center point P of the object in the previous frame. t-1 Speed v t-1 Inter-frame coordinate transformation matrix The initial displacement is calculated using kinematic equations (containing a 4th-order homogeneous matrix R and T) and a time interval Δt. A multilayer perceptron (MLP) is then used to learn kinematic residuals for compensation, generating an aligned position code.
[0021] Steps 2-3: For instance feature encoding, an implicit modeling method is used. A neural network is used to generate affine transformation parameters Λ and B for historical instance features. Perform an affine transformation to obtain the compensated features. The formula is expressed as:
[0022]
[0023] Where Γ(·) is the positional coding function, and ζ1 and ζ2 are independent linear layers.
[0024] Step 3 involves performing time-aware enhancement by combining the current frame instance candidate with motion-compensated historical frame information;
[0025] The specific steps are as follows:
[0026] Step 3-1: In the camera branch, use the visual enhancement instance as the query and compare it with the native image features. Generate perspective view queries through 2D detector interaction. And mapped to BEV space via the view transformation module, along with motion-compensated historical frame instance information. Combine and output the candidate camera instances for the current frame.
[0027] Step 3-2: In the LiDAR branch, explicitly model historical instances using the Temporal-Enhanced Instance Integrator (TEII). Based on the spatiotemporal dependencies with the current instance, a Temporal-aware Instance (TAI) is generated, and a hybrid self-attention mechanism is used to focus on key historical information. The formula is as follows:
[0028]
[0029] Among them, Q Randn A spatial prior query for random initialization;
[0030] Step 3-3: Combine the output H of the hybrid self-attention with the LiDAR feature F of the current frame. L Cross-attention interaction is performed to further refine the spatial and feature representations of instances, and LiDAR instances of the current frame are generated using the detection head. Implement instance-space temporal information fusion of the LiDAR branch.
[0031] Step 4 describes the adaptive fusion of cross-modal and cross-temporal instance information through a lightweight cross-attention mechanism to generate 3D object detection results;
[0032] The specific steps are as follows:
[0033] Step 4-1: In the fusion detector, the current frame candidate queries of the LiDAR branch and the camera branch are concatenated;
[0034] Step 4-2: Enhance interaction through self-attention, then perform cross-attention fusion with historical frame instance information from LiDAR, camera, and fusion branches. Finally, generate 3D target detection results through the detection head, expressed by the formula:
[0035]
[0036] Here, f(·) and g(·) are learnable linear layers.
[0037] Beneficial effects of this invention:
[0038] 1. To avoid the problem of geometric deviation accumulation in traditional BEV feature alignment, this invention constructs a multimodal temporal unified fusion framework based on instance-level sparse representation, which significantly reduces the computational complexity of cross-modal and cross-temporal interactions.
[0039] 2. To provide high-quality temporal alignment information for subsequent fusion, this invention proposes an explicit-implicit collaborative motion modeling mechanism, which optimizes the temporal transfer of geometric motion and modal features. Specifically, geometric position is compensated for with high precision and semi-explicit motion by fusing physical motion equations with learnable residual terms.
[0040] 3. In order to overcome the problem that the deep semantic association and evolution pattern between historical instances and current frame queries are difficult to be spontaneously captured by standard attention mechanisms, this invention proposes Temporal Enhanced Instance Integrator (TEII). It explicitly models spatiotemporal dependencies and generates temporally aware instances through cascaded cross-attention and self-attention mechanisms, and introduces randomly initialized spatial priors to enhance the model's generalization ability, thereby improving the utilization efficiency of temporal context.
[0041] 4. This invention proposes TempoCross, a multimodal temporal fusion 3D object detector based on instanced sparse representations. By integrating LiDAR and camera modal features in a unified instance space, the detection query can adaptively and selectively focus on key modal and temporal information. TempoCross aligns historical frame instance information through semi-explicit-implicit hybrid cooperative motion modeling and combines lightweight cross-attention to adaptively fuse heterogeneous information across time and space, reducing the geometric bias and computational burden of multimodal temporal fusion in dense BEV features. On the nuScenes dataset, TempoCross outperforms mainstream baseline models with 74.1% mAP and 75.7% NDS. Attached Figure Description
[0042] Figure 1 This is a flowchart of a multimodal temporal fusion 3D target detection method based on instantiated sparse representation according to the present invention;
[0043] Figure 2 This is a flowchart illustrating step 1 of one embodiment of the present invention;
[0044] Figure 3 This is a flowchart illustrating step 2 of one embodiment of the present invention.
[0045] Figure 4 This is a flowchart illustrating step 3 of one embodiment of the present invention;
[0046] Figure 5 This is a flowchart illustrating step 4 of one embodiment of the present invention.
[0047] Figure 6 This is a summary flowchart of one embodiment of the present invention;
[0048] Figure 7 This is the result of comparing the present invention with mainstream models on the validation dataset;
[0049] Figure 8 This is the result of comparing the present invention with mainstream models on the test dataset;
[0050] Figure 9To unify the representation and interaction of features at different times and in different modalities in the instance space, this invention accurately captures the dynamic characteristics and multimodal complementary information of the target. Detailed implementation method:
[0051] An embodiment of the present invention will be further described below with reference to the accompanying drawings.
[0052] In this embodiment of the invention, a multimodal temporal fusion 3D target detection method based on instantiated sparse representations, such as... Figure 1 As shown, it includes the following steps:
[0053] Step 1 describes extracting native modal features through independent backbone networks of LiDAR and camera branches, and generating complementary cross-modal augmented instances to initialize the query;
[0054] Step 1-1: For the camera branch, extract multi-view, multi-scale image features through the image backbone network to obtain a two-dimensional image feature set. in Let V represent the m-th scale feature, and let V, C, H, W represent the number of multi-viewpoints, feature dimension, image height, and image width, respectively.
[0055] Steps 1-2: For the LiDAR branch, extract and generate point cloud features in the bird's-eye-view (BEV) space using the LiDAR backbone network. Where X, Y, and C represent the grid and feature dimensions of the BEV space, respectively;
[0056] Steps 1-3: Through the cross-modal augmentation instance generation mechanism, the point cloud is projected onto the image coordinate system to generate a sparse depth image, which is then convolved with the image features and fused to form a visual augmentation instance. At the same time, the BEV space grid points are back-projected onto the image coordinate system to sample semantic features, which are then added to the LiDAR features and generated through deformable attention interaction to initialize a high-quality query.
[0057] Step 2 describes the proposed hybrid motion modeling mechanism, which combines explicit rigid body motion transformation with implicit learnable deformable residuals to perform motion compensation and alignment on historical frame instances.
[0058] Step 2-1: Maintain a dynamic memory queue to continuously cache the instance states of the most recent historical frames, including position, motion attributes and modal embeddings, and retain the most confident instance objects for each frame;
[0059] Step 2-2: For position encoding, a semi-explicit modeling method is used, based on the 3D center point P of the object in the previous frame. t-1 Speed v t-1 Inter-frame coordinate transformation matrix The initial displacement is calculated using kinematic equations (containing a 4th-order homogeneous matrix R and T) and a time interval Δt. A multilayer perceptron (MLP) is then used to learn kinematic residuals for compensation, generating an aligned position code.
[0060] Steps 2-3: For instance feature encoding, an implicit modeling method is used. A neural network is used to generate affine transformation parameters Λ and B for historical instance features. Perform an affine transformation to obtain the compensated features. The formula is expressed as:
[0061]
[0062] Where Γ(·) is the positional coding function, and ζ1 and ζ2 are independent linear layers.
[0063] Step 3 involves performing time-aware enhancement by combining the current frame instance candidate with motion-compensated historical frame information;
[0064] Step 3-1: In the camera branch, use the visual enhancement instance as the query and compare it with the native image features. Generate perspective view queries through 2D detector interaction. And mapped to BEV space via the view transformation module, along with motion-compensated historical frame instance information. Combine and output the candidate camera instances for the current frame.
[0065] Step 3-2: In the LiDAR branch, explicitly model historical instances using the Temporal-Enhanced Instance Integrator (TEII). Based on the spatiotemporal dependencies with the current instance, a Temporal-aware Instance (TAI) is generated, and a hybrid self-attention mechanism is used to focus on key historical information. The formula is as follows:
[0066]
[0067] Among them, Q Randn A spatial prior query for random initialization;
[0068] Step 3-3: Combine the output H of the hybrid self-attention with the LiDAR feature F of the current frame. L Cross-attention interaction is performed to further refine the spatial and feature representations of instances, and LiDAR instances of the current frame are generated using the detection head. Achieve instance-space temporal information fusion in the LiDAR branch;
[0069] Step 4 describes the adaptive fusion of cross-modal and cross-temporal instance information through a lightweight cross-attention mechanism to generate 3D object detection results;
[0070] The specific steps are as follows:
[0071] Step 4-1: In the fusion detector, the current frame candidate queries of the LiDAR branch and the camera branch are concatenated;
[0072] Step 4-2: Enhance interaction through self-attention, then perform cross-attention fusion with historical frame instance information from LiDAR, camera, and fusion branches. Finally, generate 3D target detection results through the detection head, expressed by the formula:
[0073]
[0074] Here, f(·) and g(·) are learnable linear layers.
[0075] like Figure 7 As shown, TempoCross achieved a mean accuracy (mAP) of 72.6% and a nuScenes detection score (NDS) of 74.4% on the validation set. Detailed analysis shows that its mAP ranks first among similar multimodal temporal fusion methods, demonstrating the significant improvement in target detection accuracy. Its NDS score is also among the top, proving the superior overall detection quality.
[0076] like Figure 8 As shown, TempoCross achieved 74.1% mAP and 75.7% NDS on the test set, outperforming other mainstream baseline models that fuse LiDAR, camera, and temporal information, strongly demonstrating the effectiveness of the proposed multimodal temporal fusion strategy based on instantiated sparse representation.
[0077] like Figure 9 As shown, by unifying the representation and interaction of features from different times and modalities within the instance space, TempoCross can more accurately capture the dynamic characteristics and multimodal complementary information of the target. In particular, the hybrid motion modeling method proposed in this invention, combining explicit rigid body motion transformations with implicit learnable network parameters, effectively alleviates the alignment deviation problem between historical instances and the current frame in complex motion scenes, improving the robustness of cross-frame instance association. Furthermore, a lightweight cross-attention mechanism enables efficient fusion of current frame instance features with motion-compensated aligned historical frame information.
[0078] The above description is merely the most basic specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any substitutions that can be understood by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A multimodal temporal fusion 3D target detection method based on instantiated sparse representation, characterized in that, Includes the following steps: Step 1: Extract native modal features through independent backbone networks of (Light Detection and Ranging, LiDAR) and camera branches, and generate complementary cross-modal augmented instances to initialize the query; Step 2: Propose a hybrid motion modeling mechanism that combines explicit rigid body motion transformation with implicit learnable deformable residuals to perform motion compensation and alignment on historical frame instances. Step 3: Perform time-aware enhancement on the current frame instance candidates and the motion-compensated historical frame information; Step 4: Adaptively fuse cross-modal and cross-temporal instance information through a lightweight cross-attention mechanism to generate 3D object detection results.
2. The multimodal temporal fusion 3D target detection method based on instantiated sparse representation according to claim 1, characterized in that, Step 1 describes extracting native modal features through independent backbone networks of LiDAR and camera branches, and generating complementary cross-modal augmented instances to initialize the query; The specific steps are as follows: Step 1-1: For the camera branch, extract multi-view, multi-scale image features through the image backbone network to obtain a two-dimensional image feature set. in Let V represent the m-th scale feature, and let V, C, H, W represent the number of multi-viewpoints, feature dimension, image height, and image width, respectively. Steps 1-2: For the LiDAR branch, extract and generate point cloud features in the bird's-eye-view (BEV) space using the LiDAR backbone network. Where X, Y, and C represent the grid and feature dimensions of the BEV space, respectively; Steps 1-3: Through the cross-modal augmentation instance generation mechanism, the point cloud is projected onto the image coordinate system to generate a sparse depth image, which is then stitched with the image features and fused by convolution to form a visual augmentation instance; Simultaneously, the BEV spatial grid points are back-projected onto the image coordinate system to sample semantic features. After being added to LiDAR features, point cloud augmentation instances are generated through deformable attention interaction to initialize high-quality queries.
3. The multimodal temporal fusion 3D target detection method based on instantiated sparse representation according to claim 1, characterized in that, The hybrid motion modeling mechanism described in step 2; The specific steps are as follows: Step 2-1: Maintain a dynamic memory queue to continuously cache the instance states of the most recent historical frames, including position, motion attributes and modal embeddings, and retain the most confident instance objects for each frame; Step 2-2: For position encoding, a semi-explicit modeling method is used, based on the 3D center point P of the object in the previous frame. t-1 Speed v t-1 Inter-frame coordinate transformation matrix The initial displacement is calculated using kinematic equations (containing a 4th-order homogeneous matrix R and T) and a time interval Δt. A multilayer perceptron (MLP) is then used to learn kinematic residuals for compensation, generating an aligned position code. Steps 2-3: For instance feature encoding, an implicit modeling method is used. A neural network is used to generate affine transformation parameters Λ and B for historical instance features. Perform an affine transformation to obtain the compensated features. The formula is expressed as: Where Γ(·) is the positional coding function, and ζ1 and ζ2 are independent linear layers.
4. The multimodal temporal fusion 3D target detection method based on instantiated sparse representation according to claim 1, characterized in that, Step 3: Perform time-aware enhancement on the current frame instance candidates and motion-compensated historical frame information; The specific steps are as follows: Step 3-1: In the camera branch, use the visual enhancement instance as the query and compare it with the native image features. Generate perspective view queries through 2D detector interaction. And mapped to BEV space via the view transformation module, along with motion-compensated historical frame instance information. Combine and output the candidate camera instances for the current frame. Step 3-2: In the LiDAR branch, explicitly model historical instances using the Temporal-Enhanced Instance Integrator (TEII). Based on the spatiotemporal dependencies with the current instance, a Temporal-aware Instance (TAI) is generated, and a hybrid self-attention mechanism is used to focus on key historical information. The formula is as follows: Among them, Q Randn A spatial prior query for random initialization; Step 3-3: Combine the output H of the hybrid self-attention with the LiDAR features F of the current frame. L Cross-attention interaction is performed to further refine the spatial and feature representations of instances, and LiDAR instances of the current frame are generated using the detection head. Implement instance-space temporal information fusion of the LiDAR branch.
5. The multimodal temporal fusion 3D target detection method based on instantiated sparse representation according to claim 1, characterized in that, Step 4 adaptively fuses cross-modal and cross-temporal instance information through a lightweight cross-attention mechanism to generate 3D object detection results; The specific steps are as follows: Step 4-1: In the fusion detector, the current frame candidate queries of the LiDAR branch and the camera branch are concatenated; Step 4-2: Enhance interaction through self-attention, then perform cross-attention fusion with historical frame instance information from LiDAR, camera, and fusion branches. Finally, generate 3D target detection results through the detection head, expressed by the formula: Here, f(·) and g(·) are learnable linear layers.