A full sparse cross-modal three-dimensional target detection method based on hierarchical state space model

Abstract

The application discloses a full sparse cross-modal three-dimensional target detection method based on a hierarchical state space model, and belongs to the technical field of automatic driving three-dimensional perception. The method takes a sparse instance set of a camera and a laser radar as a unified intermediate representation, performs foreground enhancement and dynamic Token filtering on multi-modal sparse instances, and adds a reliability descriptor. After three-dimensional Hilbert space filling curve serialization and Z-axis multi-angle rotation compensation, the method realizes hierarchical bidirectional state space interaction through intra-modal encoding, cross-modal Token alignment and semantic auxiliary fusion, realizes adaptive weighted fusion of features through reliability gating of uncertainty perception, sets a residual enhancement branch for small target categories of sparse point clouds, and outputs fused and enhanced instance features. The application realizes cross-modal feature interaction with linear complexity while preserving the neighborhood topology of sparse instances, improves the positioning error and orientation error accuracy, and is suitable for full sparse multi-modal perception in long-distance and high-occlusion scenes.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving 3D perception technology, and specifically relates to a fully sparse cross-modal 3D target detection method based on a hierarchical state space model. It is particularly suitable for fully sparse multimodal perception in long-distance, high-occlusion scenarios, taking into account both feature interaction efficiency and cross-modal fusion accuracy. Background Technology

[0002] Fully sparse multimodal 3D object detection uses a sparse set of instances from cameras and LiDAR as a unified intermediate representation, avoiding the computational overhead of dense bird's-eye view feature maps that grow quadratically with the detection range. However, existing methods face two core challenges when performing cross-modal feature interaction at the fully sparse instance level: First, the discrete instances in the fully sparse instance representation are unordered, cross-modal interaction is costly, and the spatial neighborhood topology is easily lost when reducing the 3D discrete instances to the one-dimensional sequence required for the state space model, making it difficult to efficiently model global dependencies in long-distance scenarios; Second, cross-modal interaction often uses Transformers, creating a complexity bottleneck, and the fixed-threshold hard fusion strategy is difficult to adapt to the differences in confidence levels of heterogeneous modalities, easily leading to fusion inaccuracies in occlusion and cropping scenarios. These shortcomings directly threaten the detection accuracy and timeliness in long-distance and highly occluded scenarios. Summary of the Invention

[0003] To address the shortcomings of the existing technologies, the technical problem to be solved by this invention is to achieve efficient and topology-preserving cross-modal feature interaction at the fully sparse instance level. Specifically, it addresses the problems of spatial neighborhood topology loss and high complexity of cross-modal interaction caused by discrete instance serialization, as well as the problem of fusion inaccuracy easily generated by fixed threshold hard fusion strategy under heterogeneous modal confidence differences.

[0004] To achieve the above objectives, this invention provides a fully sparse cross-modal 3D target detection method based on a hierarchical state-space model. Using the sparse instance set of cameras and LiDAR as a unified intermediate representation, the method comprises four progressive stages: foreground enhancement and dynamic token filtering, where the top K tokens of multimodal sparse instances are retained based on foreground targetability scores and a reliability descriptor is added; sparse instance spatial serialization, where a 3D Hilbert space is filled with curve mapping and multi-angle rotation compensation along the Z-axis is applied to achieve one-dimensional serialization that preserves the neighborhood topology; hierarchical bidirectional state-space interaction, sequentially executing three stages: intramodal encoding (SSF), cross-modal token alignment (MTA), and semantic-assisted fusion (SAF), where SSF employs a bidirectional mapping from sequence to space and back to sequence to balance long-range dependencies and 3D local geometry; and reliability-gated fusion, where gating coefficients are used to adaptively weight and fuse the features of the two modalities to obtain fused and enhanced instance features.

[0005] The specific technical solution of the present invention is as follows: A fully sparse cross-modal 3D target detection method based on a hierarchical state-space model includes the following steps: Step 1: Acquire multimodal sensor data, including camera images and LiDAR point clouds; sequentially perform two-dimensional backbone feature extraction and instance segmentation on the camera images, and perform viewpoint cone projection based on the segmentation results to obtain a sparse set of camera instances; sequentially perform voxelization and sparse coding on the LiDAR point clouds to obtain a sparse set of LiDAR instances; use the camera sparse set and the LiDAR sparse set as a unified intermediate representation of full sparseness. Step 2: Perform foreground enhancement and dynamic token filtering on the sparse instance set of the camera and the sparse instance set of the LiDAR. Evaluate the instances by foreground targetability scoring and retain the top K tokens. At the same time, attach a reliability descriptor to each token to obtain the filtered instance set. Step 3: Perform sparse instance space serialization on the filtered instance set to obtain an ordered instance sequence; Step 4 involves performing hierarchical bidirectional state-space interaction on the ordered instance sequence, which includes three progressive stages: intramodal encoding, cross-modal token alignment, and semantic-assisted fusion, ultimately yielding camera modal features and LiDAR modal features. Intramodal encoding performs forward and reverse state-space scanning on both the camera and LiDAR modalities, and fuses the bidirectional states through a bidirectional mapping from sequence to space and back to sequence. Cross-modal token alignment uses a lightweight alignment network to predict positional offsets and corrects the spatial coordinates of the two modalities symmetrically. Semantic-assisted fusion predicts the semantic category distribution of each instance and performs semantic-guided rearrangement and context extraction accordingly. Step 5: Adaptively weightedly fuse camera modal features and lidar modal features using a reliability gating coefficient to obtain fused enhanced instance features and output them for use in downstream detection, tracking, or trajectory prediction tasks.

[0006] Compared with the prior art, the beneficial effects of the present invention are: (1) Regarding long-range perception efficiency, this invention uses fully sparse instances as a unified intermediate representation, with computational complexity approximately linear to the number of instances and decoupled from the detection range, avoiding the computational overhead of dense BEV feature maps that increases with the square of the distance. 3D Hilbert curve serialization and Z-axis multi-angle rotation compensation compensate for anisotropy while preserving the spatial neighborhood topology, resulting in linear complexity for global dependency modeling in long-range scenarios. Comparative experiments on the nuScenes and Argoverse 2 datasets validated these advantages: this invention improved the baseline FSF by 1.2 / 1.1 on nuScenes and by an average of 1.8 / 1.4 on Argoverse 2, and the computational overhead did not increase with the square of the distance when the detection range exceeded 200 meters.

[0007] (2) In terms of cross-modal interaction accuracy, this invention replaces Transformer with a three-stage hierarchical bidirectional state space interaction of "intramodal encoding - cross-modal token alignment - semantic-assisted fusion". The bidirectional mapping from sequence to space and back to sequence in SSF retains the three-dimensional local geometric structure while using the state space model to model long-range dependencies, which significantly improves the accuracy of positioning error and orientation error and is more friendly to downstream tracking and trajectory prediction tasks.

[0008] (3) In terms of fusion robustness, the present invention replaces the fixed threshold hard fusion strategy with the reliability gating coefficient adaptively changing with the confidence level of each modality, thereby alleviating the fusion inaccuracy caused by the difference in confidence level of heterogeneous modalities; foreground enhancement and dynamic token screening compress the number of background tokens while retaining effective target information, further reducing the interactive computation overhead. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the overall method framework of the present invention.

[0010] Figure 2 This is a flowchart of the overall process of the method of the present invention.

[0011] Figure 3 This is a schematic diagram of the hierarchical H-Mamba cross-modal feature interaction module structure of the present invention.

[0012] Figure 4 This diagram illustrates a comparison of the detection performance of our invention and mainstream methods on the nuScenes dataset.

[0013] Figure 5 This is a schematic diagram showing the average accuracy of the present invention across various categories on nuScenes.

[0014] Figure 6This is a schematic diagram comparing the present invention with mainstream methods in nuScenes for mAP, NDS, mATE, and mAOE.

[0015] Figure 7 This is a schematic diagram illustrating the ablation experimental trends at each stage of the hierarchical H-Mamba method of this invention.

[0016] Figure 8 This is a schematic diagram comparing the mAP of this invention with mainstream methods on the Argoverse 2 dataset for each category.

[0017] Figure 9 This is a schematic diagram comparing the computational overhead of the present invention with dense BEV fusion and FSF baseline at different sensing distances.

[0018] Figure 10 This is a schematic diagram of the three-dimensional object detection inference results (BEV bird's-eye view) of the present invention on nuScenes. Detailed Implementation

[0019] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0020] This invention proposes a fully sparse cross-modal 3D target detection method based on a hierarchical state-space model. Using a fully sparse instance set as a unified intermediate representation, it comprises four progressive stages: foreground enhancement and dynamic token filtering, sparse instance space serialization, hierarchical bidirectional state-space interaction, and reliability-gated fusion, forming a complete fully sparse cross-modal feature interaction chain. For example... Figure 1 and Figure 2 As shown, this invention generates camera sparse instance sets and LiDAR sparse instance sets from camera images and LiDAR point clouds, respectively. After foreground filtering and Hilbert serialization, the fused enhanced instance features are output for downstream tasks through three-stage bidirectional state space interaction and reliability-gated fusion, which includes intra-modal encoding, cross-modal token alignment and semantic-assisted fusion.

[0021] Following the above technical solution, refer to Figure 2 The present invention specifically includes the following steps: Step 1, Multimodal Sparse Instance Extraction Two types of multimodal sensor data are acquired: camera images and LiDAR point clouds. For camera images, a two-dimensional backbone network is used to extract multi-scale two-dimensional features, perform instance segmentation, and generate two-dimensional bounding boxes and masks for image targets based on the instance segmentation results. Subsequently, the two-dimensional bounding boxes are back-projected along the camera intrinsic parameters to form a view cone in three-dimensional space. The LiDAR point clouds falling within the same view cone are then aggregated with the pseudo-point clouds generated by the image feature projection to form a sparse instance set for the camera. For the LiDAR point cloud, a sparse voxel encoder is used for voxelization. Sparse 3D convolution and point voting are performed on non-empty voxels to generate instance centers, forming a sparse instance set of the LiDAR. Camera Sparse Instance Set With LiDAR sparse instance set Each instance in the system is represented in the form of a token, and each token carries the instance features of the corresponding modality, serving as a unified intermediate representation for subsequent modules. This avoids the computational overhead of constructing a dense bird's-eye view feature map that grows quadratically with the detection range.

[0022] Step 2, foreground enhancement and dynamic token filtering.

[0023] For the first Instance Token From the foreground goal-oriented scoring function The foreground relevance is assessed by combining the semantic category confidence of its instances with the 2D detection score. And retain the top scores in descending order. The Tokens are obtained as a filtered set of instances. To compress the number of background tokens while retaining valid target information, where For the set of multimodal sparse instances to be filtered, Based on the retention ratio The number of tokens to be retained, determined by the size of the set, i.e. The proportion retained in this embodiment Set the value to 0.75, and simultaneously set the minimum number of tokens and the minimum number of tokens to be retained per modality to ensure that both modalities are covered; further configure the quota by category group, and set the minimum number of tokens to be retained for small target category groups to avoid small target categories being filtered out as a whole due to low scores.

[0024] This invention relates to a set Each token is appended with a 9-dimensional reliability descriptor The 9-dimensional reliability descriptor It consists of a basic 6-dimensional and an extended 3-dimensional system: the basic 6-dimensional system consists of normalized center distance, modality label, class confidence, class entropy, 2D detection score, and 2D valid label; the extended 3-dimensional system consists of range confidence (1 minus normalized distance), 2D bounding box aspect ratio, and projection consistency (the product of 2D detection score and 2D valid label), which are used to measure the credibility of each modality for the instance during subsequent gating fusion; among them, class entropy characterizes the uncertainty of the prediction distribution, and 2D detection score and 2D valid label are derived from 2D detection prior.

[0025] Step 3, sparse instance space serialization.

[0026] This invention uses a three-dimensional Hilbert space-filling curve to map discrete three-dimensional instances into a one-dimensional ordered sequence with local neighborhood preservation. Specifically, it maps the three-dimensional spatial coordinates of each instance in the filtered instance set. First press Bit quantization is normalized to a discrete grid (in one embodiment, ...). Then calculate its three-dimensional Hilbert mapping index. The batch identifier is then shifted left and appended to the high-order bits of the index to isolate instances from different batches. The resulting sequence is then sorted in ascending order by the final index to obtain a one-dimensional ordered sequence that preserves the local neighborhood topology. The three-dimensional Hilbert curve, as a space-filling curve, uses a recursive and circuitous path to ensure that spatially adjacent instances are also adjacent in the sequence, thereby mitigating the loss of spatial topology when three-dimensional discrete instances are reduced to a one-dimensional sequence.

[0027] Building upon this, Z-axis multi-angle rotation compensation is introduced. A region-to-global sliding window organization strategy is applied to the one-dimensional ordered sequence. Specifically, the rotation angle is randomly sampled. The instance coordinates are rotated to compensate for the anisotropy of the space-filling curves under different orientations. For each rotation angle, the 3D Hilbert mapping index is recalculated and arranged in ascending order to obtain the locally ordered sequence corresponding to that rotation angle. Subsequently, a region-to-global sliding window organization strategy is adopted. The locally ordered sequence is divided into several local sub-sequence regions with a fixed sliding window size. Each region is kept ordered according to its local Hilbert index to preserve local neighborhood details. The local sub-sequences are then concatenated into a globally ordered sequence according to the region index order to account for global dependencies. A serialized copy is generated for each rotation angle, ultimately yielding the ordered instance sequence.

[0028] Step 4: Hierarchical bidirectional state space interaction.

[0029] This step breaks down the interaction into three progressive stages: intramodal encoding (SSF), cross-modal token alignment (MTA), and semantic-assisted fusion (SAF), structured as follows: Figure 3 As shown.

[0030] In SSF, a selective state-space model is employed, performing forward and backward scanning for both the camera and LiDAR modes, respectively. The state-space interaction module in this embodiment consists of an input projection layer, a depthwise separable one-dimensional convolution, an input-dependent projection layer, a stride projection layer, a parameterized state matrix, residual direct connections, and an output normalized projection layer. The input projection layer projects instance features into two paths: a data stream and a gated stream. The depthwise separable one-dimensional convolution (kernel size...) Aggregate local context; input depends on projection layer predicting time steps from data flow. Input matrix With output matrix ,in Controlling the rate of memory decay Controls the writing of new information to the gating system to filter out view mirror noise. Controlling hidden state readout gating to coordinate geometry and semantics; stride projection layer will Upgrade to internal dimension; state transition matrix Log-parameterized The values ​​are stored and initialized to negative real numbers to ensure stability. This embodiment employs a selective state-space model. , , All input features are predicted as input dependencies by a projection network, enabling the model to dynamically filter mirror noise and complete sparse LiDAR features. Its continuous-time state equation is shown in equation (1): in For a jump-connected matrix, It is in a hidden state. The instance feature sequence input into the state space model is at the 1st... The input at time step , i.e., the instance token features, is projected into a data stream by the input projection layer and then aggregated with local context by a depthwise separable one-dimensional convolution. The resulting input sequence is at time step . The value at each time step, For the state-space model in the th The output features at each time step, i.e., those derived from the hidden state Output matrix Projection and connected to the jump matrix Direct input The superimposed encoded features; a zero-order hold is used with time steps. Discretization yields , The discrete-time update is shown in the following equation: in , These are the forward and reverse hidden states, respectively. For modality The bidirectional splicing state. Discrete time step The input, i.e., the input in the continuous-time state equation Consistent, the instance token features are projected into a data stream by the input projection layer, and the resulting input sequence is aggregated with local context through depthwise separable one-dimensional convolution. The value of each time step.

[0031] SSF further achieves sequence-to-space and then-to-sequence through a bidirectional mapping from region to global: first, a local state space module performs a local scan of the sequence using a double-offset sliding window strategy, where the sliding window size is... The offset step size is The outputs of the non-offset sliding window and the offset sliding window are averaged to cover the window boundaries and alleviate the segmentation truncation effect; then, a one-dimensional spatial convolution (with a kernel size of 3 in this example) maps the sequence features to the spatial domain to capture the three-dimensional local geometry; finally, the global state space module performs a global scan on the fused sequence, progressing from local to global, forming a bidirectional mapping from sequence to space and back to sequence, preserving the three-dimensional local geometric structure while using the state space model to model long-range dependencies.

[0032] In MTA, the process is as follows: First, a cross-modal cost matrix is ​​constructed based on the Euclidean distance between the center of instances in the camera modality and the LiDAR modality. Then, semantic similarity is calculated using the semantic category probabilities of the two modalities to modulate the cost (higher semantic similarity results in lower cost). Next, bidirectional nearest neighbor matching is used to select a cross-modal alignment partner for each instance. Specifically, for each camera instance, the LiDAR instance with the lowest cost in the LiDAR modality is selected as the alignment partner. Similarly, for each LiDAR instance, the camera instance with the lowest cost in the camera modality is selected as the alignment partner, ensuring each instance has one cross-modal alignment partner. The alignment network predicts the alignment offset based on its own features, partner features, relative center coordinates, and semantic score. This offset is then applied to its own features using a residual weighting method based on distance decay (the closer the relative center distance, the greater the alignment weight). in, For the first The 3D instance center coordinates of each instance. To align partners The difference between the three-dimensional instance center coordinates and the coordinates of the instance. The coordinates are relative to the center. This is the alignment offset prediction function implemented by the alignment network, whose input is its own features. Partner characteristics Relative center coordinates With semantic score The concatenation outputs the alignment offset. For the first Instance features obtained by intramodal encoding of an instance; To align partners Instance features obtained through intramodal encoding; For semantic score, Aligned instance features are used to alleviate geometric ambiguity caused by 2D to 3D projection.

[0033] In SAF, the following procedure is followed: the semantic confidence score is determined by taking the maximum class probability from the semantic class distribution of each instance. via semantic projection network The aligned interaction features are projected and augmented using semantic confidence as weights in a residual manner. This process enhances the contextualization of instances with high semantic consistency. Furthermore, since multi-angle rotation compensation along the Z-axis generates multiple rotated copies, each copy is sequentially processed by SSF, MTA, and SAF, then restored in reverse order and stacked for averaging to obtain multi-rotation aggregated features. These features are then used to fuse the serialized results of space-filling curves under different orientations, improving robustness to instance anisotropy. The multi-rotation aggregated features obtained in this step are the final instance features of the corresponding modality obtained after hierarchical bidirectional state-space interaction, which are also the camera modality features and LiDAR modality features participating in reliability-gated fusion in step 5.

[0034] Step 5, reliability gating fusion.

[0035] This step involves adaptively weighting and fusing camera modal features and LiDAR modal features using a reliability-gated coefficient to obtain and output the fused enhanced instance features. The adaptive weighted fusion is performed as follows: Each instance is defined by its reliability descriptor. The gating coefficients are calculated using a gating network and Sigmoid activation, as shown in equation (3): in , These are the gated network parameters; then the camera modal features are evaluated using the gate coefficients. With lidar modal characteristics Adaptive weighted fusion, as shown in equation (4): in For element-wise product, To integrate and enhance instance features; in this embodiment, the two features in equation (4) correspond to residual features (filtered instance features) and interactive features (multi-rotation aggregation features), respectively.

[0036] This embodiment further employs uncertainty-aware gating fusion: residual features, post-interaction features, and reliability descriptors are concatenated into a context, and the gating head predicts the gating coefficients. And the log-variance is predicted by the variance head. (truncation to) ), and calculate the inverse variance weight accordingly. Mixing coefficient with uncertainty perception : Finally, the fused and enhanced instance features are obtained. : in The residual features of the filtered instances obtained in step 2 are obtained through skip connections and direct connections without hierarchical interaction. The multi-rotation aggregated features obtained in step 4 through intra-modal encoding, cross-modal token alignment, and semantic-assisted fusion, along with the reliability descriptor of the instance, are... The input is concatenated into a context; the log-variance is predicted from the variance head (truncated to a preset range), and the inverse variance weight and uncertainty-aware mixing coefficient are calculated accordingly. In this step, the log-variance characterizes the uncertainty of each modality for this instance, giving higher weights to reliable modalities and automatically attenuating them on uncertain modalities. This replaces the hard fusion strategy with a fixed threshold with a differentiable adaptive weighting, mitigating fusion inaccuracies caused by differences in confidence levels among heterogeneous modalities.

[0037] Furthermore, for small target categories (such as bicycles, motorcycles, traffic cones, etc., which are sparse point cloud categories), this embodiment sets up a small target residual enhancement branch: the small target mask identifies instances belonging to the small target category, the small target head predicts feature increments, orientation residuals, scale residuals, and temperature coefficients, and the feature increments are enhanced by applying hyperbolic tangent activation with the small target residual scale as a scaling factor, and the fused features are enhanced in a residual manner. At the same time, the orientation residuals, scale residuals, and temperature coefficients are output for the detection head to use, so as to alleviate the orientation and scale estimation bias caused by the sparse point cloud of small targets.

[0038] After the above five stages, a fused and enhanced instance feature set is output as a unified intermediate representation for downstream detection, tracking, or trajectory prediction tasks.

[0039] To verify the effectiveness of the present invention, the following specific examples are provided.

[0040] 1. Dataset (1) nuScenes dataset (focusing on complex urban congestion scenarios): This dataset contains a large number of urban road environments, characterized by dense targets, severe occlusion, and rich categories. Using this dataset, we will focus on verifying the robustness and accuracy improvement of the model when dealing with severely occluded and crowded samples.

[0041] (2) Argoverse 2 dataset (focusing on high-speed and long-distance scenarios): Compared to traditional datasets, Argoverse 2 provides LiDAR detection data at longer distances (usually exceeding 200 meters). This dataset will be used as the core test to verify the advantages of the model based on a fully sparse architecture and H-Mamba interaction in avoiding the explosion of feature map size in long-distance BEV, and to test its ability to maintain linear computational complexity under large-scale perception.

[0042] 2. Evaluation Indicators mAP (mean accuracy): As a core basic indicator for object detection, it measures the overall detection accuracy of the model across various categories.

[0043] NDS (nuScenes Detection Score): A comprehensive evaluation metric proposed by nuScenes. It not only includes mAP, but also integrates True Positive metrics such as translation error (mATE), scale error (mASE), orientation error (mAOE), velocity error (mAVE), and attribute error (mAAE), which can more comprehensively reflect the physical and geometric quality of 3D detection boxes.

[0044] Algorithm efficiency metrics include: Inference Latency / FPS: Records the end-to-end time taken for the model to process a single frame of multimodal data, verifying whether it can meet the real-time requirements of autonomous driving.

[0045] GPU Memory Footprint: Under the same batch size, the peak memory consumption of different models is compared to verify the outstanding advantages of the fully sparse architecture and the Mamba module in terms of lightweight design.

[0046] 3. Quantitative analysis The hierarchical H-Mamba cross-modal feature interaction method of this invention (denoted as FSF_HierMamba) is compared with the current mainstream single-modal and multi-modal 3D object detection methods on the nuScenes and Argoverse 2 datasets, respectively.

[0047] Table 1. Comparison of the algorithm with existing algorithms on the nuScenes dataset. Method Modal mAP↑ NDS↑ mATE↓ mAOE↓ FSD L 62.5 68.7 — — TransFusion C+L 68.9 71.6 0.259 0.359 DeepInteraction C+L 70.2 72.9 0.261 0.329 FSF C+L 70.8 73.4 0.257 0.325 Ours C+L 72.0 74.5 0.258 0.308 Experimental results on the nuScenes dataset demonstrate that this invention... and All of these methods outperform the baseline FSF and all comparison methods. The baseline FSF was increased from 70.8 to 72.0. The accuracy improved from 73.4 to 74.5; meanwhile, the directional error... Significantly superior to all comparison methods in both aspects. The decrease from 0.325 to 0.308 confirms the differentiated contribution of the hierarchical H-Mamba's "sequence → space → sequence" bidirectional mapping mechanism to long-range spatial relationship modeling, making it more suitable for downstream tracking and trajectory prediction tasks.

[0048] Table 2. APs by category in nuScenes Class AP bus 78.51 car 67.65 truck 64.88 barrier 58.19 bicycle 57.66 motorcycle 57.51 trailer 54.48 pedestrian 54.48 construction_vehicle 51.96 traffic_cone 45.80 In the nuScenes data, large target categories such as bus, car, and truck have high AP, while small targets and sparse categories such as bicycle, motorcycle, and traffic_cone also achieve reasonable accuracy, verifying the detection capability of this invention for small targets and sparse categories.

[0049] Table 3. Comparison of mAP between the algorithm and existing algorithms on the Argoverse 2 dataset. method average vehicle the bus pedestrian truck motorcycle bike logo guardrail CenterPoint 13.5 61.0 36.0 33.0 26.0 12.5 9.5 6.5 4.0 VoxelNet 30.5 72.0 39.7 63.2 64.5 44.9 40.7 7.3 30.5 FSD 28.2 68.1 40.9 59.0 42.6 49.0 38.6 11.9 22.0 FSF 33.2 70.8 44.1 60.8 50.9 60.9 47.6 12.2 27.5 Ours 35.0 71.6 45.8 63.5 52.4 63.2 50.1 12.8 29.0 Table 4. Comparison of the algorithm with existing algorithms on the Argoverse 2 dataset using CDS. method average vehicle the bus pedestrian truck motorcycle bike logo guardrail CenterPoint 17.6 57.2 32.0 35.7 31.0 28.2 19.6 4.8 16.7 VoxelNet 22.7 57.7 34.2 47.5 34.4 41.4 32.0 9.5 16.5 FSD 25.5 59.6 35.6 48.5 40.1 48.9 37.2 9.2 19.0 FSF 25.8 58.4 36.0 49.2 39.5 50.1 38.8 8.4 18.6 Ours 27.2 60.2 37.4 51.6 41.8 52.3 40.5 9.0 20.2 Tables 3 and 4 show that the present invention is effective on the Argoverse 2 dataset. and All are superior to baseline FSF and all comparison methods, with an average The baseline FSF increased from 33.2 to 35.0, and the average... The performance improved from 25.8 to 27.2; the gains were particularly significant for distant small target categories such as pedestrians, motorcycles, and bicycles, which confirms the characteristics of the fully sparse architecture, which uses the number of instances as a linear complexity measure and is decoupled from the detection range, as well as the advantages of hierarchical bidirectional state space interaction in long-distance global dependency modeling, avoiding the computational overhead of dense bird's-eye view feature maps that increase with the square of the distance.

[0050] like Figure 7 As shown, ablation of the four components of the hierarchical H-Mamba cross-modal feature interaction method is performed on the nuScenes dataset. The configurations are as follows: Baseline (Base HierMamba), adding SSF bidirectional Mamba, adding MTA cross-modal alignment, adding SAF semantic assistance, and adding reliability gating. Experimental results show that the five configurations are effective. and Monotonically increasing, in and The monotonically decreasing trend, with no negative interference at any stage, confirms the reasonable design of the coupling relationship between the four components; among them, the reliability gating stage contributes the largest single-item value. The gain validates the necessity of "gating instead of hard fusion" in scenarios with varying confidence levels across heterogeneous modalities. This ablation result also demonstrates that the four stages—intra-modal encoding, cross-modal alignment, semantic-assisted fusion, and reliability gating—are progressively coordinated from local to global levels and from intra-modal to cross-modal perspectives, and none can be omitted.

[0051] 4. Visualization Results like Figure 4 The diagram shows a comparison of the detection performance of various methods on the nuScenes dataset, with the results of FSD, TransFusion, DeepInteraction, FSF, and the proposed FSF_HierMamba method presented in that order. The visual comparison demonstrates that the proposed method is superior in terms of target bounding box localization accuracy and orientation estimation, with smaller translation and orientation deviations between the bounding box and the ground truth. This intuitively verifies the ability of the hierarchical H-Mamba cross-modal feature interaction's "sequence → space → sequence" bidirectional mapping to improve long-range spatial relationship modeling.

[0052] Figure 5 To achieve the average accuracy of this invention across all categories on nuScenes, large target categories such as car, bus, and truck have higher AP, while small targets such as bicycle, motorcycle, and traffic_cone, as well as sparse categories, also achieve reasonable accuracy.

[0053] Figure 6 The present invention is compared with mainstream methods in terms of mAP, NDS, mATE, and mAOE on nuScenes. Consistent with Table 1, the present invention achieves mAP of 72.0 and NDS of 74.5, which are superior to all the comparison methods such as FSD, TransFusion, DeepInteraction, and FSF baseline. It also significantly outperforms all the comparison methods in terms of positioning error mATE (0.242) and orientation error mAOE (0.308).

[0054] Figure 8 To compare the mAP of this invention with mainstream methods on the Argoverse 2 dataset for each category, this invention outperforms all the comparison methods in categories such as vehicles, pedestrians, motorcycles, and bicycles, with the average mAP increasing from 33.2 of the baseline FSF to 35.0.

[0055] Figure 9The computational overhead of this invention compared with that of dense BEV fusion and FSF baseline at different sensing distances shows that the computational overhead of dense BEV fusion increases quadratically with sensing distance. The computational overhead of this invention, based on hierarchical state space interaction with fully sparse instances, is approximately linear with the number of instances and decoupled from the detection range, showing significant advantages in long-distance scenarios of over 200 meters.

[0056] Figure 10 The image shows the 3D target detection inference results (BEV bird's-eye view) of the FSF_HierMamba model on nuScenes. The lower half is the LiDAR point cloud (colored by height). Solid boxes represent targets detected by the present invention, and dashed boxes represent targets missed. Some targets were not detected due to occlusion and distance, reflecting the difficulty of detection in real-world scenarios and the completeness of the present invention's detection.

Claims

1. A fully sparse cross-modal 3D target detection method based on a hierarchical state-space model, characterized in that, Includes the following steps: Step 1: Acquire multimodal sensor data, which includes camera images and LiDAR point clouds; sequentially perform two-dimensional backbone feature extraction and instance segmentation on the camera images, and perform view cone projection based on the segmentation results to obtain a sparse set of camera instances; sequentially perform voxelization and sparse coding on the LiDAR point clouds to obtain a sparse set of LiDAR instances. The sparse instance set of the camera and the sparse instance set of the lidar are used as a fully sparse unified intermediate representation. Step 2: Perform foreground enhancement and dynamic token filtering on the sparse instance set of the camera and the sparse instance set of the LiDAR. Evaluate the instances by foreground targetability scoring and retain the top K tokens. At the same time, attach a reliability descriptor to each token to obtain the filtered instance set. Step 3: Perform sparse instance space serialization on the filtered instance set to obtain an ordered instance sequence; Step 4: Perform hierarchical bidirectional state space interaction on the ordered instance sequence, which includes three progressive stages: intramodal encoding, cross-modal token alignment, and semantic-assisted fusion, and finally obtain camera modal features and lidar modal features. In the intramodal coding, both the camera mode and the lidar mode undergo forward and reverse state space scanning, respectively, and the bidirectional states are fused through a bidirectional mapping from sequence to space and back to sequence. Cross-modal token alignment is achieved by a lightweight alignment network that predicts positional offsets and corrects the spatial coordinates of the two modalities in a symmetrical manner; semantic-aided fusion predicts the semantic category distribution of each instance and performs semantic-guided rearrangement and context extraction accordingly. Step 5: Adaptively weightedly fuse camera modal features and lidar modal features using a reliability gating coefficient to obtain fused enhanced instance features and output them for use in downstream detection, tracking, or trajectory prediction tasks.

2. The fully sparse cross-modal 3D target detection method based on a hierarchical state-space model according to claim 1, characterized in that, In step 1, a two-dimensional backbone network is used to extract multi-scale two-dimensional features from the camera image, and two-dimensional bounding boxes and masks of the image targets are generated based on the instance segmentation results. Then, the two-dimensional bounding boxes are back-projected along the camera intrinsic parameters to form a view cone in three-dimensional space, and the LiDAR point cloud falling within the same view cone is aggregated with the pseudo point cloud generated by the image feature projection to obtain the sparse instance set of the camera. The lidar point cloud is voxelized using a sparse voxel encoder, and sparse 3D convolution and point voting are performed on non-empty voxels to obtain instance centers, thus forming the lidar sparse instance set. The sparse instance set of the camera With the sparse instance set of the lidar Each instance in the token is represented by a token and carries the instance characteristics of the corresponding modality.

3. The fully sparse cross-modal 3D target detection method based on a hierarchical state-space model according to claim 1, characterized in that, In step 2, the foreground enhancement and dynamic token filtering are performed as follows: For the first... Instance Token From the foreground goal-oriented scoring function The foreground relevance is assessed by combining the semantic category confidence of its instances with the 2D detection score. And retain the top scores in descending order. The Tokens are obtained as a filtered set of instances. ,in For the set of multimodal sparse instances to be screened, set the minimum number of tokens and the minimum number of tokens to be retained for each modality to ensure that both modalities are covered, and configure quotas by category grouping. Set the minimum number of tokens to be retained for small target category groups to avoid small target categories being filtered out as a whole due to low scores.

4. The fully sparse cross-modal 3D target detection method based on a hierarchical state-space model according to claim 3, characterized in that, The reliability descriptor It is a 9-dimensional vector, consisting of a basic 6-dimensional vector and an extended 3-dimensional vector: the basic 6-dimensional vector consists of normalized center distance, modality label, class confidence, class entropy, 2D detection score, and 2D valid label; the extended 3-dimensional vector consists of range confidence, 2D bounding box aspect ratio, and projection consistency, which are used to measure the credibility of each modality to the instance during subsequent gating fusion.

5. The fully sparse cross-modal 3D target detection method based on a hierarchical state-space model according to claim 1, characterized in that, In step 3, the sparse instance space serialization uses a three-dimensional Hilbert space filling curve to map discrete three-dimensional instances into a one-dimensional ordered sequence with local neighborhood preservation. Specifically, this is performed as follows: For each instance in the filtered instance set, the three-dimensional spatial coordinates... First press Bit quantization is normalized to a discrete mesh, and then its three-dimensional Hilbert mapping index is calculated. The batch identifier is then shifted left and appended to the high-order bits of the index to isolate instances from different batches. The resulting sequence is then sorted in ascending order by the final index to obtain a one-dimensional ordered sequence that preserves the local neighborhood topology. .

6. The fully sparse cross-modal 3D target detection method based on a hierarchical state-space model according to claim 5, characterized in that, In step 3, a Z-axis multi-angle rotation compensation and a region-to-global sliding window organization strategy are applied to the one-dimensional ordered sequence, as follows: randomly sample the rotation angle. The instance coordinates are rotated to compensate for the anisotropy of the space-filling curves under different orientations. The 3D Hilbert mapping index is recalculated for the rotated coordinates at each rotation angle and arranged in ascending order of the index to obtain the locally ordered sequence corresponding to that rotation angle. Then, a region-to-global sliding window organization strategy is adopted to divide the locally ordered sequence into several local sub-sequence regions with a fixed sliding window size. Each region is kept ordered according to its local Hilbert index to preserve local neighborhood details. The local sub-sequences are then concatenated into a globally ordered sequence according to the region index order to take into account global dependencies. A serialized copy is generated for each rotation angle, and finally, the ordered instance sequence is obtained.

7. The fully sparse cross-modal 3D target detection method based on a hierarchical state-space model according to claim 1, characterized in that, In step 4, the intramodal encoding employs a selective state-space model, performing forward and reverse scanning for both the camera mode and the lidar mode, with the time step being... Input matrix Output matrix All input features are predicted as input dependencies by a projection network, and the state transition matrix is... Log-parameterized Store and initialize to negative real numbers to ensure stability; Its continuous-time state equation is: in For a jump-connected matrix, It is in a hidden state. For the first Input at each time step, For the state-space model in the th Output features at each time step; Using a zero-order hold with time steps Discretization yields , The discrete-time update is as follows: in , These are the forward and reverse hidden states, respectively. For modality The bidirectional splicing state; The cross-modal token alignment is performed as follows: First, a cross-modal cost matrix is ​​constructed based on the Euclidean distance between the centers of the camera modality and the lidar modality instances. Then, the semantic similarity is calculated using the semantic category probabilities of the two modalities to modulate the cost; the higher the semantic similarity, the lower the cost. Subsequently, for each camera instance, the lidar instance with the lowest cost in the lidar modality is selected as the alignment partner, and for each lidar instance, the camera instance with the lowest cost in the camera modality is selected as the alignment partner independently. The alignment network predicts the alignment offset based on its own features, partner features, relative center coordinates, and semantic score, and applies it to its own features in a residual manner according to the distance decay weight. For the first The 3D instance center coordinates of each instance. To align partners The difference between the three-dimensional instance center coordinates and the coordinates of the instance. The coordinates are relative to the center. This is the alignment offset prediction function implemented by the alignment network, whose input is its own features. Partner characteristics Relative center coordinates With semantic score The concatenation outputs the alignment offset. For the first Instance features obtained by intramodal encoding of an instance; To align partners Instance features obtained through intramodal encoding; For semantic score, Aligned instance features to alleviate geometric ambiguity caused by 2D to 3D projection; The semantic-assisted fusion is performed as follows: the maximum category probability from the semantic category distribution of each instance is taken as the semantic confidence level. via semantic projection network The aligned interaction features are projected, and semantic confidence is used as the weight to enhance the interaction features using residuals. Since Z-axis multi-angle rotation compensation generates multiple rotation copies, each rotation copy is sequentially processed by intramodal encoding, cross-modal token alignment and semantic-assisted fusion, and then restored in reverse order and stacked to take the average, resulting in multi-rotation aggregate features. The multi-rotation aggregate features are the final instance features of the corresponding modality obtained after hierarchical bidirectional state space interaction.

8. The fully sparse cross-modal 3D target detection method based on a hierarchical state-space model according to claim 7, characterized in that, In step 4, the intramodal encoding further achieves sequence-to-space and then-to-sequence through a bidirectional mapping from region to global: firstly, a local state space module performs a local scan of the sequence using a double-offset sliding window strategy, with a sliding window size of [missing information]. The offset step size is The outputs of the non-offset sliding window and the offset sliding window are averaged; then, the sequence features are mapped to the spatial domain by one-dimensional spatial convolution to capture three-dimensional local geometry; finally, the global state space module performs a global scan on the fused sequence to form a bidirectional mapping from sequence to space and back to sequence.

9. The fully sparse cross-modal 3D target detection method based on a hierarchical state-space model according to claim 1, characterized in that, In step 5, the adaptive weighted fusion is performed as follows: for each instance, it is performed by its reliability descriptor. The gating coefficients are calculated using a gating network and Sigmoid activation: in , These are the gated network parameters; then the camera modal features are evaluated using the gate coefficients. With lidar modal characteristics Perform adaptive weighted fusion: in For element-wise product, To integrate enhanced instance features.

10. The fully sparse cross-modal 3D target detection method based on a hierarchical state-space model according to claim 9, characterized in that, In step 5, uncertainty-aware gated fusion is used to concatenate residual features, post-interaction features, and reliability descriptors into a context, and the log-variance is predicted by the variance head. Based on this, the inverse variance weight is calculated. Mixing coefficient with uncertainty perception : Obtain fusion enhancement instance features in The residual features of the filtered instances obtained in step 2 are obtained through skip connections and direct connections without hierarchical interaction. The multi-rotation aggregated features obtained in step 4 through intra-modal encoding, cross-modal token alignment, and semantic-assisted fusion, along with the reliability descriptor of the instance, are... Concatenate as context input; A small target residual enhancement branch is set up for the small target category: instances belonging to the small target category are identified by the small target mask, and the feature increment, orientation residual, scale residual and temperature coefficient are predicted by the small target head. After applying hyperbolic tangent activation to the feature increment with the small target residual scale as the scaling factor, the fused features are enhanced in a residual manner. At the same time, the orientation residual, scale residual and temperature coefficient are output for the detection head to use.

Images