Cross-modal fusion three-dimensional target detection method and system and storage medium

Abstract

The invention discloses a cross-modal fusion three-dimensional target detection method and system based on a Mama model, and a storage medium, and the method comprises the steps: firstly providing a shallow BEV state space fusion method, constructing a hidden state space through a two-dimensional selective scanning mechanism of the Mama model, enhancing the shallow features through feature channel exchange operation and a visual state space block, and carrying out the detection of the hidden state space; and cross-modal feature association and interaction are realized. Secondly, forming a deep BEV state space fusion method by using a cross attention mechanism and a visual state space block, calculating difference features of different modes by using the attention mechanism, and respectively feeding the difference features and the enhanced shallow multi-mode features into the visual state space block for deep fusion; and finally, by designing a channel adaptive module, the original BEV feature representation is enhanced, and the limitation of the Mama model in the aspect of modeling the relationship between the channels is made up. According to the method, the problem of BEV feature information loss is effectively solved, the advantages of two modes can be effectively combined in a BEV space, and the detection performance is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a cross-modal fusion three-dimensional target detection method, system and storage medium based on the Mamba model. Background Art

[0002] In recent years, object detection technology has developed rapidly, with detection capabilities improving in both static and dynamic environments. Simultaneously, two-dimensional detection technology has gradually evolved into three-dimensional detection, a shift that has been widely applied in fields such as mobile robotics and autonomous driving. LiDAR (light detection and ranging) and cameras are the two fundamental sensors used in current three-dimensional object detection methods for autonomous driving, each with its own unique characteristics. Cameras capture RGB images, providing rich texture and color information, but they are sensitive to illumination changes and lack sufficient depth information, resulting in limitations in three-dimensional reconstruction and depth estimation. In contrast, LiDAR can accurately measure the distance between objects and the environment, providing high-precision three-dimensional point cloud data with depth and spatial resolution typically higher than cameras. However, LiDAR equipment is expensive, and the generated point cloud data is often sparse and lacks texture detail. Furthermore, LiDAR has limitations when detecting occluded or distant objects, increasing the complexity of data processing.

[0003] Early research focused primarily on exploring a single modality. Drawing on successful approaches in 2D detection, most image-based methods first detect 2D candidate objects and then predict 3D bounding boxes using neural networks or geometric constraint models. However, monocular cameras lack direct depth information. In contrast, LiDAR-based detection methods rely on high-resolution point cloud data and generally outperform monocular camera-based methods in depth perception. However, the point cloud data generated by LiDAR is relatively sparse, making processing sparse data a significant challenge. To address this issue, voxel-based networks divide the point cloud into small voxels and extract features from each voxel, improving the ability to handle sparse data and reducing the computational burden. Researchers have further improved the efficiency of processing sparse point clouds by projecting the point cloud into a columnar structure, thereby enhancing detection performance. With the continuous advancement of LiDAR technology and the reduction in its cost, its application in object detection has gradually increased.

[0004] As can be seen from the above analysis, each single modality has its advantages and disadvantages. To overcome the limitations of single sensors, multimodal fusion methods that integrate the advantages of multiple modalities have gradually emerged. These methods fuse data from LiDAR and cameras, leveraging the rich texture information provided by images and the precise depth information provided by LiDAR to achieve more accurate 3D object detection. However, due to the inherent characteristics of LiDAR and cameras, effectively integrating the representations of these two modalities remains a research challenge. Existing multimodal fusion data representations can be divided into two categories: unified representation and raw representation. Raw representation aims to preserve as much available information as possible without using redundant encoding methods, but this approach requires significant computation and inference time, making it unsuitable for real-time detection. In contrast, unified representation methods aim to transform heterogeneous data into a homogeneous format. Methods based on the Bird's Eye View (BEV) representation have demonstrated high interpretability and scalability for multimodal data. By unifying multimodal features into a shared BEV representation space, these methods preserve geometric structure and semantic density while naturally supporting most 3D perception tasks. Two BEVFusion frameworks proposed by researchers are representative works of this approach. These two frameworks fuse camera images and LiDAR point cloud data into a unified BEV representation space, achieving more accurate and robust object detection. The former primarily optimizes the camera's BEV pooling method, thereby improving the efficiency of feature conversion and fusion. The latter uses an improved Transformer architecture (, introducing an attention mechanism and spatially modulated attention to address sensor calibration errors and image quality differences).

[0005] However, existing multimodal fusion 3D object detection methods still face some challenges when fusing from a BEV (bird's-eye view) perspective. The previously proposed BEVFusion framework directly inputs the extracted BEV features into the subsequent fusion process without additional processing. After undergoing multi-layer convolution operations such as the backbone network and view conversion, these heterogeneous BEV features may experience local information loss in the shallow layers. This leads to insufficient fusion of deep features due to the lack of relevance and complementarity of shallow cross-modal features, and ultimately insufficient expressiveness of the fused features, thus affecting detection performance. Summary of the Invention

[0006] In response to the above problems, the present invention provides a cross-modal fusion three-dimensional target detection method, system and storage medium based on the Mamba model, aiming to solve the problem that when the existing cross-modal fusion three-dimensional target detection method is fused in a bird's-eye view, the local information of shallow features is lost after the heterogeneous features undergo multi-layer convolution processing, resulting in insufficient fusion of deep features due to the lack of correlation and complementarity of shallow cross-modal features, and the final fused features cannot be accurately expressed.

[0007] According to a first aspect of an embodiment of the present disclosure, a cross-modal fusion three-dimensional object detection method based on a Mamba model is provided, the method comprising the following steps:

[0008] For multi-view image and LiDAR point cloud input, the visual backbone network and point cloud backbone network are used to generate the first camera BEV feature and the first point cloud BEV feature respectively;

[0009] Use the feature channel exchange module to perform channel segmentation and splicing on the first camera BEV feature and the first point cloud BEV feature to obtain the second camera BEV feature and the second point cloud BEV feature;

[0010] The second camera BEV feature and the second point cloud BEV feature are processed respectively by using the two-dimensional selective scanning mechanism of the visual state space block to obtain the third camera BEV feature and the third point cloud BEV feature;

[0011] Use the attention mechanism to calculate the difference features between the third camera BEV features and the third point cloud BEV features;

[0012] The third camera BEV feature, the third point cloud BEV feature, and the difference feature are projected into the hidden state space using the gating mechanism of the visual state space block to obtain the first hidden camera BEV feature, the first hidden point cloud BEV feature, the first hidden difference feature, and the corresponding gating parameters; the first hidden camera BEV feature and the first hidden point cloud BEV feature are deeply fused using the first hidden difference feature and the corresponding gating parameters to obtain the first hidden state feature after the interaction of the camera feature and the difference feature, and the second hidden state feature after the interaction of the LiDAR feature and the difference feature;

[0013] Project the first hidden state feature and the second hidden state feature back to the original feature space, and obtain the first deep fusion feature and the second deep fusion feature through residual connection respectively;

[0014] After processing the first camera BEV feature and the first point cloud BEV feature respectively using the channel adaptation module, a camera BEV enhancement mode and a point cloud BEV enhancement mode are obtained;

[0015] The first deep fusion feature is combined with the camera BEV enhancement mode, and the second deep fusion feature is combined with the point cloud BEV enhancement mode to obtain the final camera BEV feature and point cloud BEV feature and send them to the detection head to complete target detection.

[0016] In some embodiments, the visual state space block constructs the hidden state space using the two-dimensional selective scanning mechanism of the Mamba model, which divides the image into multiple blocks and generates independent sequences in four directions, thereby capturing information from different directions and constructing a global receptive field.

[0017] In some embodiments, the channel adaptation module uses adaptive average pooling and maximum pooling to extract global features, and weights the importance of each channel through the feature channel attention mechanism. The obtained attention weights are applied to the first camera BEV features and the first point cloud BEV features, and the reduced dimensionality features are restored to the original number of channels through a convolution to obtain the camera BEV enhanced mode and the point cloud BEV enhanced mode.

[0018] In some embodiments, the convolution is a 1x1 convolution.

[0019] In some embodiments, the feature channel exchange module is used for channel segmentation and splicing, specifically comprising the following steps:

[0020] The first camera BEV feature and the first point cloud BEV features Divide into 4 equal parts along the channel dimension;

[0021] from Select the first and third parts from Select the second and fourth parts and stitch them into the second camera BEV feature in sequence

[0022] from Select the second and fourth parts from Select the first and third parts and stitch them together in sequence to form the second point cloud BEV feature

[0023] In some embodiments, the visual state space block includes a two-dimensional selective scanning mechanism module and an MLP module,

[0024] The two-dimensional selective scanning mechanism module processes the input second camera BEV features and the second point cloud BEV features to obtain the selective attention weight W SS2D And applied to the input features;

[0025] The output features of the two-dimensional selective scanning mechanism module and the MLP module are combined through residual connection, and regularization technology is used to combine them with the input feature map to obtain the output features after combining SS2D and MLP.

[0026] According to a second aspect of an embodiment of the present disclosure, a cross-modal fusion three-dimensional object detection system based on a Mamba model is provided, the system comprising:

[0027] An initial BEV feature acquisition module is used to generate the first camera BEV feature and the first point cloud BEV feature using the visual backbone network and the point cloud backbone network for multi-view image and LiDAR point cloud input respectively;

[0028] A feature channel exchange module is used to perform channel segmentation and splicing on the first camera BEV feature and the first point cloud BEV feature to obtain the second camera BEV feature and the second point cloud BEV feature;

[0029] A visual state space block is used to process the second camera BEV feature and the second point cloud BEV feature respectively by using a two-dimensional selective scanning mechanism to obtain a third camera BEV feature and a third point cloud BEV feature;

[0030] An attention mechanism module, used to calculate the difference features between the third camera BEV features and the third point cloud BEV features using the attention mechanism;

[0031] A visual state space block with a gating mechanism is used to project the third camera BEV feature, the third point cloud BEV feature, and the difference feature into the hidden state space using the gating mechanism to obtain the first hidden camera BEV feature, the first hidden point cloud BEV feature, the first hidden difference feature, and the corresponding gating parameters; the first hidden camera BEV feature and the first hidden point cloud BEV feature are fused in a deep hidden state using the first hidden difference feature and the corresponding gating parameters to obtain the first hidden state feature after the camera feature and the difference feature interact, and the second hidden state feature after the LiDAR feature and the difference feature interact; the first hidden state feature and the second hidden state feature are projected back to the original feature space, and the first deep fusion feature and the second deep fusion feature are respectively obtained through residual connection;

[0032] A channel adaptation module is used to process the first camera BEV feature and the first point cloud BEV feature respectively to obtain a camera BEV enhancement mode and a point cloud BEV enhancement mode;

[0033] The final BEV feature acquisition module is used to combine the first deep fusion feature with the camera BEV enhancement mode, and the second deep fusion feature with the point cloud BEV enhancement mode to obtain the final camera BEV feature and point cloud BEV feature and send them to the detection head to complete target detection.

[0034] In some embodiments, the visual state space block uses the two-dimensional selective scanning mechanism of the Mamba model to construct a hidden state space. The two-dimensional selective scanning mechanism divides the image into multiple blocks and generates independent sequences in four directions, thereby capturing information from different directions and constructing a global receptive field.

[0035] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned cross-modal fusion three-dimensional target detection method based on the Mamba model are implemented.

[0036] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the above-mentioned cross-modal fusion three-dimensional target detection method based on the Mamba model are implemented.

[0037] The embodiments of the present disclosure provide a cross-modal fusion three-dimensional target detection method, system, electronic device and storage medium based on the Mamba model. First, to address the problem of shallow feature information loss, a shallow BEV state space fusion method is proposed. This method utilizes the two-dimensional selective scanning mechanism of the Mamba model to construct a hidden state space, enhances shallow features through feature channel exchange operations and visual state space blocks, and realizes cross-modal feature association and interaction. Secondly, a deep BEV state space fusion method is formed using a cross-attention mechanism and a visual sequence space block. The attention mechanism is used to calculate the differential features of different modalities, and the enhanced shallow multimodal features are respectively sent to the visual sequence space block for deep fusion. Finally, by designing a channel adaptive module, the original BEV feature representation is enhanced to make up for the limitations of the Mamba model in modeling the relationship between channels. The beneficial effects of the present invention are as follows:

[0038] 1) A shallow BEV state space fusion method is proposed. The two-dimensional selective scanning mechanism of the Mamba model is used to construct the hidden state space. Through feature channel exchange and Visual State Space Block (VSSblock) operations, it not only enhances shallow features but also effectively supplements lost information, thereby promoting the association and interaction of cross-modal features.

[0039] 2) A deep BEV state space fusion method is proposed. It uses an attention mechanism to align and compare features from different modalities, adaptively capturing their interrelationships and feature differences. It dynamically measures the feature importance of each position and calculates differential features. These features are fed into the VSS block along with the enhanced shallow multimodal features. Deep fusion is then performed in the previously constructed hidden state space to further enhance cross-modal features.

[0040] 3) Design a channel adaptation module. This module uses adaptive average pooling and max pooling to extract global features and capture the overall distribution of features. Through a feature channel attention mechanism, the importance of each channel is weighted, enhancing the feature representation of key channels. This improves the representation of single-modal features and compensates for the limitations of the Mamba model in modeling inter-channel relationships.

[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0043] Figure 1 This is a flow chart of a cross-modal fusion three-dimensional target detection method based on the Mamba model in an embodiment of the present invention;

[0044] Figure 2 This is an overall block diagram of a cross-modal fusion three-dimensional object detection method based on the Mamba model in an embodiment of the present invention;

[0045] Figure 3 2. It is a schematic diagram of the principle of the two-dimensional selective scanning mechanism in an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of the principle of a characteristic channel exchange module in an embodiment of the present invention;

[0047] Figure 5 2 is a schematic diagram of the visual state space block principle in an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of the principle of deep feature fusion in an embodiment of the present invention;

[0049] Figure 7 Schematic diagram of the structure of a cross-modal fusion three-dimensional target detection system based on the Mamba model in an embodiment of the present invention;

[0050] Figure 8 is a schematic diagram of an electronic device according to an embodiment of the present invention;

[0051] Figure 9 is a visualization result diagram on the nuScenes dataset in an embodiment of the present invention;

[0052] Figure 10 1 is a comparison chart of the precision-recall (PR) curves of the automobile category in an embodiment of the present invention;

[0053] Figure 11 is a graph of the classification and regression loss results of the model in an embodiment of the present invention;

[0054] Figure 12 It is a vehicle platform used for real-world testing in the embodiments of the present invention;

[0055] Figure 13 This is a diagram showing the actual average detection accuracy of different modules on the main detection targets in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0057] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0058] The present invention has the following embodiments:

[0059] A cross-modal fusion 3D target detection method based on the Mamba model, such as Figure 1 As shown, the method includes the following steps:

[0060] S1. For multi-view image and LiDAR point cloud input, generate the first camera BEV feature and the first point cloud BEV feature using the visual backbone network and the point cloud backbone network respectively;

[0061] S2. Use the feature channel exchange module to perform channel segmentation and splicing on the first camera BEV feature and the first point cloud BEV feature to obtain the second camera BEV feature and the second point cloud BEV feature;

[0062] S3, using the two-dimensional selective scanning mechanism of the visual state space block to process the second camera BEV feature and the second point cloud BEV feature respectively to obtain the third camera BEV feature and the third point cloud BEV feature;

[0063] S4, using the attention mechanism to calculate the difference features between the third camera BEV features and the third point cloud BEV features;

[0064] S5. Project the third camera BEV feature, the third point cloud BEV feature, and the difference feature into the hidden state space using a visual state space block with a gating mechanism to obtain the first hidden camera BEV feature, the first hidden point cloud BEV feature, the first hidden difference feature, and corresponding gating parameters;

[0065] S6. Using the first hidden difference feature and the corresponding gating parameter, the first hidden camera BEV feature and the first hidden point cloud BEV feature are deeply hidden in the state to obtain a first hidden state feature after the camera feature and the difference feature interact, and a second hidden state feature after the LiDAR feature and the difference feature interact;

[0066] S7, projecting the first hidden state feature and the second hidden state feature back to the original feature space, and obtaining the first deep fusion feature and the second deep fusion feature respectively through residual connection;

[0067] S8. Using a channel adaptive module to process the first camera BEV feature and the first point cloud BEV feature respectively, to obtain a camera BEV enhancement mode and a point cloud BEV enhancement mode;

[0068] S9. Combine the first deep fusion feature with the camera BEV enhancement mode, and combine the second deep fusion feature with the point cloud BEV enhancement mode to obtain the final camera BEV feature and point cloud BEV feature and send them to the detection head to complete target detection.

[0069] Specifically, multimodal 3D object detection fusion methods based on the complementarity between modalities have gradually become a research hotspot, especially for LiDAR and cameras. Current fusion methods are divided into two categories: learning-independent fusion and learning-based fusion. Learning-independent fusion: This method mainly relies on arithmetic operations on features (such as addition or averaging) or simple splicing. Although this method is computationally efficient, its scalability and robustness when dealing with complex scenes are relatively weak. MV3D (multi-view 3D networks) captures spatial information by averaging features from three different viewpoints, simplifying the fusion process. ContFuse (deep continuous fusion) aligns features from different modalities using sensor coordinates and fuses features of the same dimension using element-by-element addition. PointPainting splices the pixel segmentation scores from the semantic segmentation network with the point cloud, preserving the geometric information of the point cloud while enhancing the semantic information.

[0070] Learning-based fusion: These methods primarily use attention mechanisms to fuse features, focusing on high-weight, important information. Despite their higher computational complexity, they offer improved scalability and accuracy when processing multimodal features. Essentially, these methods are based on the attention mechanism, which is closely related to the Transformer architecture. DETR (Detection Transformer) is a landmark algorithm that applies the attention mechanism and Transformer architecture to object detection. It globally models input image features using a self-attention mechanism and predicts the object's bounding box and category from a set of learnable queries, eliminating the need for anchor points and non-maximum suppression, thus achieving end-to-end object detection. TransFusion extracts LiDAR BEV features and image features using a standard backbone network and projects the image features into the BEV space using a cross-attention mechanism for modal fusion. 3D-CVF (3D-cross view fusion) proposes an adaptive gated fusion network that significantly simplifies convolutional layer computation through attention mapping and enhances point cloud features by supplementing image features, thereby improving fusion effectiveness. While these methods have effectively improved object detection performance, challenges remain in cross-modal discrepancy fusion and computational efficiency.

[0071] The Mamba model, originally designed for natural language processing (NLP), gradually evolved and expanded into computer vision, becoming a popular research method with widespread application in various vision tasks. VMamba (visual state space model) introduces a two-dimensional selective scanning mechanism to overcome the limitations of traditional one-dimensional sequential scanning when processing two-dimensional images, enabling the extension of the Mamba model to vision tasks. Building on the VMamba model, the disparity-guided multispectral Mamba (DMM) leverages the Mamba model's efficient modeling capabilities to construct a disparity-guided multimodal fusion module that adaptively fuses RGB and infrared image features based on modal disparity. Furthermore, it employs multi-scale target-aware attention to enhance the feature representation of target regions in each modality. The Mamba model has demonstrated significant potential in the field of vision.

[0072] In a specific embodiment, Figure 2 As shown, the implementation process of the embodiment mainly relies on four main components: a multi-view image and LiDAR point cloud input module, a shallow BEV state space fusion module, a deep BEV state space fusion module, and a channel adaptation module.

[0073] State Space Models (SSMs)

[0074] The state space model originated from the Kalman filter and is widely used in various fields. Recently, some scholars have proposed the structured state space model (Structured State Spaces for Sequence, S4), and conducted a series of studies and extensions on the state space model. The state space model represents the internal state of the system through a set of linear differential equations. Due to its ability to effectively capture and represent long-range dependencies in time series data, these models are widely used to model long-term dynamic systems. Specifically, SSMs can be expressed as a linear time-invariant system that passes a one-dimensional input sequence x(t)∈R to an intermediate hidden state h(t)∈R N Processing is performed to produce an output y(t)∈R, where N is the dimension of the hidden layer. Mathematically, the system is often represented by the following linear ordinary differential equation:

[0075] h′(t)=Ah(t)+Bx(t),

[0076] y(t)=Ch(t)+Dx(t), (1)

[0077] Among them, the state transfer matrix A∈R N×N , projection parameter B∈R N×1 , C∈R 1×N And the skip connection D∈R (usually D=0).

[0078] Discretization of SSMs: Due to the continuous time characteristics of SSMs, direct application of them in the field of deep learning faces significant challenges. In the context of computer vision research, discretization becomes a necessary means to better adapt to subsequent tasks such as target detection, image classification and recognition. Generally speaking, in deep learning methods, the ordinary differential equation of formula (1) is discretized and converted into discrete functions to promote efficient computational operations. First, the time scale parameter Δ is introduced, and then the continuous parameters A and B are converted into discrete parameters through the commonly used zero-order hold rule and first-order Taylor series. and

[0079]

[0080] After discretization, formula (1) can be rewritten as follows:

[0081]

[0082] y(t)=Ch(t) (3)

[0083] The visual state space block constructs the hidden state space using the two-dimensional selective scanning mechanism of the Mamba model, which divides the image into multiple blocks and generates independent sequences in four directions, thereby capturing information from different directions and building a global receptive field.

[0084] 2D-Selective-ScanMechanism (SS2D

[0085] SS2D is an innovative mechanism designed to address the receptive field limitation problem encountered when applying one-dimensional sequence modeling methods (such as S6 (Structured StateSpace for Sequences Version 6), an extension of the S4 model) to visual tasks. Although S6 performs well in processing natural language sequences, its direct application to visual data has limitations due to the nonlinearity and spatial complexity of visual information itself. To solve this problem, Figure 3 As shown in Figure 2, SS2D divides the image into multiple blocks and generates independent sequences in four directions, thereby capturing information from different directions and building a global receptive field. This four-directional scanning mechanism not only overcomes the limitations of convolution operations but also combines the sequence processing advantages of S6, effectively enhancing cross-modal feature fusion without significantly increasing computational complexity.

[0086] The feature channel exchange module is used for channel segmentation and splicing, and specifically includes the following steps:

[0087] The first camera BEV feature and the first point cloud BEV features Divide into 4 equal parts along the channel dimension;

[0088] from Select the first and third parts from Select the second and fourth parts and stitch them into the second camera BEV feature in sequence

[0089] from Select the second and fourth parts from Select the first and third parts and stitch them together in sequence to form the second point cloud BEV feature

[0090] Specifically, the shallow BEV State-Space Fusion Module (SBSF) uses feature channel swapping to generate new multi-view image BEV features. and LiDAR point cloud BEV features

[0091] Among them, FCE(·,·) represents the feature channel exchange operation, which is implemented by channel segmentation and splicing. Figure 4 As shown, first and Divide into 4 equal parts along the channel dimension, and then Select the first and third parts from Select the second and fourth parts and stitch them together in sequence to form a new image BEV feature Similarly, generate new LiDAR point cloud BEV features Through this operation, the information construction of different channels of cross-modal features can be simply and effectively integrated, enriching the diversity of feature channels and thus improving the subsequent fusion performance. and As input to the VSS block.

[0092] The VSS block is the visual processing core of the Mamba model, designed for feature learning. VMamba replaces the original S6 block in the Mamba model with a VSS block, reducing branch complexity and making the newly formed VSS block more suitable for visual tasks. The embodiment is based on a new design of VMamba and modifies the feature processing mechanism to adapt it to processing BEV features of cameras and LiDAR. Figure 5 As shown in Figure 3, the redesigned VSS block retains the advantages of VMamba, enabling the model to effectively capture the global receptive field, dynamic weights, and linear complexity, while also being applicable to the cross-modal relationship and interaction between camera and LiDAR data.

[0093] After processing by the VSS block, the cross-modal interaction of shallow camera and LiDAR features can be enhanced:

[0094]

[0095] Among them, VSS(·,·) is the processing of VSS block, and It is the output feature of the shallow state space fusion module of camera and LiDAR.

[0096] The visual state space block includes a two-dimensional selective scanning mechanism module and an MLP module.

[0097] The two-dimensional selective scanning mechanism module processes the input second camera BEV features and the second point cloud BEV features to obtain the selective attention weight W SS2D And applied to the input features;

[0098] The output features of the two-dimensional selective scanning mechanism module and the MLP module are combined through residual connection, and regularization technology is used to combine them with the input feature map to obtain the output features after combining SS2D and MLP.

[0099] Specifically, in the VSS block, the SS2D module selectively focuses on the important areas in the input feature map and suppresses irrelevant noise information. As shown in formula (6), specifically, the input feature F is processed using the SS2D module, and the obtained selective attention W is SS2D The weights are applied to the feature maps. The VSS block also effectively combines the output features of the SS2D module and the MLP module through residual connections, and uses the regularization technique DropPath to improve computational efficiency and ensure model performance and generalization ability. The output feature F after combining SS2D and MLP is output As shown in formula (7):

[0100] F SS2D =W SS2D F, (6)

[0101] F output =F+DropPath(MLP(Norm2(F SS2D ))) (7)

[0102] Deep BEV State-Space Fusion Module (DBSF)

[0103] To further reduce cross-modal feature discrepancies and enhance BEV feature fusion, an attention mechanism is employed to calculate differential features between the camera and LiDAR. These differential features are projected into the previously constructed hidden state space, and a gating mechanism is used to construct hidden state transitions. These features are then deeply fused with the shallowly fused output features of the two modalities to form the deep BEV state space fusion module.

[0104] By comparing camera and LiDAR features, focusing on the complementary information between the two, capturing the long-range dependency between camera and LiDAR features, and calculating the similarity between LiDAR and camera features, the embodiment can adaptively select the most important modal information and enhance the cross-modal fusion effect. The downsampled features are converted into a form suitable for attention mechanism input ([B, L, C]), so that the camera features can be used as queries (Q), and the LiDAR features as keys (K) and values ​​(V). The multi-head attention mechanism can be expressed as:

[0105]

[0106] Among them, dk is the scaling factor, and the difference feature obtained is F diff express.

[0107] Then, by stacking multiple processing layers in the VSS block, the two modal features processed by the shallow feature fusion module and the previously calculated difference features are projected into the hidden state space, and the deep convolution and gating mechanism are used to promote the deep fusion of cross-modal BEV features. Figure 6 shown.

[0108] First, after obtaining shallow fusion features and And the difference characteristics Finally, a VSS block with a gating mechanism is used to project them into the hidden state space and obtain the corresponding gating parameters:

[0109]

[0110] Among them, P hide (·) represents the operation of projecting features into the hidden state space, g θ (·), g ω (·) denotes a gating operation with parameters θ, ω. Is the gating parameter used to adjust the subsequent hidden state features and Simultaneously achieve deep hidden state fusion:

[0111]

[0112] in, Represents the hidden state features after the interaction of camera features and difference features, Represents the hidden state features after the interaction of LiDAR features and difference features. · Represents element-wise multiplication.

[0113] Afterwards, and Project back to the original feature space and obtain deep fusion features through residual connection for the final feature fusion:

[0114]

[0115] Among them, P back (·) is a projection operation with a linear transformation, which ensures that the features do not change their original dimensions when projected back to the original space.

[0116] The channel adaptation module uses adaptive average pooling and maximum pooling to extract global features. It then uses a feature channel attention mechanism to weight the importance of each channel. The resulting attention weights are applied to the first camera BEV feature and the first point cloud BEV feature. A convolution is then performed to restore the reduced-dimensional features to their original number of channels, yielding the camera BEV enhancement modality and the point cloud BEV enhancement modality. Preferably, the convolution is a 1x1 convolution.

[0117] Channel Adaptation Model (CAM)

[0118] While the Mamba model excels at handling long-range dependencies, particularly capturing long-range features in time series data, it still has limitations in modeling inter-channel relationships in 3D object detection, particularly when processing BEV features. This is because the Mamba model was originally designed for one-dimensional sequential data and, therefore, cannot fully model the dependencies and interactions between these channels when capturing cross-channel information.

[0119] For the original input BEV feature map F i , first apply the SiLU activation function to enhance its nonlinear representation ability. Then, use a 1x1 convolution to reduce the dimension and reduce the number of channels to obtain F reduced Then, calculate the global average pooling to get the feature F avg And the global maximum pooling gets the feature F max , to capture global information across channels:

[0120] F avg =GlobalAvgPool(SiLU(F reduced )),

[0121] F max =GlobalMaxPool(SiLU(F reduced )) (13)

[0122] The pooling result F avg and F max Processed by RELU nonlinear function, and generate channel weight α:

[0123] a=σ(Conv1x1(F avg )+Conv1x1(F max )) (14)

[0124] Among them, σ(·) is the Sigmoid activation function, which ensures that the generated weight α is between [0, 1] and is used to dynamically adjust the weight of each channel.

[0125] The calculated attention weight α will be applied to the input feature F i And restore the reduced dimension features to the original number of channels C through a 1x1 convolution:

[0126] F CA =Conv1x1(a·F reduced ) (15)

[0127] Finally, the two enhanced modes F CA-camera and F CA-lidar Combined with the previous deep fusion features and The final fusion BEV feature is combined and sent to the subsequent detection head for testing. The complete process is shown in the algorithm in Table 1 below:

[0128]

[0129]

[0130] Another embodiment is used to illustrate a cross-modal fusion 3D target detection system based on the Mamba model, such as Figure 7 As shown, the system 700 includes:

[0131] An initial BEV feature acquisition module 710 is configured to generate a first camera BEV feature and a first point cloud BEV feature using a visual backbone network and a point cloud backbone network, respectively, for a multi-view image and a LiDAR point cloud input;

[0132] A feature channel exchange module 720 is configured to perform channel segmentation and splicing on the first camera BEV feature and the first point cloud BEV feature to obtain a second camera BEV feature and a second point cloud BEV feature;

[0133] A visual state space block 730 is configured to process the second camera BEV feature and the second point cloud BEV feature respectively by using a two-dimensional selective scanning mechanism to obtain a third camera BEV feature and a third point cloud BEV feature;

[0134] an attention mechanism module 740 , configured to calculate a difference feature between the third camera BEV feature and the third point cloud BEV feature using an attention mechanism;

[0135] A visual state space block 750 with a gating mechanism is used to project the third camera BEV feature, the third point cloud BEV feature, and the difference feature into the hidden state space using the gating mechanism to obtain a first hidden camera BEV feature, a first hidden point cloud BEV feature, a first hidden difference feature, and corresponding gating parameters; perform deep hidden state fusion on the first hidden camera BEV feature and the first hidden point cloud BEV feature using the first hidden difference feature and the corresponding gating parameters to obtain a first hidden state feature after the camera feature and the difference feature interact, and a second hidden state feature after the LiDAR feature and the difference feature interact; project the first hidden state feature and the second hidden state feature back to the original feature space, and obtain a first deep fusion feature and a second deep fusion feature respectively through residual connections;

[0136] a channel adaptation module 760 for processing the first camera BEV feature and the first point cloud BEV feature respectively to obtain a camera BEV enhancement mode and a point cloud BEV enhancement mode;

[0137] The final BEV feature acquisition module 770 is used to combine the first deep fusion feature with the camera BEV enhancement mode, and the second deep fusion feature with the point cloud BEV enhancement mode to obtain the final camera BEV feature and point cloud BEV feature and send them to the detection head to complete target detection.

[0138] Preferably, the visual state space block adopts the two-dimensional selective scanning mechanism of the Mamba model to construct the hidden state space, which divides the image into multiple blocks and generates independent sequences in four directions, so as to capture information from different directions and construct a global receptive field.

[0139] In addition to the above modules, the system 700 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present disclosure, their illustration and description are omitted here.

[0140] For other specific working processes of the cross-modal fusion three-dimensional target detection system 700 based on the Mamba model, refer to the description of the embodiment of the cross-modal fusion three-dimensional target detection method based on the Mamba model above, and will not be repeated here.

[0141] Another embodiment is used to illustrate that the system of the present invention can also be used with the help of Figure 8 The architecture of the computing device shown is implemented. Figure 8 The architecture of the computing device is shown in FIG. Figure 8As shown, a computer system 810, a system bus 830, one or more CPUs 840, an input / output 820, a memory 850, etc. The memory 850 can store various data or files used for computer processing and / or communication, as well as program instructions executed by the CPU, including the embodiment of the cross-modal fusion three-dimensional object detection method based on the Mamba model. Figure 8 The architecture shown is only exemplary and may be adjusted based on actual needs when implementing different devices. Figure 8 One or more components in. The memory 850, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the cross-modal fusion 3D target detection method based on the Mamba model in the embodiment of the present invention (for example, the initial BEV feature acquisition module 710, the feature channel exchange module 720, the visual state space block 730, the attention mechanism module 740, the visual state space block with the gating mechanism 750, the channel adaptation module 760 and the final BEV feature acquisition module 770 in the cross-modal fusion 3D target detection system 700 based on the Mamba model). One or more CPUs 840 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions and modules stored in the memory 850, that is, to implement the above-mentioned cross-modal fusion 3D target detection method based on the Mamba model, which includes:

[0142] For multi-view image and LiDAR point cloud input, the visual backbone network and point cloud backbone network are used to generate the first camera BEV feature and the first point cloud BEV feature respectively;

[0143] Use the feature channel exchange module to perform channel segmentation and splicing on the first camera BEV feature and the first point cloud BEV feature to obtain the second camera BEV feature and the second point cloud BEV feature;

[0144] The second camera BEV feature and the second point cloud BEV feature are processed respectively by using the two-dimensional selective scanning mechanism of the visual state space block to obtain the third camera BEV feature and the third point cloud BEV feature;

[0145] Use the attention mechanism to calculate the difference features between the third camera BEV features and the third point cloud BEV features;

[0146] The third camera BEV feature, the third point cloud BEV feature, and the difference feature are projected into the hidden state space using the gating mechanism of the visual state space block to obtain the first hidden camera BEV feature, the first hidden point cloud BEV feature, the first hidden difference feature, and the corresponding gating parameters; the first hidden camera BEV feature and the first hidden point cloud BEV feature are deeply fused using the first hidden difference feature and the corresponding gating parameters to obtain the first hidden state feature after the interaction of the camera feature and the difference feature, and the second hidden state feature after the interaction of the LiDAR feature and the difference feature;

[0147] Project the first hidden state feature and the second hidden state feature back to the original feature space, and obtain the first deep fusion feature and the second deep fusion feature through residual connection respectively;

[0148] After processing the first camera BEV feature and the first point cloud BEV feature respectively using the channel adaptation module, a camera BEV enhancement mode and a point cloud BEV enhancement mode are obtained;

[0149] The first deep fusion feature is combined with the camera BEV enhancement mode, and the second deep fusion feature is combined with the point cloud BEV enhancement mode to obtain the final camera BEV feature and point cloud BEV feature and send them to the detection head to complete target detection.

[0150] Of course, the processor of the server provided in the embodiment of the present invention is not limited to executing the method operations described above, but can also execute relevant operations in the cross-modal fusion three-dimensional target detection method based on the Mamba model provided in any embodiment of the present invention.

[0151] The memory 850 may primarily include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal, etc. Furthermore, the memory 850 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some instances, the memory 850 may further include memory remotely located relative to one or more CPUs 840, and these remote memories may be connected to the system via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0152] The input / output 820 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the system. The input / output 820 may also include a display device such as a display screen.

[0153] An embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the cross-modal fusion three-dimensional target detection method based on the Mamba model described in the above embodiment is implemented. The computer-readable storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system or device.

[0154] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, system, or device.

[0155] The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0156] In addition, other specific working processes of a non-temporary computer-readable storage medium refer to the description of the embodiment of the cross-modal fusion three-dimensional target detection method based on the Mamba model above, and are not repeated here.

[0157] In order to verify the effectiveness of the method, system and storage medium of the present invention, a verification example was carried out. The data set used the open data set nuScenes and Waymo to evaluate the performance of the proposed method. nuScenes collected 1,000 complete scenes, divided into 700 training sets, 150 validation sets and 150 test sets. Each scene lasts about 20 seconds and contains various complex scenes, such as sunny days, rainy days and dark conditions. nuScenes uses six cameras and a 32-line LiDAR device. These multi-angle cameras and LiDAR allow nuScenes to obtain views from different angles, thereby obtaining a 360-degree panoramic view. The verification example mainly reports results based on two indicators: mean average precision (mAP) and nuScenes detection score (NDS). Among them, NDS is a comprehensive indicator that combines attribute factors such as the predicted object position, size, direction and speed. At the same time, results of 10 categories are also reported to facilitate detailed comparison with other methods. The Waymo dataset contains 1,150 scenes, of which approximately 800 are used for training, 200 for validation, and 150 for testing, with each scene lasting 20 seconds. Waymo data is collected by five LiDAR sensors and five high-resolution pinhole cameras. The evaluation uses the mean average precision (mAP) and the mean average precision including heading angle (mAPH). The targets are divided into two levels (LEVEL 1 and LEVEL 2). The example focuses on the comparison of three types of targets in LEVEL 2: vehicles, pedestrians, and cyclists.

[0158] Experimental details: Based on the open source frameworks BEVFusion and MMDetection3D. For camera input, SwinTransformer is used as the backbone network, and FPN is used to fuse multi-scale image features to generate a feature map of 1 / 8 input size. The resolution of the input image is adjusted to 256×704 to meet the network structure and computational requirements. For LiDAR input, the SECOND backbone network is used to extract voxel features with a voxel size of [0.075m, 0.075m, 0.2m]. The range of the point cloud in the X, Y, and Z dimensions is set to [-54.0m, -54.0m, -5.0m, 54.0m, 54.0m, 3.0m]. A series of data augmentations are applied to the LiDAR and camera inputs to improve generalization capabilities. These enhancements include random flipping ([-5.4°, 5.4°]), scaling ([-0.38, 0.55]), rotation The optimization was performed using AdamW, with a maximum learning rate of 0.001 and a weight decay of 0.01. To mitigate overfitting, 3D objects and point clouds were randomly flipped. The examples do not use test-time augmentation (TTA) or multi-model ensembles during inference.

[0159] Experimental performance

[0160] On the nuScenes dataset, the embodiments are compared with 14 current advanced methods, including single LiDAR methods: Pointpillars, CenterPoint, TransFusion-L; camera and LiDAR fusion methods: MVP, GraphAlign, PointAugmenting, UVTR, BEVFusion, BEVFusion, CMT, MSMDFusion, SparseFusion, 3D-CVF, FUTR3D, ObjectFusion (.

[0161] On the Waymo dataset, the embodiment is compared with six advanced methods, including Pointpillars, PVRCNN, CenterPoint, TransFusion, PointAugmenting, and LoGoNet.

[0162] Tables 1, 2, and 3 show that (the bold font in each table indicates the optimal value in each column) the method of the embodiment of the present invention achieves advanced performance, reaching 72.4% mAP and 73.9% NDS on the nuScenes test set. These two indicators are 2.2% and 1% higher than the baseline BEVFusion in this paper, respectively. In terms of the individual detection results of 10 categories, it achieves a leading position in 7 categories. The visualization effect is shown in Figure 2. Figure 9 shown.

[0163] Table 1 3D object detection performance on the nuScenes test set

[0164]

[0165] Table 2 3D object detection performance on the nuScenes validation set

[0166]

[0167] Table 3 Performance comparison on the Waymo validation set (category 2 / mAPH)

[0168]

[0169] Loss function evaluation: The embodiment model is trained using RTX 4090 GPU and performance is tested. The precision-recall (PR) curve is used to evaluate the model regression performance. Taking the car category as an example, it is compared with other methods on the nuScenes dataset, such as Figure 10 As shown, the curves indicate that the proposed MA-CMF3D exhibits good convergence. Figure 11For the classification and regression losses of the model, the gradient loss decreases rapidly in the initial training stage and then gradually stabilizes without significant changes throughout the training process.

[0170] Actual vehicle test

[0171] Simulation experiments on datasets alone cannot demonstrate the performance of the proposed method in real scenarios. Therefore, the embodiment was tested on a real vehicle platform. Figure 12 As shown in Figure 1, the experimental platform includes an 80-line lidar, two GNSS antennas, a binocular camera, a micro-electromechanical system inertial measurement unit, and a GNSS / INS navigation system. The experimental platform was tested in Xiangcheng District, Suzhou.

[0172] In order to intuitively demonstrate the effectiveness of each module of the algorithm proposed in this paper, the average accuracy of each module between the baseline model and the proposed algorithm is compared for the main objects of target detection in actual road conditions (cars, pedestrians, trucks, buses). Figure 13 Table 4 (Note: Baseline is BEVFusion) shows the test results for the main detection targets on the experimental platform. As shown in Table 4, in complex real-world environments, the proposed method can still effectively improve the performance of the baseline model by adding different modules (where S represents SBSF and D represents DBSF).

[0173] Table 4 Performance of some main detection targets on the real vehicle platform

[0174]

[0175] Ablation test

[0176] To investigate the contribution of each component module in the model, the baseline model follows the official BEVFsuion implementation. Ablation studies were conducted using the nuScenes dataset to verify the effectiveness of the SBSF, DBSF, and CAM modules. Furthermore, the impact of the number of VSS block layers on model performance was tested. The performance improvements of each component are shown in Table 5. A two-layer VSS was used for testing. Using CAM alone does not significantly improve the model; it primarily serves to address the limitations of the Mamba model in modeling channel relationships. Its performance is maximized when combined with the other two modules.

[0177] Table 5 Ablation study of each module in the model on the nuScenes validation set

[0178]

[0179] Table 6 Impact of the number of VSS block layers in SBSF and BDSF on model performance

[0180]

[0181] The Mamba model-based cross-modal fusion 3D object detection method, system, electronic device, and storage medium provided in the above-mentioned embodiments effectively enhance the correlation and complementarity of cross-modal features in the BEV space by designing a shallow BEV state space fusion module, a deep BEV state space fusion module, and a channel adaptation module. This method is another attempt to use the Mamba model to fuse cameras and LiDAR for cross-modal 3D object detection. Leveraging the hidden state space constructed by the Mamba model, the added innovative module design excels in improving shallow feature interaction and enhancing deep feature fusion, providing a new approach and paradigm for multimodal 3D object detection.

[0182] In this document, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that comprises a series of elements includes not only those elements, but also includes other elements not expressly listed, or also includes elements inherent to such step or method.

[0183] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A cross-modal fusion 3D target detection method based on the Mamba model, characterized in that: The method comprises the following steps: For multi-view image and LiDAR point cloud input, the visual backbone network and point cloud backbone network are used to generate the first camera BEV feature and the first point cloud BEV feature respectively; Use the feature channel exchange module to perform channel segmentation and splicing on the first camera BEV feature and the first point cloud BEV feature to obtain the second camera BEV feature and the second point cloud BEV feature; The second camera BEV feature and the second point cloud BEV feature are processed respectively by using the two-dimensional selective scanning mechanism of the visual state space block to obtain the third camera BEV feature and the third point cloud BEV feature; Use the attention mechanism to calculate the difference features between the third camera BEV features and the third point cloud BEV features; Projecting the third camera BEV features, the third point cloud BEV features, and the difference features into the hidden state space using the gating mechanism of the visual state space block to obtain the first hidden camera BEV features, the first hidden point cloud BEV features, the first hidden difference features, and corresponding gating parameters; Performing deep hidden state fusion on the first hidden camera BEV feature and the first hidden point cloud BEV feature using the first hidden difference feature and the corresponding gating parameters to obtain a first hidden state feature after the camera feature and the difference feature interact, and a second hidden state feature after the LiDAR feature and the difference feature interact; Project the first hidden state feature and the second hidden state feature back to the original feature space, and obtain the first deep fusion feature and the second deep fusion feature through residual connection respectively; The channel adaptation module is used to process the first camera BEV feature and the first point cloud BEV feature respectively to obtain a camera BEV enhancement mode and a point cloud BEV enhancement mode; The first deep fusion feature is combined with the camera BEV enhancement mode, and the second deep fusion feature is combined with the point cloud BEV enhancement mode to obtain the final camera BEV feature and point cloud BEV feature and send them to the detection head to complete target detection.

2. The cross-modal fusion three-dimensional target detection method based on the Mamba model according to claim 1 is characterized in that: The visual state space block constructs the hidden state space using the two-dimensional selective scanning mechanism of the Mamba model, which divides the image into multiple blocks and generates independent sequences in four directions, thereby capturing information from different directions and building a global receptive field.

3. The cross-modal fusion three-dimensional target detection method based on the Mamba model according to claim 1 is characterized in that: The channel adaptation module uses adaptive average pooling and maximum pooling to extract global features, and weights the importance of each channel through the feature channel attention mechanism. The obtained attention weights are applied to the first camera BEV feature and the first point cloud BEV feature, and the reduced dimensionality features are restored to the original number of channels through a convolution to obtain the camera BEV enhancement mode and the point cloud BEV enhancement mode.

4. The cross-modal fusion three-dimensional target detection method based on the Mamba model according to claim 3 is characterized in that: The convolution is a 1x1 convolution.

5. The cross-modal fusion three-dimensional target detection method based on the Mamba model according to claim 1 is characterized in that: The feature channel exchange module is used for channel segmentation and splicing, and specifically includes the following steps: The first camera BEV feature and the first point cloud BEV features Divide into 4 equal parts along the channel dimension; from Select the first and third parts from Select the second and fourth parts and stitch them into the second camera BEV feature in sequence from Select the second and fourth parts from Select the first and third parts and stitch them together in sequence to form the second point cloud BEV feature 6. The cross-modal fusion three-dimensional target detection method based on the Mamba model according to claim 2 is characterized in that: The visual state space block includes a two-dimensional selective scanning mechanism module and an MLP module, The two-dimensional selective scanning mechanism module processes the input second camera BEV features and the second point cloud BEV features to obtain the selective attention weight W SS2D And applied to the input features; The output features of the two-dimensional selective scanning mechanism module and the MLP module are combined through residual connection, and regularization technology is used to combine them with the input feature map to obtain the output features after combining the two-dimensional selective scanning mechanism module and MLP.

7. A cross-modal fusion 3D target detection system based on the Mamba model, characterized by: The system comprises: An initial BEV feature acquisition module is used to generate the first camera BEV feature and the first point cloud BEV feature using the visual backbone network and the point cloud backbone network for multi-view image and LiDAR point cloud input respectively; A feature channel exchange module is used to perform channel segmentation and splicing on the first camera BEV feature and the first point cloud BEV feature to obtain the second camera BEV feature and the second point cloud BEV feature; A visual state space block is used to process the second camera BEV feature and the second point cloud BEV feature respectively by using a two-dimensional selective scanning mechanism to obtain a third camera BEV feature and a third point cloud BEV feature; An attention mechanism module, used to calculate the difference features between the third camera BEV features and the third point cloud BEV features using the attention mechanism; A visual state space block with a gating mechanism is used to project the third camera BEV feature, the third point cloud BEV feature, and the difference feature into the hidden state space using the gating mechanism to obtain the first hidden camera BEV feature, the first hidden point cloud BEV feature, the first hidden difference feature, and the corresponding gating parameters; the first hidden camera BEV feature and the first hidden point cloud BEV feature are fused in a deep hidden state using the first hidden difference feature and the corresponding gating parameters to obtain the first hidden state feature after the camera feature and the difference feature interact, and the second hidden state feature after the LiDAR feature and the difference feature interact; the first hidden state feature and the second hidden state feature are projected back to the original feature space, and the first deep fusion feature and the second deep fusion feature are respectively obtained through residual connection; A channel adaptation module is used to process the first camera BEV feature and the first point cloud BEV feature respectively to obtain a camera BEV enhancement mode and a point cloud BEV enhancement mode; The final BEV feature acquisition module is used to combine the first deep fusion feature with the camera BEV enhancement mode, and the second deep fusion feature with the point cloud BEV enhancement mode to obtain the final camera BEV feature and point cloud BEV feature and send them to the detection head to complete target detection.

8. The cross-modal fusion three-dimensional object detection system based on the Mamba model according to claim 7 is characterized in that: The visual state space block constructs the hidden state space using the two-dimensional selective scanning mechanism of the Mamba model, which divides the image into multiple blocks and generates independent sequences in four directions, thereby capturing information from different directions and constructing a global receptive field.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the cross-modal fusion three-dimensional target detection method based on the Mamba model are implemented as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the cross-modal fusion three-dimensional target detection method based on the Mamba model are implemented as described in any one of claims 1 to 6.

Images