A multi-modal bev three-dimensional target detection method and device

Abstract

The application discloses a kind of multi-modal BEV three-dimensional target detection method and equipment, by extracting the semantic feature of original multi-view image, then view angle embedding is injected into camera feature in the form of channel bias, so that camera feature explicitly carries view angle information, to ensure the effectiveness of camera modal in BEV space participates in fusion, on the other hand, the quality prior composed of distance prior and point density prior is constructed in the fusion stage, and the quality prior, camera BEV feature and lidar BEV feature are input into the gating network, so that the fusion result is closer to the actual observation quality distribution, and in the case of sparse point cloud at a distance, obvious occlusion or image quality degradation, it can still maintain more stable fusion performance, thereby effectively solving the problem of unstable cross-view representation, obvious fusion conflict and insufficient robustness in complex working conditions in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving environment perception and computer vision technology, specifically relating to a multimodal BEV three-dimensional target detection method and device. Background Technology

[0002] 3D object detection is one of the core capabilities of autonomous driving perception systems. It is used to locate and identify vehicles, pedestrians, and other traffic participants in the environment surrounding the vehicle, providing crucial input for subsequent tasks such as object tracking, motion prediction, and path planning. In real-world traffic scenarios, perception systems need to maintain stable performance under complex conditions such as long distances, small targets, dynamic occlusion, nighttime conditions, and rain / fog, which places higher demands on the robustness of detection algorithms.

[0003] Existing autonomous driving platforms typically employ multiple sensors, including cameras and LiDAR. LiDAR point clouds offer high geometric accuracy and scale consistency, facilitating 3D structure reconstruction and localization; however, point clouds tend to be sparser at long distances and are susceptible to occlusion, reflection characteristics, and environmental factors, potentially leading to echo loss or increased noise. Camera images provide dense texture and semantic cues, offering advantages for distant or visually distinctive objects; however, their 3D geometric information is not directly available and is more significantly affected by lighting variations, weather interference, and occlusion, making it difficult to balance accuracy and robustness when relying on a single sensor. Therefore, fusing information from multiple sensors to complement each other's strengths is a crucial direction for improving the 3D detection performance of autonomous driving systems.

[0004] In recent years, Bird's-Eye (BEV) representation has been widely adopted due to its natural alignment with the vehicle coordinate system and its ease of multi-sensor geometric alignment and unified modeling. In a typical scheme, the camera branch elevates perspective image features to 3D space and projects them onto the BEV plane using depth prediction or implicit geometric mapping; the LiDAR branch typically extracts 3D features through voxel encoding and sparse 3D convolution, then compresses them along the height dimension to obtain BEV features. These features are then fused in the BEV space, and the BEV encoder and 3D detector head output the target category and 3D bounding box parameters. The BEV-based fusion framework has significant advantages in computational efficiency and spatial alignment, and is more suitable for engineering deployment.

[0005] Despite significant progress in BEV fusion, existing technologies still have several shortcomings. First, in multi-camera systems, the imaging distribution, visible area, and occlusion patterns differ significantly across different viewpoints. However, many methods employ approximately uniform feature processing and projection strategies for each camera viewpoint, lacking explicit modeling of these viewpoint differences. This easily leads to inconsistencies in cross-viewpoint features, resulting in increased fluctuations in BEV representations across different viewpoints / scenes. Second, multimodal reliability varies significantly with distance, point cloud density, occlusion level, and lighting and weather conditions. For example, point clouds are sparser and geometric constraints are weaker in distant regions, while semantic cues may degrade under low illumination or strong reflective conditions. Existing fusion methods often rely on static weights or employ dynamic fusion approaches with insufficient spatial granularity, making it difficult to adaptively select more reliable modal contributions at the fine-grained BEV mesh level. This leads to cross-modal conflicts, fusion instability, and decreased detection performance under complex conditions. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal BEV 3D target detection method and device to solve the problems of unstable camera BEV representation caused by the lack of explicit modeling of multi-camera viewpoint differences in the prior art, and the difficulty of making spatial fine-grained adaptive adjustments to the fusion strategy with changes in quality such as distance and point cloud sparsity, resulting in cross-modal conflicts and insufficient robustness.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal BEV three-dimensional target detection method includes the following steps: Image features are extracted based on the acquired multi-view camera images; The acquired camera identifiers are mapped into trainable viewpoint embedding vectors and injected into image features in a channel-biased manner. The image features embedded with the injected viewpoint are projected and aggregated into the BEV space to obtain the camera BEV features; The acquired lidar point cloud is voxelized, and the three-dimensional features are extracted and then compressed along the height dimension to obtain the lidar BEV features. The camera BEV features and radar BEV features are channel aligned, concatenated with pre-constructed quality prior information, and then input into a gating network to generate adaptive fusion weights. The camera BEV features and the radar BEV features are dynamically weighted and fused according to the adaptive fusion weights, and the fused BEV features are obtained through channel recalibration. The target detection result is obtained based on the fused BEV features.

[0008] Furthermore, the image features are extracted from multi-view camera images using an image backbone network and a feature pyramid network.

[0009] Furthermore, the obtained camera identifiers are mapped to trainable viewpoint embedding vectors and injected into image features using a channel-biased approach, satisfying the following:

[0010] In the formula, This represents the feature values ​​after incorporating the viewpoint prior. This represents the original FPN feature values. Let b represent the feature of the l-th layer of FPN, b represent the b-th sample, v represent the v-th camera view, and k represent the feature of the l-th layer. One channel, Let (h, w) be the component of the viewpoint embedding vector in the k-th channel, meaning that all spatial positions of the same viewpoint v share the same channel bias, and (h, w) be the spatial coordinates of the feature map.

[0011] Furthermore, the image features embedded with the injected viewpoint are projected and aggregated into the BEV space using depth probability prediction and geometric mapping to obtain camera BEV features.

[0012] Furthermore, after extracting the three-dimensional features through a voxel feature encoder and a sparse three-dimensional convolutional backbone network, the radar BEV features are compressed along the height dimension, including: Within a preset spatial range, the lidar point cloud is discretized according to the voxel size to obtain voxels; Voxel features are obtained by aggregating and encoding the in-point features of non-empty voxels. The voxel features are input into a sparse 3D convolutional network to extract 3D feature volumes, and summation or pooling is performed along the height dimension to obtain the BEV features of the LiDAR.

[0013] Furthermore, the quality prior information includes distance prior and point density prior: The distance prior includes information that reflects the increasing uncertainty of observation of the target or grid position as the distance increases; The point density prior is used to reflect the echo richness and geometric constraint strength of the lidar in the corresponding BEV grid. The point density prior is obtained by counting the number of lidar points falling in the BEV grid, and then normalized after truncation and logarithmic compression. The distance prior and the point density prior are concatenated in the channel dimension to form a mass prior tensor aligned with the BEV grid, which is used as part of the input to the gated network.

[0014] Furthermore, the two BEV features after channel alignment are weighted according to grid-by-grid weights and summed to obtain the fused BEV features. The fusion result satisfies:

[0015] In the formula, The fused BEV feature map For the fusion weight map of camera modes, The camera BEV features are channel-aligned. This is the fusion weight map of radar modes. The radar BEV feature after channel alignment. Let be a four-dimensional real tensor, B represents the number of samples in a batch, C is the number of feature channels, H is the height of the BEV feature map, and W is the width of the BEV feature map.

[0016] Secondly, the present invention provides a multimodal BEV three-dimensional target detection system, comprising an extraction module, an embedding module, a first feature acquisition module, a second feature acquisition module, a weight generation module, a fusion module, and an output module, wherein: Extraction module: Used to extract image features based on the acquired multi-view camera images; Embedding module: used to map the acquired camera identifiers into trainable viewpoint embedding vectors and inject image features in a channel-biased manner; First feature acquisition module: used to project and aggregate the image features after the injected viewpoint is embedded into the BEV space to obtain camera BEV features; The second feature acquisition module is used to perform voxelization on the acquired lidar point cloud, extract three-dimensional features, and then compress them along the height dimension to obtain the lidar BEV features. Weight generation module: used to align the camera BEV features and radar BEV features through channels, concatenate them with pre-constructed quality prior information, and input them into the gating network to generate adaptive fusion weights; Fusion module: used to dynamically weight and fuse the camera BEV features and the radar BEV features according to the adaptive fusion weights, and obtain the fused BEV features through channel recalibration; Output module: used to obtain target detection results based on the fused BEV features.

[0017] Thirdly, a terminal device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0018] Fourthly, a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0019] Compared with the prior art, the present invention has the following beneficial technical effects: This invention provides a multimodal BEV 3D target detection method. By introducing a view prior injection mechanism based on view index into the camera branch, the semantic features of the original multi-view images are first extracted using a shared image backbone network and a multi-scale fusion network. Then, the view embedding is injected into the camera features in the form of channel bias, so that the camera features explicitly carry view information. By enhancing the consistency of different camera view features through view embedding, the stability of the camera BEV representation obtained by subsequent DepthLSSTransform can be improved, making the obtained camera BEV features more consistent under different views and different scenes, thereby ensuring the effectiveness of camera modalities participating in fusion in the BEV space. On the other hand, this invention constructs a quality prior consisting of distance prior and point density prior during the fusion stage, and inputs the quality prior, camera BEV features, and LiDAR BEV features into the gated network GateNet to predict grid-by-grid fusion weights. By introducing the quality prior, the gated network can adaptively adjust the contributions of the two modes at the BEV grid level according to reliability factors such as distance and point cloud sparsity, so that the fusion result is closer to the actual observation quality distribution. It can still maintain more stable fusion performance even in cases of sparse point clouds at long distances, obvious occlusion, or image quality degradation, thereby effectively solving the problems of unstable cross-view representation, obvious fusion conflicts, and insufficient robustness under complex conditions in the existing technology.

[0020] This invention discloses the specific process for obtaining camera BEV features. It involves feature extraction and multi-scale fusion processing of multi-view camera images to obtain camera fusion features; constructing learnable view embeddings for each camera view based on view indexes and injecting these embeddings into the camera fusion features to obtain view enhancement features; constructing a DepthLSS Transform module; predicting the discrete depth distribution of the view enhancement features; and lifting the 2D features along the depth dimension according to the depth distribution to form view frustum features; projecting the view frustum features onto a BEV grid using camera calibration parameters and aggregating them within the same BEV cell to obtain the camera BEV features. It is evident that the camera BEV features obtained by this invention contain both semantic information from multi-view images and spatial alignment information introduced by depth distribution and geometric projection. Furthermore, the view embedding explicitly characterizes view differences, thereby improving the consistency and stability of the camera BEV representation.

[0021] This invention discloses the specific process of gated fusion, which involves constructing camera BEV features and radar BEV features separately, and aligning the two BEV features through channels to ensure semantic consistency. A quality prior is generated on the BEV grid, where the distance prior characterizes the increasing observation uncertainty with increasing distance, and the point density prior is obtained by statistically analyzing the number of points within a BEV cell and then truncated and normalized to improve stability. The camera BEV features, radar BEV features, and quality prior are concatenated and input into a gated network, GateNet, which outputs grid-by-grid two-way modal weights. Based on these weights, the two BEV features are weighted and fused grid-by-grid to obtain the fused BEV features. Further channel recalibration can be performed on the fused BEV features to refine the feature representation. It is evident that this invention, through quality prior-guided gating networks, allocates modal weights in a fine-grained manner in space. This fully leverages the advantages of LiDAR in terms of geometric structure and the advantages of cameras in terms of semantic texture. Furthermore, it adaptively selects more reliable modal contributions in regions with different distances and point cloud densities, thereby more comprehensively representing the complementary relationship between the two modal information in complex scenes and improving the overall accuracy and robustness of 3D target detection. Attached Figure Description

[0022] Figure 1 This is a flowchart of a multimodal BEV three-dimensional target detection method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall network structure; Figure 3 Structural diagrams generated for sparse 3D convolutional backbone networks and BEV features; Figure 4 A structural diagram of the basic coding unit of the Swin-T for the camera branch; Figure 5 A visualization of the results of multi-view camera and radar detection in low-light nighttime scenarios; Figure 6 A visualization of the results of multi-view camera and radar detection in a rainy scene; Figure 7 A visualization of the results of multi-view camera and radar detection in a crossroads scene; Figure 8 A visualization of the results of multi-view camera and radar detection in a dense parking scenario; Figure 9 A visualization of the results of multi-view camera and radar detection on urban main roads; Figure 10 This is a schematic diagram of the electronic device. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0025] It should be noted that the terms "first" and "second" in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] This invention provides a multimodal BEV (Browser-Earned Activated Vehicle) 3D target detection method that integrates viewpoint prior injection and adaptive gating. The method acquires multi-view camera images and LiDAR point cloud data. The LiDAR point cloud is voxelized, and 3D features are extracted using a voxel feature encoder and a sparse 3D convolutional backbone network, followed by height-dimensional compression to obtain LiDAR BEV features. Multi-scale features are extracted from the multi-view camera images using a shared image backbone network and a feature pyramid network. Camera identifiers are mapped to trainable viewpoint embedding vectors, and camera features are injected using a channel-biased approach to introduce discrete viewpoint priors. Subsequently, the multi-view image features are fused based on depth probability prediction and geometric mapping. The camera's BEV features are obtained by projecting and aggregating the features. After aligning the two BEV features by channel, they are concatenated with the quality prior and input into a gating network to predict the softmax gating weights of the two modes at each BEV grid cell to achieve grid-by-grid dynamic weighted fusion. After fusion, the fused BEV representation is refined by a channel recalibration module, and the refined BEV features are input into the BEV encoder and the detection head to output the 3D detection results of the target. Compared with the prior art, this invention can improve cross-view consistency and fusion stability in complex scenes such as sparse point clouds at long distances, occlusion, and changes in illumination, thereby improving the robustness and accuracy of 3D target detection.

[0027] Multi-view camera images can be simultaneously acquired by multiple surround-view cameras, and LiDAR point clouds can be simultaneously scanned by vehicle-mounted LiDAR. In one optional implementation, the multi-view camera includes six different azimuth views. Both the camera and LiDAR can be configured with timestamps and synchronized in time. The acquisition system can simultaneously obtain camera calibration parameters (intrinsic and extrinsic parameters) to establish the geometric correspondence between the image plane and the vehicle coordinate system / world coordinate system, thereby providing a foundation for subsequent cross-modal alignment and BEV spatial mapping. The acquisition steps may also include preprocessing such as resizing and normalizing the camera images, as well as preprocessing such as spatial range cropping and invalid point removal of the point cloud to ensure that different sensor inputs meet the requirements of network computation and fusion alignment.

[0028] The present invention will now be described in further detail with reference to the accompanying drawings: The network structure involved in this invention is as follows: Figure 2 As shown, it includes camera branch, radar branch, fusion branch and detection branch.

[0029] like Figure 1 As shown, a multimodal BEV three-dimensional target detection method includes the following steps: Step 1: Extract image features based on the acquired multi-view camera images; Multi-view camera images can be simultaneously acquired by multiple surround-view cameras. The specific process for extracting image features from multi-view camera images using an image backbone network and a feature pyramid network is as follows: Given a batch of multi-camera images ,in The batch size, The number is 6 (number of camera angles).

[0030] Flatten the image according to the perspective dimension. The images are fed into the same shared SwinTransformer-Tiny (Swin-T) image backbone network, which encodes the images from different viewpoints in a unified manner by sharing parameters, thereby outputting multi-scale backbone features.

[0031] Swin-T models feature relationships within a local window through windowed multi-head self-attention and achieves cross-window information interaction through a shifted window mechanism. Its core self-attention calculation follows a scaled dot product attention form with relative positional bias, as shown in the formula:

[0032] in, These represent query, key, and value, respectively. It is the dimension of the query / key; since the relative position on each axis is in [ M+1; M Within the range of 1], parameterize a small deviation matrix. , The value in is taken from , is a relative position bias matrix, i.e., geometric prior.

[0033] To integrate the aforementioned multi-scale information into a representation more suitable for subsequent depth prediction and BEV projection, the multi-scale features output from the backbone network are input into the Generalized LSSFPN as the camera branch neck, performing top-down pyramid fusion: guided by high-level semantic features, deep features are upsampled to the same resolution as adjacent shallow features, and concatenation and convolution transformations are performed in the channel dimension to obtain fused features with a unified number of channels. The fusion process is shown in the formula:

[0034] in This represents the features of the l-th layer of FPN. Indicates an upsampling operation. Indicates channel splicing. This represents the feature transformation implemented by the convolution module. Indicates that the backbone is in the Lateral characteristics of the layer.

[0035] The structure of the basic coding unit of Swin-T is explained in detail below: A schematic diagram of the basic coding unit of Swin-T used in the camera branch of this invention is shown below. Figure 4As shown, the basic unit consists of two consecutive Transformer Blocks. The left Block uses Window-based Multi-Head Self-Attention (W-MSA), and the right Block uses Shifted Window-based Multi-Head Self-Attention (SW-MSA). Within each Block, a pre-normalized residual structure of "Layer Normalization (LN) → Attention / Feedforward → Residual Addition" is employed: First, the input features pass through LN and enter the attention module. W-MSA performs multi-head self-attention within a fixed-size local window to model the correlation of pixels within the window, thereby extracting local context with low computational overhead. Subsequently, the attention output is added to the original input residual to form intermediate features, which are then fed back through LN into a Multilayer Perceptron (MLP) for channel-dimensional nonlinear mapping and feature enhancement. Finally, the output of the Block is obtained again through residual addition. Unlike W-MSA, SW-MSA periodically shifts the window partition while keeping the window size constant, allowing features in adjacent windows to interact during attention computation. This enables cross-window information fusion and broader context aggregation. By alternately stacking W-MSA and SW-MSA, the effective receptive field can be expanded layer by layer and spatial modeling capabilities can be enhanced while controlling the computational complexity of attention. This provides a more stable and consistent image feature representation for subsequent multi-scale feature fusion and BEV representation construction.

[0036] Step 2: Map the obtained camera identifiers into trainable viewpoint embedding vectors and inject image features in a channel-biased manner. Considering that the camera viewpoint is relatively fixed in real-world deployments, and that the imaging distribution and visible area differ significantly from different viewpoints, this invention introduces a viewpoint prior based on Camera ID.

[0037] For each camera perspective Learn a trainable embedding vector And this is injected as a channel bias into the corresponding viewpoint's FPN features. Let the Generalized LSSFPN's... Layer output is For the first piece in the cut One sample, perspective The features of the injected view prior are:

[0038] In the formula, This represents the feature values ​​after incorporating the viewpoint prior. This represents the original FPN feature values. Let b represent the feature of the l-th layer of FPN, b represent the b-th sample, v represent the v-th camera view, and k represent the feature of the l-th layer. One channel, is the component of the viewpoint embedding vector in the k-th channel, meaning that all spatial positions of the same viewpoint v share the same channel bias, and (h, w) are the spatial coordinates of the feature map, corresponding to the height and width, respectively.

[0039] Step 3: Project and aggregate the image features after the injected viewpoint is embedded into the BEV space to obtain the camera BEV features; The specific process of projecting and aggregating image features into the BEV space using depth probability prediction and geometric mapping to obtain camera BEV features is as follows: The multi-scale image features with pre-injected viewpoint priors are input into the depth prediction module. For each viewpoint feature, the corresponding depth probability distribution is predicted in the pixel dimension. The depth probability distribution can represent the confidence of each pixel at different depth positions within a preset discrete depth range. Based on the camera's intrinsic and extrinsic parameters, as well as the transformation relationship between the vehicle coordinate system and the camera coordinate system, the pixel rays are restored to candidate points in three-dimensional space under different depth assumptions, and these three-dimensional points are mapped to the BEV plane mesh in the bird's-eye view coordinate system. The image features at the pixel are weighted and expanded to different depth positions according to their depth probability (i.e., "lifted" to three-dimensional voxel space), and the weighted features are "scattered" into the corresponding BEV mesh cells according to the geometric mapping relationship, thereby realizing the feature transformation from the image view to the bird's-eye view. Finally, the BEV features from different camera perspectives are aggregated and fused on a unified BEV grid to obtain the camera BEV features of the camera branch.

[0040] Step four: Voxelize the acquired lidar point cloud, extract the three-dimensional features, and compress them along the height dimension to obtain the lidar BEV features: The specific process of performing voxelization on the LiDAR point cloud, extracting 3D features through a voxel feature encoder and a sparse 3D convolutional backbone network, and then compressing them along the height dimension to obtain the LiDAR BEV features is as follows: A. Receive a frame of LiDAR point cloud data within a preset spatial range, perform voxelization processing on the point cloud, and divide and organize the original unstructured point set into a sparse three-dimensional voxel mesh representation according to the preset voxel size; in an optional embodiment, the voxel size can be set to 0.1m×0.1m×0.2m to achieve a balance between resolution and computational overhead. B. For each non-empty voxel obtained after voxelization, a voxel feature encoder (VFE) is used to aggregate and encode point-level attributes within the voxel, compressing information such as local point distribution and reflection intensity into a fixed-dimensional voxel feature vector, thereby converting the variable-length point set representation into a voxel-level representation that can be processed by convolutional networks. C. The voxel features are constructed into sparse voxel feature tensors and input into a multi-layer sparse 3D convolutional backbone network for hierarchical abstraction and stepwise downsampling, so as to extract more discriminative three-dimensional spatial semantic and geometric structural information while maintaining the advantages of sparse computing. D. To achieve alignment and fusion with the camera branch in a unified bird's-eye view space, the three-dimensional feature volume is compressed along the height dimension to generate radar BEV features on a two-dimensional BEV grid. Since this representation transformation only aggregates height information in the feature dimension and does not involve coordinate system transformation or perspective projection, it does not introduce additional geometric distortion or alignment error caused by projection.

[0041] The following provides a detailed explanation of the sparse 3D convolutional backbone network and the BEV feature generation process: The structure diagram of the sparse 3D convolutional backbone network and BEV feature generation is as follows: Figure 3 As shown, the structure, from left to right, includes: a voxelized point cloud feature input module, an input convolutional module (Conv Input), encoder layers, an output convolutional module (Conv Out), a sparse-to-dense conversion module, and a height compression module, ultimately outputting the radar's bird's-eye view features. .

[0042] Figure 3In this context, the voxelized point cloud features are sparse voxel feature representations obtained by voxelizing and encoding the original point cloud; the input convolutional module is used to perform initial feature transformation on the sparse voxel features, which includes a submanifold sparse 3D convolutional layer SubMConv3d, a normalization layer BatchNorm1d, and a nonlinear activation layer ReLU, to complete channel mapping and shallow feature extraction while maintaining the sparse activation positions from spreading; the encoder layer module is used to perform hierarchical abstraction and progressive encoding of the 3D voxel features, including four encoding stages from Stage 1 to Stage 4, where each stage from Stage 1 to Stage 3 is enhanced by several sparse residual basic blocks at the same scale, and at the end of each stage... The first stage uses a sparse 3D convolutional layer, SparseConv3d, to achieve downsampling and receptive field expansion. Stage 4 consists of several SparseBasicBlocks for high-level feature integration without further downsampling. The output convolutional module further integrates the high-level sparse 3D features output by the encoder, and internally includes SubMConv3d, BatchNorm1d, and ReLU to improve the consistency and stability of feature representation. The sparse-to-dense conversion module maps the sparse 3D features to dense feature volumes on a regular voxel grid for subsequent uniform dimensionality transformation. The height compression module compresses the dense 3D features along the height dimension to generate radar BEV features on a 2D BEV plane, thus obtaining... This provides an input basis for subsequent alignment and fusion with camera BEV features in the same bird's-eye view coordinate system.

[0043] Step 5: Align the camera BEV features and radar BEV features by channel, concatenate them with the pre-built quality prior information, and then input them into the gating network to generate adaptive fusion weights. This invention introduces a quality prior dependent on LiDAR. Let the BEV mesh size be... For grid cells Its center coordinates are , To fall into the grid cell LiDAR points; Distance prior is defined as:

[0044] in The maximum radius within the BEV coverage area is used to normalize the distance. .

[0045] The prior definition of point density is:

[0046] in The truncation threshold is used to suppress the impact of extremely high-density regions on training stability.

[0047] Final quality prior It is obtained by splicing two prior maps, distance and density, in the channel dimension.

[0048] The specific process of aligning camera BEV features and radar BEV features by channel, then stitching them with quality priors, and inputting them into the gating network is as follows: Align the camera BEV features obtained in step 3 with the radar BEV features obtained in step 4 to the same BEV grid resolution and coordinate reference. When the number of channels of two features is inconsistent, the channel dimension is mapped by linear transformation or convolutional transformation so that the two features meet the channel alignment requirements required for subsequent fusion. The quality prior map obtained in step S6 is matched one-to-one with the BEV grid in the spatial dimension, and then stitched together with the camera BEV features and radar BEV features that have completed channel alignment in the channel dimension to form a fusion input tensor containing "camera representation, radar representation and reliability hints". Finally, the fused input tensor is input into the gating network.

[0049] Step 6: Dynamically weight and fuse the camera BEV features and radar BEV features according to the adaptive fusion weights, and obtain the fused BEV features through channel recalibration. The specific process of generating adaptive fusion weights at the BEV mesh level to achieve dynamic weighted fusion, and obtaining refined fused BEV features through channel recalibration includes: First, the camera BEV features and radar BEV features are concatenated through channel alignment and quality prior, and then input into the gated network (GateNet). GateNet consists of two 3×3 convolutional layers (with BN and ReLU) and one 1×1 convolutional layer. The forward computation process is as follows:

[0050]

[0051]

[0052] in Represents ReLU; The scoring graph for the two modes is split into modal channels. and To achieve grid-by-grid adaptive fusion, the scores are normalized using softmax in the modal dimension. The specific calculation formula is as follows:

[0053]

[0054] Two-channel BEV features after channel alignment The fusion is achieved by using element-wise multiplication and weighted summation:

[0055] In the formula, The fused BEV feature map For the fusion weight map of camera modes, The camera BEV features are channel-aligned. This is the fusion weight map of radar modes. The radar BEV feature after channel alignment. Let be a four-dimensional real tensor, B represents the number of samples in a batch, C is the number of feature channels, H is the height of the BEV feature map, and W is the width of the BEV feature map.

[0056] Next, the Squeeze-and-Excitation (SE) module is used to recalibrate the channels of the fused BEV features to enhance the focus on key channels.

[0057] The SE module performs full-channel aggregation across the spatial dimension using global average pooling and generates weight coefficients for each channel. It then calculates the global response for each channel. as follows:

[0058] Subsequently, channel weights are generated through a two-layer bottleneck structure. First, channel information is compressed by dimensionality reduction and global dependencies are extracted. Then, nonlinearity is introduced through ReLU. Finally, dimensionality is increased to restore the original number of channels, and the output is constrained to a sigmoid function. , as the gating coefficient for each channel Finally, by scaling the channels, the weights are multiplied back to obtain the refined output features. :

[0059] in It represents Sigmoid. Represents ReLU. This indicates dimensionality reduction. It indicates dimensional ascension.

[0060] Step 7: Obtain the target detection results based on the fused BEV features: The BEV representation output by the fusion module is then input into the BEV encoder to further aggregate the neighborhood context and refine multi-scale semantics on the bird's-eye view plane, making the features more sensitive to the target shape and spatial structure.

[0061] The detection head performs position-by-position prediction on the BEV grid, generates a confidence map of the target center and the corresponding 3D bounding box parameters (including center position, size, height and orientation, etc.), and obtains the final 3D detection result through candidate screening and post-processing.

[0062] To verify the effectiveness of the above detection process, this embodiment conducts experiments on the nuScenes dataset, which contains 1000 sequences, divided into a training set of 700 scenes, a validation set of 150 scenes, and a test set of 150 scenes. Each sequence contains images from six surround-view cameras and one LiDAR scan. Annotations are provided at a frequency of 2Hz and cover 10 categories, totaling approximately 1.4 million annotated 3D bounding boxes. Camera calibration parameters are also provided to establish the correspondence between 3D points and 2D image planes.

[0063] The operating system used in this embodiment is Ubuntu 20.04, the CPU (Central Processing Unit) is a 16-core Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz, the GPU (Graphics Processing Unit) is an RTX 4090 with 24GB of video memory, and the RAM is 120GB.

[0064] All models used in this embodiment are implemented based on PyTorch, with CUDA version 11.3.

[0065] This embodiment uses mAP (mean Average Precision), NDS (nuScenesDetection Score), and AP (Average Precision) for 10 specific detection categories to evaluate the 3D object detection results. The following provides a detailed explanation of mAP, NDS, and the AP for the 10 specific detection categories: mAP is obtained by calculating AP for each class and averaging the APs across all classes. AP is calculated based on the relationship between precision and recall. Precision-recall curves are plotted at different prediction confidence thresholds, and the area under the curve is calculated. A higher mAP value indicates better model performance in object detection.

[0066] NDS is a comprehensive metric proposed by the nuScenes dataset for a more comprehensive evaluation of object detection models in autonomous driving scenarios. This metric considers both the accuracy of detection results and the quality of localization, better reflecting the actual performance of the model in autonomous driving applications. A higher NDS value indicates better model performance in object detection. The formula for calculating NDS is as follows:

[0067] in, These represent translation error, scale error, orientation error, velocity error, and attribute error, respectively. Used to trim each error term to The range is determined, thus converting the error into a scoring item.

[0068] The AP scores for 10 specific detection categories provide a more comprehensive evaluation of 3D object detection performance. These 10 categories include: Car, Truck, Construction Vehicle (CV), Bus, Trailer (TL), Barrier (BR), Motorcycle (MT), Bike, Pedestrian, and Traffic Cone (TC). The differences in AP scores across these 10 categories are primarily due to variations in target size, appearance stability, and perceptual difficulty. Larger targets with relatively fixed shapes and a larger sample size are easier to detect stably; conversely, smaller targets with greater shape variations or fewer samples are more susceptible to distance, occlusion, and noise, leading to increased detection difficulty and relatively lower scores.

[0069] This embodiment applies the proposed method to the nuScenes dataset, combining it with CenterPoint (Center-based 3D Object Detection and Tracking), UVTR (Unifying Voxel-based Representation with Transformer), TransFusion (Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers), FUTR3D (A Unified Sensor Fusion Framework for 3D Detection), BEVFusion-PKU (A Simple and Robust LiDAR-Camera Fusion Framework), and BEVFusion (Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View). The six algorithms (representation, unified BEV representation, multi-task multi-sensor fusion framework) were compared, and the comparison results are shown in Table 1, where "-L" indicates only the radar branch.

[0070] Table 1 Comparative Experimental Results

[0071] The data in Table 1 shows that the method proposed in this invention achieves the highest mAP and NDS scores in both core evaluation metrics, and also the highest scores in most of the 10 specific categories. Therefore, the method proposed in this invention performs better than the six algorithms CenterPoint, UVTR-L, TransFusion-L, FUTR3D, BEVFusion-PKU, and BEVFusion, and is able to perform 3D object detection tasks more effectively.

[0072] To further analyze the robustness of the method under different environmental conditions, this invention constructs scene-based subsets on the nuScenes validation set. Specifically, based on the weather and lighting conditions of the scene, the data is divided into four subsets: Sunny, Rainy, Day, and Night. The mAP and NDS of each subset are then statistically analyzed to characterize the model's performance under image degradation (such as rainy days and low-light conditions at night) and changes in perception difficulty.

[0073] Table 2 Results of subset comparison experiment

[0074] Table 2 summarizes the comparative results on different subsets, showing that the proposed method achieves stable gains on all subsets, with more significant improvements under more challenging conditions such as rain and night. This indicates that the fusion of perspective prior injection and reliability-aware gating can effectively alleviate cross-modal conflicts in complex environments and enhance overall robustness.

[0075] To more intuitively analyze the model's perception performance in complex environments, this invention selects five representative scenarios to present the 3D detection visualization results of multi-view cameras and radar (nighttime low-light conditions, rainy day imaging degradation, intersection interactions, dense parking obstruction, and mixed traffic on urban main roads), such as... Figures 5-9 As shown, the reliability of the proposed method for target localization and detection in different scenarios is demonstrated qualitatively. The visualization results show that, under adverse conditions such as rainy imaging degradation, low illumination, and sparse point clouds at long distances, the method of this invention can effectively reduce missed detections and false detections, and make the spatial position and orientation estimation of the detection box more consistent.

[0076] In another embodiment of the present invention, a multimodal BEV three-dimensional target detection system is provided, comprising an extraction module, an embedding module, a first feature acquisition module, a second feature acquisition module, a weight generation module, a fusion module, and an output module, wherein: Extraction module: Used to extract image features based on the acquired multi-view camera images; Embedding module: used to map the acquired camera identifiers into trainable viewpoint embedding vectors and inject image features in a channel-biased manner; First feature acquisition module: used to project and aggregate the image features after the injected viewpoint is embedded into the BEV space to obtain camera BEV features; The second feature acquisition module is used to perform voxelization on the acquired lidar point cloud, extract three-dimensional features, and then compress them along the height dimension to obtain the lidar BEV features. Weight generation module: used to align the camera BEV features and radar BEV features through channels, concatenate them with pre-constructed quality prior information, and input them into the gating network to generate adaptive fusion weights; Fusion module: used to dynamically weight and fuse the camera BEV features and the radar BEV features according to the adaptive fusion weights, and obtain the fused BEV features through channel recalibration; Output module: used to obtain target detection results based on the fused BEV features.

[0077] Please see Figure 10 As shown, the present invention also provides an electronic device 100 for a multimodal BEV three-dimensional target detection method that combines view prior injection and adaptive gating fusion; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0078] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the multimodal BEV three-dimensional target detection method described in Embodiment 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0079] The at least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor. The processor 102 is the control center of the electronic device 100, connecting various parts of the electronic device 100 via various interfaces and lines.

[0080] The memory 101 in the electronic device 100 stores multiple instructions to implement a multimodal BEV three-dimensional target detection method that combines viewpoint prior injection and adaptive gating fusion. The processor 102 can execute the multiple instructions to achieve the following: Acquire multi-view camera images and LiDAR point cloud data; The point cloud of the lidar is voxelized, and the three-dimensional features are extracted by a voxel feature encoder and a sparse three-dimensional convolutional backbone network and then compressed along the height dimension to obtain the lidar BEV features. Image features are extracted from multi-view camera images using an image backbone network and a feature pyramid network; The camera identifier is mapped to a trainable viewpoint embedding vector, and image features are injected in a channel-biased manner. Image features are projected and aggregated into the BEV space using depth probability prediction and geometric mapping to obtain camera BEV features; Construct quality prior information to characterize modal reliability, including distance prior and point density prior; The camera BEV features and radar BEV features are channel aligned, then stitched together with the quality prior and input into the gating network; Adaptive fusion weights are generated at the BEV mesh level to achieve dynamic weighted fusion, and refined fused BEV features are obtained through channel recalibration. The refined fused BEV features are input into the BEV encoder and the 3D detection head to output the target detection results.

[0081] If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0082] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multimodal BEV three-dimensional target detection method, characterized in that, Includes the following steps: Image features are extracted based on the acquired multi-view camera images; The acquired camera identifiers are mapped into trainable viewpoint embedding vectors and injected into image features in a channel-biased manner. The image features embedded with the injected viewpoint are projected and aggregated into the BEV space to obtain the camera BEV features; The acquired lidar point cloud is voxelized, and the three-dimensional features are extracted and then compressed along the height dimension to obtain the lidar BEV features. The camera BEV features and radar BEV features are channel aligned, concatenated with pre-constructed quality prior information, and then input into a gating network to generate adaptive fusion weights. The camera BEV features and the radar BEV features are dynamically weighted and fused according to the adaptive fusion weights, and the fused BEV features are obtained through channel recalibration. The target detection result is obtained based on the fused BEV features; The obtained camera identifiers are mapped to trainable viewpoint embedding vectors, and image features are injected using a channel-biased approach, satisfying the following: In the formula, This represents the feature values ​​after incorporating the viewpoint prior. This represents the original FPN feature values. Let b represent the feature of the l-th layer of FPN, b represent the b-th sample, v represent the v-th camera view, and k represent the feature of the l-th layer. One channel, The component of the view embedding vector in the k-th channel is (h, w), which means that all spatial positions of the same view v share the same channel bias. (h, w) are the spatial coordinates of the feature map. The quality prior information includes distance prior and point density prior: The distance prior includes information that reflects the increasing uncertainty of observation of the target or grid position as the distance increases; The point density prior is used to reflect the echo richness and geometric constraint strength of the lidar in the corresponding BEV grid. The point density prior is obtained by counting the number of lidar points falling in the BEV grid, and then normalized after truncation and logarithmic compression. The distance prior and the point density prior are concatenated in the channel dimension to form a mass prior tensor aligned with the BEV grid, which is used as part of the input to the gated network.

2. The multimodal BEV three-dimensional target detection method according to claim 1, characterized in that, The image features are extracted from multi-view camera images using an image backbone network and a feature pyramid network.

3. The multimodal BEV three-dimensional target detection method according to claim 1, characterized in that, The image features embedded with the injected viewpoint are projected and aggregated into the BEV space using depth probability prediction and geometric mapping to obtain camera BEV features.

4. The multimodal BEV three-dimensional target detection method according to claim 1, characterized in that, The radar BEV features are obtained by extracting the three-dimensional features through a voxel feature encoder and a sparse three-dimensional convolutional backbone network, followed by compression along the height dimension, including: Within a preset spatial range, the lidar point cloud is discretized according to the voxel size to obtain voxels; Voxel features are obtained by aggregating and encoding the in-point features of non-empty voxels. The voxel features are input into a sparse 3D convolutional network to extract 3D feature volumes, and summation or pooling is performed along the height dimension to obtain the BEV features of the LiDAR.

5. The multimodal BEV three-dimensional target detection method according to claim 1, characterized in that, The two BEV features after channel alignment are weighted according to grid-by-grid weights and summed to obtain the fused BEV features. The fusion result satisfies: In the formula, The fused BEV feature map For the fusion weight map of camera modes, The camera BEV features are channel-aligned. This is the fusion weight map of radar modes. The radar BEV feature after channel alignment. Let be a four-dimensional real tensor, B represents the number of samples in a batch, C is the number of feature channels, H is the height of the BEV feature map, and W is the width of the BEV feature map.

6. A multimodal BEV three-dimensional target detection system, characterized in that, It includes an extraction module, an embedding module, a first feature acquisition module, a second feature acquisition module, a weight generation module, a fusion module, and an output module, wherein: Extraction module: Used to extract image features based on the acquired multi-view camera images; Embedding module: used to map the acquired camera identifiers into trainable viewpoint embedding vectors and inject image features in a channel-biased manner; First feature acquisition module: used to project and aggregate the image features after the injected viewpoint is embedded into the BEV space to obtain camera BEV features; The second feature acquisition module is used to perform voxelization on the acquired lidar point cloud, extract three-dimensional features, and then compress them along the height dimension to obtain the lidar BEV features. Weight generation module: used to align the camera BEV features and radar BEV features through channels, concatenate them with pre-constructed quality prior information, and input them into the gating network to generate adaptive fusion weights; Fusion module: used to dynamically weight and fuse the camera BEV features and the radar BEV features according to the adaptive fusion weights, and obtain the fused BEV features through channel recalibration; Output module: used to obtain target detection results based on the fused BEV features; The obtained camera identifiers are mapped to trainable viewpoint embedding vectors, and image features are injected using a channel-biased approach, satisfying the following: In the formula, This represents the feature values ​​after incorporating the viewpoint prior. This represents the original FPN feature values. Let b represent the feature of the l-th layer of FPN, b represent the b-th sample, v represent the v-th camera view, and k represent the feature of the l-th layer. One channel, The component of the view embedding vector in the k-th channel is (h, w), which means that all spatial positions of the same view v share the same channel bias. (h, w) are the spatial coordinates of the feature map. The quality prior information includes distance prior and point density prior: The distance prior includes information that reflects the increasing uncertainty of observation of the target or grid position as the distance increases; The point density prior is used to reflect the echo richness and geometric constraint strength of the lidar in the corresponding BEV grid. The point density prior is obtained by counting the number of lidar points falling in the BEV grid, and then normalized after truncation and logarithmic compression. The distance prior and the point density prior are concatenated in the channel dimension to form a mass prior tensor aligned with the BEV grid, which is used as part of the input to the gated network.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements as claimed in claim 1.

5. The steps of any one of the methods described.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements as described in claim 1.

5. The steps of any one of the methods described.

Images