A pixel-level confidence-gated dynamic fusion method for multimodal BEV perception features

Abstract

This invention relates to the field of autonomous driving environmental perception technology and provides a pixel-level confidence-gated multimodal BEV perception feature dynamic fusion method. The method first acquires and processes multi-view camera images and LiDAR point clouds, outputting two types of 2D feature maps. After obtaining the dual-modal BEV feature map through BEV spatial mapping, a pixel-level confidence map is calculated. Based on the confidence differences, a dynamic fusion weight map is generated through a gating function to achieve pixel-level weighted fusion. Finally, based on the fused feature map, the target center heatmap, 3D bounding box parameters, and detection confidence are predicted, outputting the 3D target detection result. This invention overcomes the shortcomings of existing methods, such as rigid static weights and coarse global weights, dynamically adapts to environmental changes, accurately captures spatial heterogeneity, and improves the accuracy and robustness of autonomous driving environmental perception.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving environment perception technology, and in particular relates to a pixel-level confidence-gated method for dynamic fusion of multimodal BEV perception features. Background Technology

[0002] Currently, autonomous driving technology has entered the L2+ level popularization stage and is moving towards L3 / L4 levels. The accuracy of environmental perception is the core bottleneck that determines the safety level of autonomous driving. Only by accurately identifying the type and spatial location of surrounding targets can reliable data be provided for subsequent decision-making and control, avoiding safety accidents such as collisions. However, single sensors have unavoidable defects due to limitations in physical principles and technical characteristics. BEV space, on the other hand, can achieve "penetrating" detection of occluded targets by combining LiDAR depth information with camera texture, while unifying the coordinate system and laying the foundation for efficient fusion of multimodal data.

[0003] Multimodal fusion in BEV (Battery Electric Vehicle) space has become a core research direction in the field of autonomous driving perception. Its technical architecture has undergone iterations and has fully shifted from the early "post-fusion" mode to the "pre-fusion" mode. The technical logic of "post-fusion" is to first complete feature extraction and target detection independently through sensors such as cameras and LiDAR, and then merge the detection results of multiple sensors in a rule-based manner. Although this mode has the characteristics of strong module independence and low debugging difficulty, some key information is inevitably lost in the feature extraction stage of a single sensor. Moreover, if a sensor has a detection deviation (such as the camera misjudging the target category due to lighting), the post-fusion mechanism is difficult to effectively correct the deviation, ultimately resulting in limited perception accuracy.

[0004] Existing methods mostly use fixed fusion weights, which cannot dynamically adapt to environmental conditions such as nighttime, rainy days, and foggy days. When environmental changes cause fluctuations in the performance of a certain type of sensor, fixed weights cannot match real-time requirements, resulting in a decline in the quality of fused features.

[0005] Even when some methods introduce dynamic weights, they often assign a single global weight to the entire BEV feature map. However, in real-world scenarios, the sensor reliability varies significantly across different spatial regions (near and far, illuminated and unilluminated areas) of the same frame image. Global weights struggle to accurately capture this spatial heterogeneity, resulting in a mismatch between weight allocation and the actual needs of the region. Summary of the Invention

[0006] The purpose of this invention is to provide a pixel-level confidence-gated multimodal BEV perception feature dynamic fusion method, which aims to solve the technical problems existing in the prior art as identified in the background art.

[0007] This invention is implemented as follows: a pixel-level confidence-gated multimodal BEV perception feature dynamic fusion method, the method comprising the following steps:

[0008] Simultaneously, multi-view camera images and LiDAR point cloud data are acquired, and these two different types of data are processed to obtain multi-view camera 2D feature maps and LiDAR 2D feature maps, respectively.

[0009] The multi-view camera 2D feature maps and lidar 2D feature maps obtained above are mapped to a unified bird's-eye view (BEV) space to obtain camera BEV feature maps and lidar BEV feature maps.

[0010] For the camera BEV feature map and the lidar BEV feature map obtained above, pixel-level confidence estimation is performed respectively. A confidence value reflecting the reliability of each feature at the spatial location of BEV is generated, thereby forming the camera pixel-level confidence map and the lidar pixel-level confidence map.

[0011] Based on the generated camera pixel-level confidence map and LiDAR pixel-level confidence map, the confidence difference between the two modes at the same BEV pixel position is calculated. Then, through a learnable gating function, this difference is dynamically converted into a fusion weight between 0 and 1, generating a dynamic fusion weight map covering the entire BEV space.

[0012] Using the dynamically fused weight map generated above, pixel-level weighted fusion is performed on the camera BEV feature map and the LiDAR BEV feature map. Specifically, for each pixel location in the BEV space, the fusion weight at that location is used to weight and sum the feature vectors of the two modes, thereby obtaining the fused BEV feature map.

[0013] Based on the fused BEV feature map, the core task of autonomous driving environmental perception, namely 3D object detection, is performed. This step predicts the target center heatmap to locate the target, predicts 3D bounding box parameters to describe the target's geometric properties, and predicts the detection confidence to assess reliability. Finally, the filtered and correlated 3D object detection results are output.

[0014] The beneficial effects of this invention are:

[0015] This invention achieves weight adaptation through a real-time dynamic adjustment mechanism. The confidence estimation network updates the confidence map in real time with the sensor frame rate. The gating function dynamically calibrates the fusion weight of each BEV pixel based on the latest confidence value. When the performance of a sensor degrades due to environmental factors, its confidence is automatically lowered, and the gating function synchronously reduces the fusion weight of that sensor feature, ensuring that the more reliable mode always dominates the fusion in extreme scenarios, thus solving the adaptation defects of fixed weights when the environment changes.

[0016] Fine-grained weight allocation is achieved by generating pixel-level confidence maps. The confidence level is calculated separately for each BEV pixel. The differences in sensor reliability in different spatial regions (near and far, lit and shadowed areas) within the same frame are directly reflected in the numerical differences of pixel-level confidence levels. The gated fusion module assigns exclusive fusion weights to each pixel based on these differences, accurately capturing the heterogeneity of sensor performance in different spatial regions within the same frame and solving the problem of coarseness in global weights.

[0017] By employing a lightweight confidence estimation network and a gating function, the computational cost is transformed into linear growth, reducing computational complexity. At the same time, the confidence graph has a clear physical meaning, and the weights are directly driven by the confidence level. The reasons for the allocation can be traced through the heatmap, which improves the interpretability of the method.

[0018] Spatial alignment of multimodal features is achieved based on BEV spatial mapping, avoiding the errors introduced by coordinate system transformation in traditional multi-sensor fusion. At the same time, pixel-level weighted fusion maximizes the preservation of complementary information of each modality, improving the accuracy and robustness of 3D target detection. Attached Figure Description

[0019] Figure 1 This is an overall architecture diagram of a pixel-level confidence-gated multimodal BEV perception feature dynamic fusion method provided in an embodiment of the present invention;

[0020] Figure 2 The confidence scores and fusion weights of different modalities in a nighttime scene are shown.

[0021] Figure 3 This is a confidence score and fusion weight graph for different modalities in a daytime scene. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0023] like Figure 1 As shown, a pixel-level confidence-gated multimodal BEV perception feature dynamic fusion method is presented.

[0024] (I) Multimodal data acquisition and feature extraction

[0025] Autonomous vehicles are typically equipped with surround multi-view cameras and LiDAR. At any given moment, the system simultaneously acquires images from the multi-view cameras (6 views, 1280×720 resolution) and a frame of LiDAR point cloud (containing 100,000-200,000 points per second). For the multi-view camera images, feature extraction is required to obtain 2D feature representations with rich semantic information.

[0026] Specifically, multi-view camera images are input into a lightweight, high-precision backbone network (a CNN-based network). This network achieves hierarchical extraction of multi-scale visual information through a cascaded operation of convolution-batch normalization-activation-downsampling. Shallow layers capture low-order features such as edges and textures, while deep layers aggregate high-order semantic features such as target contours and category associations, optimizing computational efficiency while maintaining accuracy. Let the input image tensor be... ,in The number of RGB channels is 3, where 3 represents the batch size. , Let represent the height and width of the single-view image, respectively. The feature extraction process of the l-th layer can be formalized as follows:

[0027] ;

[0028] in , The parameters of the l-th convolutional kernel are... This is a convolution operation (used to capture local spatial feature associations). This is a batch normalization operation (to accelerate network training convergence and suppress feature distribution shift). For activation functions (such as the Swish function), nonlinear transformations are introduced to fit the feature distribution of complex scenes. The downsampling factor for the l-th layer (adaptively set according to feature scale requirements to achieve dynamic adjustment of image resolution and the number of feature channels). For downsampling operations (such as max pooling, stride convolution, to reduce the spatial dimension of the feature map to reduce the amount of subsequent computation);

[0029] After processing through layer L, the output is a 2D feature map of the multi-view camera. ,in This represents the number of feature channels (reflecting the semantic richness of the features). , This represents the spatial dimension of the feature map. This feature map preserves both the texture and category information of the target, while optimizing computational efficiency through downsampling, providing high-quality visual feature input for subsequent BEV projection.

[0030] The core purpose of lidar feature extraction is to transform disordered 3D point clouds into structured 2D feature maps, preserving the 3D spatial location and reflectivity information of the point clouds while reducing data dimensionality to improve computational efficiency.

[0031] A pillar-based point cloud feature extraction framework (such as PointPillars) is adopted. Through a technical process of "point cloud pillarization - pillar feature statistics - feature dimensionality enhancement," it achieves efficient conversion of 3D point clouds to 2D features while maximizing the preservation of local spatial distribution information of the point cloud. Let the set of input LiDAR 3D point clouds be represented as:

[0032] ;

[0033] in This represents the total number of points in a single frame of the point cloud. The first 3D spatial coordinates of a point This represents the reflectivity at that point (reflecting information about the target surface material). First, for... The BEV space is divided into several three-dimensional cylinders according to a preset cylinder size (the projection of the cylinder onto the BEV plane is square, and its height covers the effective detection range of the LiDAR), and each point in the point cloud is assigned to a corresponding cylinder. Let the side length of the cylinder on the BEV plane be... The effective detection range of a highly covered lidar is... Columns The included point set is:

[0034] ;

[0035] in For the first The coordinates of the top-left corner of each cylinder in the BEV plane. Cylinderization transforms the disordered point cloud into an ordered set of cylinders. Simultaneously, by limiting the cylinder size, it ensures that each cylinder contains a relatively uniform number of points, avoiding feature bias caused by differences in point cloud density during subsequent feature calculations. Secondly, for each cylinder... Statistical characteristics are calculated to quantify the spatial distribution and reflectance properties of point clouds within a cylinder. Let the cylinder... Include If there are 1 point, then its statistical characteristic vector is... The calculation process is as follows:

[0036] ;

[0037] in, Points inside the cylinder Coordinate mean (reflecting the overall position and average reflectivity of the column). , , , These are the maximum values ​​for the corresponding coordinates. , , , These are the minimum values ​​of the corresponding coordinates (the maximum and minimum values ​​together reflect the spatial range of the point cloud within the cylinder). , , , These represent the standard deviations of the corresponding coordinates (reflecting the dispersion of the point cloud distribution within the cylinder). Statistical features allow for the reduction of data dimensionality while fully preserving the key spatial and material information of the point cloud within the cylinder, providing a foundation for subsequent feature extraction. Finally, fully connected layers and convolutional layers are used to enhance the dimensionality and structure the cylinder features, generating a 2D feature map for the LiDAR. First, the statistical feature vector of each cylinder is... The formula for mapping to a higher-dimensional feature space via fully connected layers is:

[0038] ;

[0039] in This is the weight matrix of the fully connected layer. This is the bias vector. Subsequently, the high-dimensional features of all cylinders are... Arrange the features according to their positions on the BEV plane to form an initial 2D feature map; then, a convolutional layer is used to downsample and fuse the initial feature map, finally outputting the LiDAR 2D feature map.

[0040] ;

[0041] in This represents the number of feature channels of the lidar. This represents the spatial dimension of the feature map. This feature map transforms the 3D information of the LiDAR point cloud into a structured 2D representation, providing efficient feature input for subsequent BEV projection.

[0042] (II) BEV Space Mapping and Feature Alignment

[0043] Based on the Transformer spatial cross-attention mechanism, the feature transformation from 2D to 3D BEV space is completed through the technical process of "BEV query grid generation - multi-view feature association learning - feature aggregation", and the segmentation depth information is supplemented to improve the spatial positioning accuracy of BEV features.

[0044] First, a BEV query grid is generated to define the structure and extent of the BEV space. Let the length of the BEV space correspond to the actual physical space be... Width is The grid resolution is (That is, each BEV grid corresponds to the actual space) (Region), then the spatial dimension of the BEV query grid is , The tensor representation of the BEV query grid is as follows: ,in To determine the number of channels in the query grid (consistent with the number of channels in subsequent fused features to ensure dimensionality compatibility), this grid is used to aggregate visual information at corresponding locations in multi-view images. Secondly, spatial cross-attention is used to learn the association between the BEV query grid and multi-view 2D features, supplementing the segmentation depth information. Let the nth channel in the multi-view 2D features be... The feature map of each perspective is The calculation process of spatial cross attention can be expressed as:

[0045]

[0046] in, For the first Transpose of the view feature map. For attention dimension (by This number, along with the number of attention heads, is used to avoid gradient vanishing due to excessively large attention weights. This is a normalization operation. Through this operation, the BEV query grid can adaptively learn the image region features corresponding to itself from each viewpoint, and then supplement the segmentation depth information through the complementarity of multi-view features—for example, a BEV grid may be occluded in a frontal view but visible in a side view; the attention mechanism can effectively aggregate the features of this grid by increasing the weight of the side view features. Finally, the attention-weighted features of all views are aggregated, and channel adjustment is used to ensure compatibility with subsequent steps. The feature aggregation process is represented as follows:

[0047] ;

[0048] ;

[0049] in, Number of camera angles This is a temporary BEV feature map after multi-view feature aggregation. This is a 1×1 convolution operation (only adjusting the number of feature channels, without changing the spatial dimension). for Convolution kernel parameters. Final output: Camera BEV feature map. This feature map contains both the camera's visual semantic information and the global perspective and precise spatial positioning capability of the BEV space, laying the spatial and dimensional foundation for subsequent fusion with LiDAR BEV features.

[0050] (iii) Pixel-level confidence estimation

[0051] A lightweight convolutional network with the same structure but different parameters is used to process the BEV features of the two modalities separately, in order to adapt to the different feature distribution characteristics of different modalities and ensure the accuracy of confidence estimation. Let the BEV feature map to be processed be... (Can be a camera BEV feature map) or LiDAR BEV feature map The forward propagation process of the confidence estimation network can be divided into four steps, and the overall formula chain is expressed as:

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] The first step, through Convolutional layers capture the local spatial correlations of BEV features. In the formula... for Convolution kernel parameters, This indicates that edge padding is used during convolution operations to ensure the output feature map... Spatial Dimensions and Input Consistency , The number of channels was adjusted to (much smaller) (This achieves feature dimensionality reduction to reduce computational cost). The core function of convolution is to aggregate the feature information of each pixel and its eight neighboring pixels. Because the physical regions corresponding to adjacent pixels in the BEV space have environmental correlation (such as different parts of the same target), capturing local correlation can improve the global consistency of confidence estimation.

[0057] The second step is to introduce a nonlinear transformation using the ReLU activation function. The expression for the ReLU function is: Its function is to filter out negative values ​​in the feature map, thereby enhancing the network's ability to fit the most characteristic distribution of complex fields.

[0058] The third step, through Convolutional layers compress feature channels to 1. In the formula... for Convolution kernel parameters, this operation will The number of channels from Compress to 1, output single-channel feature map . Convolution only performs a linear combination of channel dimensions without changing the spatial dimension. Its core purpose is to map high-dimensional BEV features into single-channel "reliability intermediate features", providing a basis for the normalization of subsequent confidence values.

[0059] The fourth step is to map the intermediate features to confidence values ​​in the [0,1] interval using the Sigmoid activation function. The expression for the Sigmoid function is: Its function is to The eigenvalues ​​that take any value are converted into numerical values ​​between 0 and 1, forming the final confidence map. The physical meaning of the confidence level is clear: Indicates the first In each sample, the BEV coordinates The characteristics at that location are completely reliable; This indicates that the feature at that location is completely unreliable; when the value is between 0 and 1, the larger the value, the higher the reliability. For both camera and LiDAR modes, the camera confidence map is calculated using the above network. confidence map of lidar Because the two types of network parameters are not shared, the reliability evaluation criteria for camera features and lidar features can be learned separately, ensuring that the confidence estimation results are highly consistent with the modal characteristics.

[0060] (iv) Dynamic fusion based on confidence gating

[0061] The core objective of gated fusion is to dynamically generate pixel-level fusion weights based on a bimodal confidence map using a lightweight gating function. This achieves a refined fusion logic where "high-confidence modes dominate fusion, while low-confidence modes provide supplementary support," while ensuring high computational efficiency and clear physical meaning of the weights. The process consists of two steps: weight calculation and weighted fusion, both performed at the pixel level, fully leveraging the advantages of different modes in local regions of the BEV space.

[0062] Dynamic fusion weight calculation:

[0063] The calculation of dynamic fusion weights is based on the confidence difference between the two modes. A gating function maps the confidence difference to weight values ​​in the 0-1 range, ensuring that weight allocation is directly linked to modal reliability. Let the first... The camera confidence level at each pixel is The confidence level of the lidar is Then the fusion weight at that pixel The calculation formula is:

[0064] ;

[0065] The physical meaning and function of each parameter in the formula are as follows: , These are learnable positive scaling parameters (initially set to 1.0 and adaptively adjusted during training), used to adjust the influence of camera confidence and LiDAR confidence on the weights, respectively. For example, if it is found during training that camera features are more reliable in object classification tasks, It will automatically increase, enhancing the contribution of camera confidence to the weights;

[0066] Similar to this, it ensures that the influence intensity of the lidar confidence level can be dynamically optimized. These are learnable bias parameters (initially set to 0) used to adapt to the inherent differences in characteristics between the two modes. For example, lidar is inherently superior to cameras in distance measurement accuracy, and this can be achieved through fine-tuning. When the confidence levels of the two modes are equal, the weights are slightly biased towards the lidar, so as to make reasonable use of the inherent advantages of the modes.

[0067] The function's purpose is to convert confidence level differences. The weights are mapped to the 0-1 interval, ensuring that the weights have a clear probabilistic meaning. The larger the value, the higher the proportion of camera features fused at that pixel, while avoiding fusion imbalance caused by excessively large or small weight values.

[0068] Using the above formula, the weights of all pixels in the BEV space are calculated one by one, ultimately generating a dynamic fusion weight map. .

[0069] Pixel-level weighted blending:

[0070] The core of pixel-level weighted fusion is to perform element-wise multiplication of the dynamic weight map and the bimodal BEV feature map, and then generate the final fused feature map through linear combination, ensuring that the features of each BEV pixel are dominated by the optimal mode at that pixel. Let the camera BEV feature map be at the... The feature vector at pixel is (“:” indicates all channels), the feature vector of the corresponding pixel in the LiDAR BEV feature map is The weight value is Then the fused feature vector at that pixel The calculation formula is:

[0071] ;

[0072] in the formula This represents an element-wise multiplication operation, which applies weight values ​​to each channel of the feature vector to ensure that the weights have a consistent effect on all semantic information of the features. The fusion weights are the features of the LiDAR and the camera weights. The sum is always 1, ensuring that the feature amplitude will not increase or decrease abnormally due to the weight superposition during the fusion process, thus avoiding interference with subsequent perception tasks.

[0073] Due to the weighted graph For single channel (dimension) ), while the BEV feature map is multi-channel (dimensional) In actual calculations, the dimensions of the weight graph need to be expanded using a broadcast mechanism. Ensure that it is perfectly aligned with the dimension of the feature map.

[0074] Finally, a BEV fusion feature map is output through pixel-by-pixel weighted fusion. .

[0075] like Figure 2 and Figure 3 As shown, based on the above analysis, visualization results of multimodal confidence and fusion weights were generated for different day and night scenarios. It can be clearly seen that in driving scenarios with insufficient light at night, the camera's confidence is significantly lower than that in well-lit daytime scenarios due to imaging limitations; correspondingly, the camera's weight in the feature fusion process is also dynamically adjusted downwards with the confidence level.

[0076] (V) Perception Decision Making and 3D Target Detection

[0077] Based on BEV fusion feature maps, the core task of autonomous driving environmental perception—3D target detection—is completed. This outputs the target category, 3D spatial bounding box, and detection confidence score, providing accurate surrounding environment information for downstream decision-making, planning, and vehicle control. An anchor-free detection framework is adopted to avoid the problems of traditional anchor-based methods, such as reliance on experience in anchor box design and high computational cost. Simultaneously, multi-task prediction achieves comprehensive output of target information. The perception decision-making process consists of two steps: feature enhancement and multi-task prediction. The overall technical workflow revolves around semantic enhancement of fused features, prediction of key target information, and result filtering, ensuring a balance between detection accuracy and real-time performance.

[0078] Semantic enhancement is performed on the BEV fusion feature map using convolutional blocks to improve its target discrimination capability. Although the BEV fusion feature map integrates bimodal information, some regions still exhibit semantic ambiguity (such as small targets or occluded targets), necessitating further aggregation of local semantic information through convolutional operations. The formula for the feature enhancement process is expressed as:

[0079] ;

[0080] in, For cascaded 3×3 convolution-batch normalization-ReLU modules, To detect specific convolutional kernel parameters, a 3×3 convolution is used to capture local semantic associations at the target level. Batch normalization suppresses feature distribution shift, and the ReLU activation function enhances non-linear expressive power. Finally, a semantically enhanced fused feature map is output. This feature map significantly improves the semantic discriminative power of the target, laying the foundation for subsequent multi-task prediction.

[0081] Multi-task prediction is performed based on the enhanced fused feature map, and the results simultaneously output a target center heatmap, 3D bounding box parameters, and detection confidence. All three prediction results are generated through independent 1×1 convolutional layers, ensuring independent optimization of prediction parameters for each task and avoiding interference between tasks.

[0082] Target center heatmap prediction: The target center heatmap is used to locate the center position of each target in the BEV space. Essentially, it is a probability density map, where each pixel value represents the probability that the location is the center of a certain type of target. Let the predicted heatmap be... ( If the target category is denoted as the number of categories (e.g., vehicles, pedestrians, bicycles, etc.), then the formula for generating the heatmap is:

[0083] ;

[0084] in, To predict convolution kernel parameters for heatmaps, The function maps predicted values ​​to probability values ​​in the interval [0,1]. The pixel with the highest probability value in the heatmap is the center of the target for the corresponding category. The center coordinates of all targets can be filtered out using the non-maximum suppression (NMS) algorithm. This marks the initial positioning of the goal.

[0085] 3D Bounding Box Parameter Prediction: 3D bounding box parameters describe the 3D spatial shape and position of the target in BEV space, including the 3D coordinates (x, y, z) of the target center, the target's length l, width w, height h, and target orientation angle θ, totaling 7 parameters. Let the predicted 3D bounding box parameter diagram be... Then the formula for generating the parametric plot is:

[0086] ;

[0087] in, Predict convolutional kernel parameters for bounding boxes. This formula directly outputs the 7 bounding box parameters corresponding to each BEV pixel. Since the target center has been located through the heatmap, only the target center coordinates need to be extracted. , Parameter value at ) This allows us to obtain the 3D bounding box parameters of the target. To ensure the correct physical meaning of the parameters, a smooth L1 loss function is used during training to optimize the prediction error of the bounding box parameters and avoid the influence of outliers.

[0088] Detection confidence prediction: Detection confidence is used to evaluate the accuracy of 3D bounding boxes and filter out invalid detection results with low confidence. Let the predicted detection confidence map be... The formula for generating the confidence plot is:

[0089] ;

[0090] in, To predict the convolution kernel parameters for confidence scores, the Sigmoid() function maps the predicted values ​​to the [0,1] interval. Similarly, it extracts the target center coordinates ( , Confidence value at ) The detection confidence level is used as the target's detection confidence level; when the confidence level is higher than the preset threshold, the detection result is retained; otherwise, it is filtered to ensure that the output perception result is highly reliable.

[0091] The selected target categories, 3D bounding box parameters, and detection confidence scores are correlated to form the final 3D target detection result. This result contains the core environmental information required for autonomous driving decisions.

[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A pixel-level confidence-gated multimodal BEV sensing feature dynamic fusion method, characterized in that, The method includes: Acquire multi-view camera images and lidar point cloud data, process the multi-view camera images and lidar point clouds, and output multi-view camera 2D feature maps and lidar 2D feature maps. Based on the 2D feature maps of multi-view cameras and LiDAR, the camera BEV feature map and LiDAR BEV feature map are output through BEV space mapping. Pixel-level confidence scores are calculated for both the camera BEV feature map and the LiDAR BEV feature map to generate a camera pixel-level confidence score map and a LiDAR pixel-level confidence score map. The dual-modal confidence difference is calculated based on the camera pixel-level confidence map and the LiDAR pixel-level confidence map, and a dynamic fusion weight map is dynamically generated through a gating function; The dynamic fusion weight map is multiplied and linearly combined with the camera BEV feature map and the lidar BEV feature map element by element to perform pixel-level weighted fusion and output the BEV fusion feature map. Based on the BEV fusion feature map, predict the target center heatmap, 3D bounding box parameters and detection confidence, and output the final 3D target detection result; The step of performing pixel-level confidence calculations on the camera BEV feature map and the LiDAR BEV feature map respectively includes: Lightweight convolutional networks with identical structures but different parameters are used to process camera BEV feature maps separately. and LiDAR BEV feature map ; The forward propagation process of a lightweight convolutional network is as follows: ； ； ； ； in, This represents the intermediate feature map obtained after performing a convolutional transformation on the BEV feature map through the first convolutional layer. Indicates the intermediate feature map Feature map after applying a nonlinear activation function This indicates that the activated feature map is processed through the second convolutional layer. The weights obtained after convolution transformation generate feature maps. This represents the pixel-level weight map generated through weight normalization. The parameters are 3×3 convolution kernel. Fill the edges. for Convolution kernel parameters, The BEV feature map to be processed; Output the camera pixel-level confidence map respectively and LiDAR pixel-level confidence map ; The generation of the dynamic fusion weight map and the pixel-level weighted fusion specifically include: Calculate the coordinates in BEV space as follows fusion weights at pixel locations : ； Where b represents the sample index in the batch dimension, used to indicate the current fusion weights. The corresponding b-th input sample; , For learnable positive scaling parameters, Generate a dynamic fusion weight graph for learnable bias parameters. , and The coordinates in the BEV space are respectively The camera confidence and LiDAR confidence at each pixel; Dynamically fused weight graphs are generated through a broadcast mechanism. Dimensional expansion to ; Coordinates in the camera's BEV feature map are calculated using pixel-by-pixel weighted fusion. fused feature vector at pixel : ； in, Let represent the feature vector of the camera's BEV feature map at pixel (u,v). This represents the feature vector of the BEV feature map of the LiDAR at the (u,v)th pixel. Output BEV fusion feature map .

2. The method according to claim 1, characterized in that, The processing steps for the 2D feature maps from the multi-view camera include: The tensor representation of the multi-view camera image is as follows: ,in For batch size, , These are the height and width of a single-view image, respectively. Layered feature extraction: ； in , The parameters of the l-th convolutional kernel are... For convolution operations, For batch normalization operations, For activation function, The downsampling factor for the l-th layer is... This is a downsampling operation; After processing through layer L, the output is a 2D feature map of the multi-view camera. ,in The number of feature channels, , represents the spatial dimension of the feature map.

3. The method according to claim 1, characterized in that, The processing steps for the lidar point cloud include: The lidar point cloud is a collection of 3D point clouds. ,in This represents the total number of points in a single frame of the point cloud. The first 3D spatial coordinates of a point For the first Reflectance at each point; Transform the prism into a prism, and let the side length of the prism be . ,but Columns The included point set is: ； in, For the first The coordinates of the top left corner of the cylinder in the BEV plane; Let the column be Includes Calculate the statistical feature vector for each column based on the given points. : ； in This represents the mean of the coordinates of the points within the cylinder. , , , These are the maximum values ​​for the corresponding coordinates. , , , These are the minimum values ​​at the corresponding coordinates. , , , These are the standard deviations of the corresponding coordinates; Statistical eigenvectors of each column Mapping to a high-dimensional feature space via fully connected layers: ； in, This is the weight matrix of the fully connected layer. The bias vector is then downsampled and fused with features through a convolutional layer to output a 2D feature map for the LiDAR radar. ,in This represents the number of feature channels of the lidar. represents the spatial dimension of the feature map.

4. The method according to claim 2, characterized in that, The camera BEV mapping process of the camera BEV feature map is as follows: Based on the Transformer spatial cross-attention mechanism, a BEV query grid is generated, and the tensor representation of the BEV query grid is as follows: ,in , , , The length and width of the BEV space correspond to the actual physical space. For grid resolution; The association between the BEV query mesh and the 2D feature maps of multi-view cameras is learned through spatial cross-attention. The calculation process is as follows: ； in For the first Feature maps from each perspective For attention dimension; Attention-weighted features from all perspectives are aggregated and channel adjustments are performed. The aggregation process is as follows: ； The channel adjustment process is represented as follows: ； in, Number of camera angles This is a temporary camera BEV feature map after multi-view feature aggregation. for Convolution kernel parameters; Output camera BEV feature map .

5. The method according to claim 3, characterized in that, The lidar BEV mapping process for the lidar BEV feature map is as follows: Upsampling is performed using transposed convolution. The upsampling process is represented as follows: ； in, This represents the BEV feature map of the LiDAR obtained through spatial size recalibration. This represents the intermediate feature map obtained from the point cloud feature extraction by lidar. These are the parameters of the transposed convolution kernel. The stride is the value of the transposed convolution. pass Convolution adjusts the number of channels in the upsampled feature map of the LiDAR to ensure compatibility with the number of channels in the BEV feature map of the camera. The channel adjustment process is represented as follows: ； in, for Convolution kernel parameters are used to obtain the temporary lidar BEV feature map. ; Spatial position calibration based on sensor extrinsic parameters is calculated using the following formula: ； in, The coordinates of the pixels in the camera's BEV feature map. This represents the coordinates of the pixel in the temporary BEV feature map of the LiDAR. The rotation matrix outputs the LiDAR BEV feature map. .

6. The method according to claim 1, characterized in that, The final 3D object detection result output specifically includes: Semantic enhancement of BEV fused feature maps using convolutional blocks: ； in, To detect convolution kernel parameters, output a semantically enhanced fused feature map. ; Multi-task prediction based on the enhanced fused feature map: Generate target center heat map Filter out the center coordinates of all targets To locate the target category; Generate 3D bounding box parameter map Read and extract the center coordinates of the target Parameter value at Obtain the 3D bounding box parameters of the target; Generate detection confidence map Extract the center coordinates of the target ( , Confidence value at ) The detection confidence level is used as the target's detection confidence level. If the confidence level value is higher than a preset threshold, the detection confidence level result is retained; otherwise, it is filtered out. in To predict convolution kernel parameters for heatmaps, Predict convolution kernel parameters for bounding boxes. Predict convolution kernel parameters based on confidence level; The filtered target categories, 3D bounding box parameters, and detection confidence scores are correlated to form the final 3D target detection results.

Images