High-performance low-cost multi-modal data fusion model and equipment for complete driving environment state perception

Abstract

The invention discloses a high-performance low-cost multi-modal data fusion model and equipment for complete driving environment state perception, and the model comprises a feature extraction module which is used for extracting initial potential features of a camera image and a laser radar point cloud; a probability cross attention mechanism constructs a feature fusion model of uncertainty perception; a query initialization model capable of reducing uncertainty is adopted, and the initialization process of query is enhanced by utilizing two-dimensional target detection priori; and based on a robustness optimization training strategy of query denoising, stable optimization is realized under uncertainty. A nuScenes data set is selected for training, verifying and testing the comprehensive performance of the model, and the TensorRT realizes CUDA acceleration and is quantitatively deployed on a vehicle-mounted calculation experiment platform. According to the multi-modal data fusion system, a perception state uncertainty explicit measurement calculation theory based on data and a model is innovated, a perception model for coping with environment uncertainty is established, the completeness, reliability and reconfigurability requirements of a driving environment are met, and driving safety and decision accuracy are ensured.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent connected vehicle environment perception, and in particular relates to a high-performance, low-cost multimodal data fusion model and equipment for complete driving environment state perception. Background Art

[0002] An autonomous driving system is usually divided into three key modules: environmental perception, decision-making, and motion control. Among them, the perception system provides comprehensive and accurate perception capabilities for autonomous driving vehicles and is the key foundation for safe autonomous driving. In recent years, environmental perception technology for intelligent connected vehicles has made significant progress. However, the widespread deployment of driverless cars has not yet been realized. At present, one of the main reasons restricting the realization of high-level intelligent driving at L4 and above is the insufficient accuracy of driving environment state estimation in complex traffic scenarios. This problem makes it difficult for existing methods to simultaneously meet multiple requirements such as high precision, high reliability, and high safety in real driving scenarios. To this end, the present invention mainly focuses on and tries to improve two aspects of perception system uncertainty: (1) external traffic environment, such as bad weather, lighting changes, etc.; (2) limitations of internal perception algorithms, such as low accuracy and robustness.

[0003] Autonomous vehicles typically use a variety of sensors, such as cameras and lidar. Although Tesla has adopted a purely visual approach, most research still tends to combine multiple sensors to achieve reliable object recognition and localization in 3D space. Visual cameras use the principle of perspective imaging and can provide high-resolution semantic information. However, their depth perception is limited, and their detection capabilities are affected at night, in rainy days, or in strong light. In contrast, lidar point clouds can accurately capture the three-dimensional structural information of the environment, but their point cloud data is sparse and disordered, lacking semantic information. In addition, occlusions in traffic scenes and limited sensor coverage may hinder the acquisition of complete scene information. Relying solely on single-modality data will inevitably lead to inaccurate perception in some scenarios. In contrast, multi-source heterogeneous sensor fusion uses the advantages of visual cameras and lidar to make up for the shortcomings of a single sensor, thereby suppressing the interference of environmental noise.

[0004] Existing methods attempt to solve the uncertainty problem from various aspects. For example, the YOLOv8-QSD algorithm significantly improves the detection accuracy of small objects in autonomous driving scenarios by introducing diverse branch blocks (DBB) and query mechanisms. CenterPointSE, a single-stage anchor-free 3D object detection algorithm based on lidar point cloud, greatly improves the detection accuracy and robustness by improving the 3D and 2D backbone networks, introducing lightweight IoU branches and foreground point segmentation auxiliary branches. Voxel-RCNN-Complex performs well in complex traffic scenes by modifying the 3D and 2D backbone networks, adding residual structures, self-calibrated convolutions, and channel / spatial attention mechanisms, especially in occlusion problems. However, the above methods rely on a single modality (using only lidar or camera) for target detection and tracking, resulting in poor performance when information is missing or inaccurate. For example, lidar is prone to losing information when detecting distant or small targets, while cameras are prone to misidentifying targets in low light or complex backgrounds. These problems will increase the uncertainty of detection and tracking. The algorithm TransFusion introduces a soft association mechanism, which significantly improves the robustness of 3D object detection in low-quality image conditions by adaptively establishing associations between LiDAR point clouds and image pixels. CMT achieves end-to-end multimodal 3D object detection without view conversion by implicitly encoding 3D position information, and maintains strong robustness in the case of sensor failure by introducing mask modality training. Although these methods have improved the performance of driving environment state perception to a certain extent, they usually assume that sensor data is complete and accurate, ignoring the uncertainty in practical applications. Therefore, these methods often find it difficult to effectively deal with uncertainty in dynamic traffic environments, especially in mixed traffic conditions or bad weather.

[0005] Uncertainty issues have received extensive attention in areas such as behavior recognition and camouflaged object detection. However, these methods have not yet been explored and applied in 3D fusion detection. The uncertainty problem in 3D fusion detection is particularly challenging due to the unique challenges of complex environments and specific driving scenarios. Previous solutions for other specific areas cannot be directly applied here. 3D fusion detection involves integrating data from multiple sensors, each of which has its inherent uncertainties. If not properly managed during the fusion process, these uncertainties may be amplified, resulting in poor detection performance. In addition, in actual driving scenarios, vehicles must navigate in highly dynamic and uncertain traffic environments. Traditional detection methods have difficulty maintaining high accuracy and robustness under such uncertain conditions. Summary of the invention

[0006] In order to solve the problem of inaccurate estimation of driving environment state in complex traffic scenes that has been ignored by previous work, and to ensure that intelligent vehicles meet multiple requirements such as high precision, high reliability, and high safety in real driving scenes. The present invention draws on the successful experience in motion recognition and camouflaged target detection, and combines the characteristics of multi-sensor data fusion for autonomous driving, and proposes a high-performance, low-cost multimodal data fusion model UAFormer (Uncertainty-Aware Cross Modal Transformer) for complete driving environment state perception, which solves the uncertainty problem from multiple aspects. The implementation of the UAFormer model of the present invention includes the following steps:

[0007] S1: Extract the initial potential features of the camera image and lidar point cloud. In the lidar point cloud processing branch, the present invention uses a voxel-based method to divide the three-dimensional space into a fixed-size voxel grid through voxelization, and performs a three-dimensional sparse convolution operation to effectively handle the sparsity of the point cloud data and reduce the amount of calculation and memory requirements. Then, the three-dimensional voxel features are compressed into a two-dimensional feature map, and the features are further extracted using two-dimensional convolution. Finally, the bird's-eye view (BEV) features of the point cloud data are obtained and input into the Transformer module. In the camera image processing branch, the Mamba model is used to extract multi-scale surround image features. Taking advantage of its advantages in capturing long-range dependencies and efficiently processing long sequence data, the image is divided into patches and mapped to a high-dimensional space through linear transformation. After combining position embedding, it is used as the input of the Transformer module.

[0008] S2: Constructing a feature fusion model for uncertainty perception. The traditional Transformer model uses a deterministic attention mechanism and cannot effectively quantify uncertainty problems in complex traffic scenarios, such as occlusion, small-scale targets, and bad weather. In order to solve this problem, the present invention proposes an uncertainty perception fusion model based on a probabilistic cross-attention mechanism, innovates the explicit measurement calculation theory of perception state uncertainty based on data and models, models the attention score as a Gaussian distribution, and uses the mean and variance to represent the predicted central value and uncertainty range. In order to train this model, the present invention adopts a reparameterization technique to introduce noise variables into the standard normal distribution, and maps them to the target distribution through differentiable transformations, so that the model can be optimized by gradient descent. This method can establish a perception model that copes with environmental uncertainty and meets the completeness, reliability, and reconfigurability requirements of the driving environment.

[0009] S3: Construct a query initialization model that can reduce uncertainty. Recent studies have generated queries in 3D space based on randomly distributed reference points and optimized them into network parameters. In three-dimensional object detection, high-quality query initialization is crucial for fast convergence and improved detection accuracy, especially for the detection of small-scale objects and occluded objects. The present invention proposes a query initialization model that can reduce uncertainty, which generates dynamic queries using the prior knowledge of 2D object detectors and monocular 3D object detectors, and combines randomly initialized queries with queries updated from historical frames to generate hybrid queries, thereby enhancing the robustness of the method. In the process of generating dynamic queries, 3D reference points are generated by fusing multi-view images and camera parameter transformations, and non-maximum suppression and cross-attention processing are performed to optimize detection performance, especially for the detection of small-scale objects.

[0010] S4: Construct a robust optimization training strategy based on query denoising. In complex traffic scenarios, uncertainty factors such as bad weather and target occlusion will hinder the learning of query-based methods and the rapid convergence of models. To meet these challenges, the present invention proposes a robust optimization training strategy based on query denoising. This strategy introduces noise in the training process by generating noisy queries with different uncertainty levels, so that the model can better understand and adapt to uncertainty. The present invention adds random noise to the real label objects, and divides the noisy queries into positive samples, negative samples, and ignored samples according to the 3D intersection over union (IoU). Positive samples are high-quality data, which is helpful for model learning; negative samples are low-quality data, which helps the model filter unreliable information; ignored samples avoid interference. In this way, the robustness of the training process and the stability of the model can be enhanced, and the accuracy of fusion detection can be improved.

[0011] S5: Finally, the fusion detection network is trained end-to-end through the model's loss function. TensorRT is used to accelerate the inference model and quantitatively deploy it on the in-vehicle computing test platform.

[0012] Beneficial effects of the present invention:

[0013] (1) A high-performance, low-cost multimodal data fusion model and method for complete driving environment state perception is proposed. An innovative data- and model-based explicit measurement calculation theory for perception state uncertainty is developed. By acquiring and modeling uncertainties in dynamic traffic environments such as small-scale targets, target occlusions, and multi-source heterogeneous sensor information, a perception model that copes with uncertainty is established to meet the completeness, reliability, and reconfigurability requirements of the driving environment, thereby ensuring driving safety and decision-making accuracy.

[0014] (2) The present invention improves the robustness of driving environment state perception in the face of various uncertain factors from multiple angles through a feature extraction module, an uncertainty-aware feature fusion model, a query initialization model that can reduce uncertainty, and a robustness optimization training strategy based on query denoising, thereby achieving complete and credible perception of the driving environment state. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the invention implementation of the present invention;

[0016] Figure 2 It is the overall network architecture of the network model UAFormer proposed in the present invention;

[0017] Figure 3 It is the uncertainty-aware feature fusion model proposed by the present invention;

[0018] Figure 4 It is a query initialization model proposed by the present invention that can reduce uncertainty;

[0019] Figure 5 This is a schematic diagram of the installation of the perception sensor of the unmanned driving experimental platform of the present invention; DETAILED DESCRIPTION

[0020] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with the embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention. Figure 1 The overall network architecture of the UAFormer model is shown in Figure 2 As shown. The uncertainty-aware multimodal data fusion framework of the present invention first processes the input data, including lidar point clouds and camera images, which extract initial potential features through their respective feature extraction modules. The extracted features are positionally encoded as keys and values, and are used in the probabilistic cross-attention mechanism to construct an uncertainty-aware feature fusion model. A query initialization model that can reduce uncertainty is designed, and the query initialization process is enhanced using two-dimensional target detection priors. Different from the cross-modal cross-attention in a deterministic setting, the attention of the uncertainty-aware fusion module of the present invention is sampled from a Gaussian distribution and uses NLL loss. The present invention also designs a robustness optimization training strategy based on query denoising to further improve the robustness of the model and its convergence ability to uncertainty factors. Finally, two feedforward networks (FFNs) receive updated query vectors respectively to predict 3D bounding boxes and categories. The implementation of the UAFormer network model of the present invention includes the following steps:

[0021] Step 1: Build a feature extraction module to extract the initial potential features of camera images and lidar point clouds

[0022] In the laser radar point cloud processing branch, the voxel-based method divides the three-dimensional space into voxels of fixed size and performs feature extraction and processing in the voxels. Compared with the point-based method, it is easier to process sparse point cloud data and can better balance the accuracy and real-time performance of the algorithm. Therefore, the present invention selects the voxel-based method in the laser radar point cloud processing branch. Figure 2 As shown, first, the point cloud data is voxelized to divide the three-dimensional space into a voxel grid of a fixed size. This step divides the point cloud data into smaller voxel units, making subsequent processing more orderly and efficient. Next, in order to efficiently process the sparsity of point cloud data and reduce the amount of calculation and memory overhead, a three-dimensional sparse convolution operation is performed on the voxel features. The three-dimensional sparse convolution greatly reduces the computational complexity and memory requirements by performing calculations only on non-empty voxels. Then, the three-dimensional voxel features are compressed in the height direction (Z axis) to obtain a two-dimensional feature map. Finally, two-dimensional convolution is used to further extract and encode the compressed two-dimensional feature map. Through multi-layer convolution operations, higher-level features are extracted layer by layer, and the resulting feature map can be regarded as a bird's-eye view feature of the point cloud data. Point cloud BEV features Combined with its position embedding, it is used as one of the inputs of the uncertainty-aware feature fusion model. The above content can be described by the following formula:

[0023]

[0024] in, Represents the input 3D point cloud data; Voxelize represents voxelization; SparseConv3D represents 3D sparse convolution operation; Compress Z Indicates compressing the 3D voxel features in the height direction (Z axis). Conv 2D Indicates that the compressed two-dimensional feature map is further extracted and encoded using two-dimensional convolution, and the BEV view of the lidar point cloud is finally obtained.

[0025] In the multi-view camera image processing branch, the Mamba model is used to extract multi-scale surround image features. This has several significant advantages: it has obvious advantages in capturing long-range dependencies in images and can capture detail information and global semantic information at the same time; it has significant advantages in computational efficiency and can achieve nearly linear computational complexity when processing long sequence data, especially when processing high-resolution images. Therefore, the present invention uses the Mamba model to extract image features. Specifically: First, the input image is divided into small blocks of size P·P, each small block is called a patch, and the kth patch is denoted as I′[k]. In this way, we can get patches, where H*W is the size of the input image. Then the M patches are flattened into a one-dimensional vector x k = Flatten(I′[k]), k = 1, 2, …, M, whose size is P 2 C, where C is the channel dimension of the image feature. In order to map each patch to a high-dimensional space, the present invention needs to perform the following linear transformation: k =W p x k +b p , W p is the projection matrix, b p is the bias. Through this linear transformation, the present invention converts the low-dimensional representation (i.e., one-dimensional vector) x of each patch k Mapping to a high-dimensional representation z k Since image data has spatial location information, it is necessary to represent the image patch in a high-dimensional representation z k Add position embedding E pos [k] to retain the spatial position information and obtain a high-dimensional representation z′ with position information k , which can be expressed by the formula, z′ k =z k +E pos [k] The representations of all patches are combined into a sequence to form the input of the Mamba model. The Mamba model is a deep learning architecture based on the state space model (SSM), which processes the input sequence through the state space equation. First, Mamba uses the state space equation to describe the state update and output of the system at each time step (here corresponding to each patch in the sequence). The state update equation and output equation can be described by the following formula:

[0026] h t =Ah t-1 +Bz t ′,y t =Ch t

[0027] Among them, z t ′The input feature vector of the current time step; h t is the hidden state vector at the current time step t, representing the internal state of the system; h t-1 is the hidden state vector of the previous time step; y t The output of the current time step t; A is the state transfer matrix, which is used to transform the hidden state h from the previous time step t-1 Linear transformation to the current time step h t ; B is the input matrix, used to convert the input z t ′ is mapped to the hidden state; C in this formula is the output matrix used to map the hidden state ht Mapped to output y t .

[0028] The output feature y of each patch t Corresponding to the features of a patch. Finally, these patch features y t Combined with its position embedding, it is input into the feature fusion model.

[0029] Step 2: Build an uncertainty-aware feature fusion model

[0030] The traditional cross-modal attention decoder assigns different attention or weights to different parts of the input data in a deterministic way, that is, directly calculating the deterministic attention score between the query (Q) and the key value (K). Where · represents the dot product operation of the vector, and d is the dimension of the query vector and the key vector. However, the attention weights learned by the model in this way will not change due to random factors or other uncertainties, which limits the ability to effectively quantify the uncertainty in the prediction. In complex traffic scenarios, uncertainty problems such as occlusion, small-scale targets, and bad weather are often encountered. These uncertainties can cause sensor failure or data quality degradation. Deterministic attention cannot capture the noise and distribution of the input, and traditional fusion detection models cannot effectively quantify its prediction uncertainty.

[0031] In order to solve the above problems, the present invention proposes a feature fusion model based on the probability cross attention mechanism to construct uncertainty perception for the multimodal feature fusion module. In order to quantify the uncertainty in the prediction, the probability cross attention mechanism converts the attention score α into ij Modeled as a Gaussian distribution:

[0032]

[0033] By the mean μ ij and variance It can clearly indicate the central value of the prediction and its degree of uncertainty. They are both probabilistic parameters and are random during training and inference. ij Represents the center position of the predicted value, that is, the most likely attention score. Indicates the uncertainty range of the predicted value. The larger the variance, the higher the uncertainty. In the face of noise and data changes, probability distribution can better capture and describe the uncertainty of the data. In order to estimate the mean μ of the Gaussian distribution ij and variance The present invention develops a simple multi-layer perceptron (MLP) using a multi-layer perceptron (MLP) to query the vector q i and key vector k j As input, generate a Gaussian distribution with mean μij and variance Among them, i represents the index of the query, j represents the index of the key and value, and k represents the index of all keys, which is used for normalization calculation:

[0034] μ ij =MLP μ (q i ,k j )

[0035]

[0036] Since the random variable μ ij and Due to the randomness generated in the process of introducing , it is difficult to perform gradient descent on the parameters of the random variable distribution. In order to enable the model to be trained by optimization algorithms such as gradient descent, the present invention adopts the reparameterization technique by introducing an auxiliary noise variable ε, which obeys the standard normal distribution. This noise variable is then mapped to the target distribution through a differentiable transformation, namely

[0037] α ij =μ ij +σ ij ε

[0038] The above steps convert the random process that is difficult to optimize directly into a deterministic function about the parameters, so that the model containing random variables can be trained using standard gradient descent methods.

[0039] By modeling the attention scores as Gaussian distributions, the probabilistic cross-attention mechanism is able to quantify the uncertainty in the predictions. When calculating the attention weights, not only the result of the dot product is considered, but also this uncertainty is taken into account, using a weighted sum of the attention scores with a Gaussian distribution:

[0040]

[0041] Among them, Q represents the query vector, which is matched with other vectors to calculate the similarity with other vectors; K is the key vector that represents the characteristics of each element in the input sequence; V is the value vector that corresponds to the key vector in the input sequence. In the specific calculation formula, α ij represents the attention score of the i-th query vector and the j-th key vector, divided by the i-th query vector and all key vectors It means that the similarity scores of all key vectors are normalized, and the similarity scores are converted into probability distributions. The normalized weights are the same as the corresponding v jThe vectors are multiplied to get the weighted output. Finally, all weighted Value vectors are summed to get the final output of the probabilistic cross-attention mechanism.

[0042] The detailed network architecture of the uncertainty-aware feature fusion model is as follows: Figure 3 As shown in the figure. The fusion-aware decoder consists of 3 layers, each of which contains a self-attention mechanism and a cross-modal attention. Following the Transformer architecture, the query vector undergoes a self-attention operation after initialization. Subsequently, cross-attention is performed with the lidar point cloud and image features to fuse multimodal features. The encoding layer outputs the updated query as input to the next layer after processing through the feedforward network. After 3 layers of fusion encoding, the final updated query vector is obtained.

[0043] To train the decoder with uncertainty-aware parameters, the present invention uses a negative log-likelihood (NLL) loss function, which can optimize for probability outputs and help the model better learn the uncertainty in the data and how to make predictions based on the existing uncertainty.

[0044]

[0045] represents the negative log-likelihood loss function.

[0046] In summary, the present invention innovates the explicit measurement calculation theory of the uncertainty of the perceptual state based on data and model, and transforms the attention score α into ij Modeling as a Gaussian distribution captures the uncertainty in target prediction, so that it can more robustly cope with changes and noise in multimodal data fusion during driving environment perception, and achieve high-performance and low-cost environmental state perception.

[0047] Step 3: Initialize the query model to reduce uncertainty

[0048] Existing research usually generates queries in 3D space based on randomly distributed reference points and optimizes them. In the 3D object detection task based on the query framework, high-quality initialization queries are crucial to achieve fast convergence and improve detection accuracy, especially when considering uncertainty factors, some small-scale targets or occluded objects are usually not detected. The present invention proposes a query initialization model that can reduce uncertainty, which uses some prior knowledge obtained from the network training and inference process to enhance the initialization of the query. 2D object detection technology has made significant progress. These methods can provide rich prior information for query-based 3D detection, and 2D detectors usually show good perception capabilities at long distances and small objects. As Figure 4As shown, the present invention visualizes the predictions of the query-based 3D detector and the 2D detector, and the 2D detector usually shows excellent perception ability when detecting distant and small objects. However, the query generation method that relies solely on the 2D detector may miss some undetectable objects. To this end, the present invention introduces a hybrid query initialization method, which combines randomly distributed queries and dynamic queries generated by the 2D detector to further enhance the robustness of the method. The details are as follows:

[0049] For each input N view images, we first use a pre-trained 2D object detector (FCOS2D) and a monocular 3D object detector (FCOS3D) to obtain 2D and 3D detection results respectively to generate the final dynamic object query. The 2D object detector will output a set of 2D object detection boxes. Represents the set of 2D bounding boxes detected in the i-th view image, and the total number of bounding boxes is N, M m Represents the number of bounding boxes detected in the i-th view image.

[0050] In order to generate 3D reference points, the present invention also needs to combine the 3D properties predicted by the monocular 3D object detector (FCOS3D). Then the 2D bounding box can be mapped to the 3D space. This step involves the conversion of camera parameters. Specifically, the present invention estimates the three-dimensional position through coordinate transformation:

[0051] C 3d =E -1 K -1 [ud,vd,d,1]

[0052] Among them, C 3d Indicates the center position of the 3D space, E -1 and K -1 Denote the camera extrinsic matrix and the camera intrinsic matrix respectively, (u, v) denotes the center position of the 2D detection box, and d here denotes the depth value. Then, non-maximum suppression (NMS) is performed in 3D space to filter the crossed boxes and select the top N boxes ranked by confidence score. 2d The dynamic target query DQs is generated through the cross-attention module. The formula is as follows:

[0053]

[0054] in, Indicates that the 3D center C is converted using the corresponding camera parameters 3d Projected onto the i-th image, Represents the m feature maps of the i-th view, M represents the number of feature maps in different images i, and N represents the total number of bounding boxes. Bilinear represents the bilinear sampling function.

[0055] Since some objects may be ignored, the present invention also retains randomly initialized queries RQs. The final query set Q can be expressed as:

[0056] Q=Prop(DQs,RQs)

[0057] Prop is the abbreviation of the English word Propagation, which means combining and propagating DQs and RQs to form the final initialization query.

[0058] The optimization goal of the above task is:

[0059]

[0060] in, Initialize the model's loss function for queries that reduce uncertainty, is the loss function for 2D detection, is the loss function for 3D detection, λ 3d is a hyperparameter of the model, which is set to 0.3 in the present invention.

[0061] In summary, the query initialization model designed by the present invention to reduce uncertainty provides a query vector with prior information, which is helpful for detecting small-scale targets, reduces the uncertainty factors in query initialization, meets the completeness, reliability and reconfigurability requirements of the driving environment, and ensures driving safety and decision-making accuracy.

[0062] Step 4: Robust optimization training strategy based on query denoising

[0063] In complex traffic scenarios, the emergence of various perception state uncertainty factors (such as bad weather, target occlusion, small-scale targets, and feature interference from multi-source heterogeneous sensors) will significantly hinder the learning of query-based methods and the rapid convergence of models. These uncertainties not only affect the clarity and accuracy of input data, but also increase the difficulty of the model to process and recognize this information during training. In order to meet this challenge, the present invention proposes a robust optimization training strategy based on query denoising. This strategy combines query denoising to enhance the training process and achieve stable optimization under perception state uncertainty.

[0064] The robust optimization training strategy based on query denoising generates noisy queries with different uncertainty levels and classifies and processes them according to the characteristics of these noisy queries. This method enables the model to better understand and adapt to different levels of uncertainty by introducing noisy queries during the training process, thereby achieving more robust and stable performance optimization. Specifically, the present invention adds random noise to all true tag (GT) objects. Each GT object is trained using multiple noisy versions.

[0065] In order to distinguish different levels of uncertainty, the present invention defines an upper threshold and a lower threshold, respectively, denoted as α and β. These thresholds classify noise queries into three categories according to their uncertainty levels. The present invention determines the positive samples (denoted as Figure 2 The “Pos” in , i.e., samples with lower uncertainty, when the 3D intersection over union (IoU) between the noise query and the true value exceeds the β threshold, i.e. These positive samples are regarded as high-quality data during the training process because they provide clear and certain information that can help the model learn and predict better.

[0066] On the contrary, negative samples (denoted as Figure 2 “Neg” in ), that is, samples with higher uncertainty, when 3D IoU is lower than the α threshold, that is Corresponding samples. These negative samples are regarded as low-quality data during the training process because they may contain a lot of noise and uncertain information, which may lead to incorrect learning and prediction of the model. Therefore, these samples are used as negative examples during the training process to help the model distinguish and filter out unreliable information.

[0067] Queries with intermediate IoU values ​​are ignored (denoted as Figure 2 The “Ign” in the training data is ignored because they fail to provide clear uncertainty classification or certainty. These ignored samples are usually between positive and negative samples and may contain some useful information, but they may also contain a certain degree of noise. Therefore, during the training process, these samples are neither treated as positive samples nor as negative samples, but are directly ignored. This processing method helps the model avoid being disturbed by data with greater uncertainty, so that it can focus on learning on high-quality data.

[0068] During the model optimization process, the loss values ​​of positive and negative samples are calculated to determine the loss function of the robust optimization training strategy based on query denoising.

[0069]

[0070] in, and They represent the focal loss function (FocalLoss) for bounding box classification of positive samples and the L1 loss function for bounding box regression. In addition, the loss function of negative samples The focal loss function is used to distinguish the background (i.e., non-target).

[0071] By combining query denoising with processing noisy queries based on their uncertainty level, the robust optimization training strategy based on query denoising enhances the robustness and convergence of the training process, resulting in more stable and accurate results. Step 5: 3D spatial position and category prediction and overall loss design

[0072] Finally, two feed-forward networks are used to predict the 3D bounding box of the detected object using the updated query. and category information The prediction process is described as follows:

[0073]

[0074] Among them, f reg and f cla Denote the feedforward networks for regression and classification tasks respectively, and Query is the updated object query. Then, bipartite matching is applied to the one-to-one matching between the prediction and the true value, using focal loss for classification and L1 loss for 3D bounding box regression:

[0075]

[0076] in, represents the ensemble prediction loss function, is the classification loss function, which calculates the predicted classification information The loss between the real classification information c, Represents the regression loss function, calculating the predicted bounding box information The loss between the real bounding box information b, λ cls and λ reg They represent the weight hyperparameters of the classification loss term and the regression loss term, respectively.

[0077] Based on the above UAFormer model, the embodiment of the present invention also proposes to use TensorRT to accelerate the inference model, and quantize the UAFormer model and deploy it on the vehicle computing test platform. The environment perception system of the unmanned driving experimental platform is equipped with a variety of sensors, mainly including RoboSense RS-Fusion-P3 laser radar, Delphi ESR millimeter wave radar and Mox UC70 industrial camera, which are installed at the following locations: Figure 5As shown. The car is also equipped with Advantech MIC-7700 industrial computer as a computing platform. The RS-Fusion-P3 laser radar is installed on the roof, with a 32-line main laser radar (model RS-LIDAR-32) in the center for panoramic perception, and an inclined 16-line auxiliary laser radar (RS-LIDAR-16) is installed on each side for blind spot perception. The data collected by the underlying hardware such as laser radar, camera, GPS, IMU, etc. are packaged into topics and published through the ROS system. The environmental perception algorithm obtains the required data by subscribing to the corresponding topics, and packages the algorithm output into topics for publication. Finally, by opening the RViz visualization node, the camera image, laser point cloud information, and the three-dimensional bounding box detected by the fusion algorithm can be displayed in real time.

[0078] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A high-performance and low-cost multimodal data fusion model for complete driving environment state perception, characterized by: Obtained by the following steps: S1: Build a feature extraction module to extract the initial potential features of camera images and lidar point clouds; S2: Construct a feature fusion model for uncertainty perception based on the probabilistic cross-attention mechanism and establish a perception model to cope with environmental uncertainty; S3: Building a query initialization model that reduces uncertainty; S4: Construct a robust optimization training strategy based on query denoising; S5: Design the overall loss function of the multimodal data fusion model.

2. The high-performance and low-cost multimodal data fusion model for complete driving environment state perception according to claim 1 is characterized in that: The S1 specifically includes: In the lidar point cloud processing branch, a voxel-based method is used. First, the three-dimensional space is divided into a fixed-size voxel grid through voxelization processing, and a three-dimensional sparse convolution operation is performed on the voxel features. The three-dimensional sparse convolution is only calculated on non-empty voxels to deal with the sparsity of point cloud data and reduce the amount of calculation and memory requirements; then, the three-dimensional voxel features are compressed to a two-dimensional feature map in the height direction; finally, the compressed two-dimensional feature map is used for feature extraction and encoding, and finally the bird's-eye view features of the point cloud data are obtained. Combined with its position embedding, it is used as one of the inputs of the uncertainty-aware feature fusion model. The specific process is shown in the following formula: in, Represents the input 3D point cloud data; Voxelize represents voxelization; SparseConv3D represents 3D sparse convolution operation; Compress Z Indicates compressing the 3D voxel features in the height direction; Conv 2D Indicates that the compressed two-dimensional feature map is extracted and encoded using two-dimensional convolution, and the BEV view of the lidar point cloud is finally obtained.

3. The high-performance and low-cost multimodal data fusion model for complete driving environment state perception according to claim 2 is characterized in that: The S1 specifically includes: In the camera image processing branch, the Mamba model is used to extract multi-scale surround view image features, including: First, the input image is divided into small blocks of size P·P. Each small block is called a patch. The kth patch is denoted as I′[k]. We can get patches; Each patch is then flattened into a one-dimensional vector x k = Flatten(I′[k]), k = 1, 2, …, M, whose size is P 2 C. In order to map each patch to a high-dimensional space, the following linear transformation is performed, z k =W p x k +b p , W p is the projection matrix, b p is the bias. Through this linear transformation, the low-dimensional representation x of each patch is k Mapping to a high-dimensional representation z k Since the image data has spatial location information, it is necessary to add position embedding E pos [i] To retain this information, z′ k =z k +E pos [k]; combine the representations of all patches into a sequence to form the input of the Mamba model; The Mamba model processes the input sequence through state space equations. First, Mamba uses state space equations to describe the state update and output of the system at each time step. The state update equation and output direction are, h t =Ah t-1 +Bz t ′,y t =Ch t , the continuous state space model is discretized using the zero-order hold method, A d =exp(A△t),B d =A -1 (exp(A△t)-I)B is the discretization of the state transfer matrix and the input matrix respectively; the output feature y of each patch t Corresponding to the features of a patch; Finally, these patch features y t Combined with its position embedding, it is input into the feature fusion model.

4. The high-performance and low-cost multimodal data fusion model for complete driving environment state perception according to claim 1 is characterized in that: The S2 specifically includes: A feature fusion model based on the probabilistic cross-attention mechanism is designed to construct uncertainty perception. An explicit measurement calculation theory based on the uncertainty of the perception state of the data and model is proposed. In order to quantify the uncertainty in the prediction, the probabilistic cross-attention mechanism converts the attention score α ij Modeled as a Gaussian distribution: By the mean μ ij and variance The central value of the prediction and its degree of uncertainty can be clearly expressed, and the mean μ ij Represents the center position of the predicted value, i.e., the most likely attention score; Indicates the uncertainty range of the predicted value. The larger the variance, the higher the uncertainty. When training the model, a reparameterization strategy is adopted by introducing an auxiliary noise variable ε, which follows a standard normal distribution. This noise variable is then mapped to the target distribution through a differentiable transformation, namely a ij =μ ij +s ij e Then, the random process that is difficult to optimize directly is converted into a deterministic function about the parameters, so that the model containing random variables can be trained using the standard gradient descent method; When calculating the attention weight, both the result of the dot product and the uncertainty are taken into account, and the attention score with a Gaussian distribution is used for weighted summation: Among them, Q represents the query vector, which is matched with other vectors to calculate the similarity with other vectors; K is the key vector that represents the characteristics of each element in the input sequence; V is the value vector that corresponds to the key vector in the input sequence. In the specific calculation formula, α ij represents the attention score of the i-th query vector and the j-th key vector, divided by the i-th query vector and all key vectors It means that the similarity scores of all key vectors are normalized, and the similarity scores are converted into probability distributions. The normalized weights are the same as the corresponding v j The vectors are multiplied to get the weighted output; finally, all weighted Value vectors are summed to get the final output of the probabilistic cross-attention mechanism.

5. The high-performance and low-cost multimodal data fusion model for complete driving environment state perception according to claim 4 is characterized in that: The mean μ of the Gaussian distribution ij and variance The estimation is implemented using a multi-layer perceptron (MLP) to query the vector q i and key vector k j As input, generate a Gaussian distribution with mean μ ij and variance Among them, i represents the index of the query, j represents the index of the key and value, and k represents the index of all keys, which is used for normalization calculation: μ ij =MLP μ (q i ,k j ) 6. The high-performance and low-cost multimodal data fusion model for complete driving environment state perception according to claim 5 is characterized in that: The decoder of the uncertainty-aware feature fusion model constructed by the probabilistic cross-attention mechanism includes three layers, each of which contains a self-attention mechanism and a cross-modal cross-attention. Following the Transformer architecture, the query vector undergoes a self-attention operation after initialization. Subsequently, cross-attention is performed with the lidar point cloud and image features to fuse multimodal features. After being processed by the feedforward network, the encoding layer outputs the updated query as input to the next layer. After three layers of fusion encoding, the final updated query vector is obtained. The loss function of this part adopts the negative log-likelihood loss function:

7. The high-performance and low-cost multimodal data fusion model for complete driving environment state perception according to claim 1 is characterized in that: The S3 specifically includes: For each input N view images, we first use a pre-trained 2D object detector (FCOS2D) and a monocular 3D object detector (FCOS 3D) to obtain 2D and 3D detection results respectively to generate the final dynamic object query. The 2D object detector will output a set of 2D object detection boxes. Represents the set of 2D bounding boxes detected in the i-th view image, and the total number of bounding boxes is N, M m Represents the number of bounding boxes detected in the i-th view image; Combined with the 3D properties predicted by the monocular 3D object detector, the 2D bounding box is mapped to the 3D space. This process involves the conversion of camera parameters. Specifically, the 3D position is estimated by coordinate transformation: C 3d =E -1 K -1 [out,vd,d,1] Among them, C 3d Indicates the center position of the 3D space, E -1 and K -1 They represent the camera extrinsic matrix and the camera intrinsic matrix respectively, (u, v) represents the center position of the 2D detection frame, and d here represents the depth value; Then, non-maximum suppression (NMS) is performed in 3D space to filter the intersecting boxes and select the top N boxes ranked by confidence score. 2d The dynamic target query DQs is generated through the cross-attention module. The formula is as follows: in, Indicates that the 3D center C is converted using the corresponding camera parameters 3d Projected onto the i-th image, represents the m feature maps of the i-th view, M represents the number of feature maps in different images i; Bilinear represents the bilinear sampling function; In case some objects may be ignored, randomly initialized queries RQs are retained, and the final query set Q can be expressed as: Q=Prop(DQs,RQs) The optimization goal of the above task is: in, Initialize the model's loss function for queries that reduce uncertainty, is the loss function for 2D detection, is the loss function for 3D detection, λ 3d is the hyperparameter of the model.

8. The high-performance and low-cost multimodal data fusion model for complete driving environment state perception according to claim 1 is characterized in that: The S4 specifically includes: By generating noisy queries with different uncertainty levels, noise is introduced during the training process to enable the model to better understand and adapt to uncertainty. Random noise is added to the real label objects, and the noisy queries are divided into positive samples, negative samples and ignored samples according to the 3D intersection over union (IoU). The positive samples are high-quality data with low uncertainty, which are helpful for model learning and prediction; the negative samples are low-quality data with high uncertainty, which help the model filter out unreliable information; the ignored samples are used for the model to filter interference; During the model optimization process, the loss values ​​of positive and negative samples are calculated to determine the loss function of the robust optimization training strategy based on query denoising. in, and They represent the focal loss function for bounding box classification of positive samples and the L1 loss function for bounding box regression; the loss function for negative samples The focal loss function is used to distinguish the background.

9. The high-performance and low-cost multimodal data fusion model for complete driving environment state perception according to claim 1 is characterized in that: The S5 specifically includes: Two feed-forward networks are used to predict the 3D bounding box and category information of the detected object using the updated query. The prediction process is described as follows: Among them, f reg and f cla Represent the feedforward networks for regression and classification tasks respectively, and Query is the updated object query; then, bipartite matching is used for one-to-one matching between predicted values ​​and true values, focal loss is used for classification, and L1 loss is used for 3D bounding box regression: in, represents the ensemble prediction loss function, is the classification loss function, which calculates the predicted classification information The loss between the real classification information c, Represents the regression loss function, calculating the predicted bounding box information The loss between the real bounding box information b, λ cls and λ reg They represent the weight hyperparameters of the classification loss term and the regression loss term, respectively.

10. The vehicle-mounted computing device based on the high-performance and low-cost multimodal data fusion model for complete driving environment state perception according to claim 1 is characterized in that: The multimodal data fusion model based on TensorRT accelerated reasoning is deployed in the vehicle-mounted computing device.

Images