High-robustness intelligent automobile aerial view multi-source space-time fusion environment sensing system

Abstract

The invention relates to an intelligent automobile aerial view angle environment sensing system, in particular to a high-robustness intelligent automobile aerial view angle multi-source space-time fusion environment sensing system. Comprising an image feature extraction module, a point cloud feature extraction module, a cross-modal interaction fusion module, a diffusion generation module, a time cross attention fusion module and an environment perception decoding module. Complementary enhancement of camera images and laser radar point cloud BEV features is realized through a bidirectional cross attention mechanism, a semantic-geometric bidirectional correlation weight matrix is constructed, and dynamic distribution of cross-modal feature weights is realized; gaussian noise is injected into the fused BEV feature space through a diffusion generation technology to simulate multi-source noise interference in a real environment, and a multi-scale denoising network guide system based on a U-Net architecture is adopted to separate real environment semantics from noisy features; and space-time alignment between the historical BEV features and the current BEV features is realized by constructing a multi-head time cross attention mechanism.

Description

Technical Field

[0001] This invention relates to an intelligent vehicle bird's-eye view environmental perception system, and more particularly to a highly robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environmental perception system. Background Technology

[0002] In recent years, intelligent vehicle environmental perception technology has developed rapidly. By collecting and processing data from environmental sensors, environmental perception systems can achieve comprehensive acquisition and accurate understanding of surrounding environmental information, providing crucial support for downstream systems such as decision-making, control, and execution in intelligent driving. Bird's-eye view (BEV), as a unified form of spatial information representation, has advantages such as simple spatial alignment, clear scale representation, ease of multi-source fusion, and strong anti-occlusion capabilities, and has gradually become the basic design paradigm for current intelligent vehicle environmental perception systems. However, at present, intelligent vehicle BEV environmental perception systems still face the following key challenges that urgently need to be addressed:

[0003] 1. Due to inherent perspective bias, measurement accuracy limitations, and environmental adaptability constraints, single sensors struggle to comprehensively capture multi-dimensional information in complex scenarios. Therefore, current BEV perception systems generally employ multi-sensor fusion. However, the significant heterogeneity of multi-source perception information from different sensors in terms of spatial location, feature scale, and semantic expression leads to problems such as inaccurate feature alignment, semantic conflicts, and lack of consistency in fused features when fusion from a unified perspective, resulting in information loss and feature redundancy. Therefore, effectively eliminating the heterogeneity contradictions among multi-source perception information in BEV perception systems, achieving consistent cross-modal feature expression during fusion, and simultaneously preserving high-confidence features from each sensor has become a core challenge for improving BEV perception performance.

[0004] 2. Environmental noise and interference pose a significant threat to sensor data quality. In dynamic scenarios, inherent sensor noise, environmental coupling interference, and target motion distortion can all contaminate the raw data, leading to systematic biases in the perception results. This nonlinear degradation of data quality makes it difficult for BEV perception systems to maintain stable perception confidence in dynamic and changing traffic environments. Therefore, constructing a feature cleaning mechanism to effectively separate sensor noise has become an important technical approach to ensure the robustness of BEV perception systems in dynamic scenarios.

[0005] 3. Environmental sensors, limited by their field of view and blind spots, are prone to interruptions in the temporal continuity of target states and the loss of environmental spatial structure information due to occlusion in scenarios with dense traffic flow, complex road topology, and extreme weather conditions. This data incompleteness not only leads to misjudgments of target movement intentions by the BEV perception system but also causes gaps in environmental understanding, severely impacting the spatiotemporal continuity of BEV environmental perception. Therefore, effectively reconstructing the environmental spatial structure and target states within occluded areas, and thus maintaining the spatiotemporal continuity of perception results under conditions of information loss, has become another technical challenge for improving the performance of BEV perception systems. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a highly robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environment perception system, which mainly includes: an image feature extraction module, a point cloud feature extraction module, a cross-modal interaction fusion module, a diffusion generation module, a temporal cross-attention fusion module, and an environment perception decoding module.

[0007] The image feature extraction module takes camera images as input, extracts multi-scale hierarchical features through a pre-trained Swin-Transformer backbone network, and achieves multi-level fusion of shallow high-resolution details and deep semantic features based on the Feature Pyramid Network (FPN). A BEV spatial projection model is constructed through a view transformation network, and implicit depth distribution estimation is used to unfold perspective features along the pixel ray direction. The features are then compressed and aggregated in the BEV spatial height direction to generate image BEV features with rich semantic information.

[0008] The point cloud feature extraction module takes LiDAR point cloud as input, converts unstructured point cloud into regular sparse voxels through point cloud voxelization network, extracts multi-scale geometric features by combining sparse convolution feature extraction network, and compresses three-dimensional voxel features to two-dimensional BEV plane through columnar sparse convolution to generate point cloud BEV features with rich geometric information.

[0009] The cross-modal interaction and fusion module takes the outputs of the image feature extraction module and the point cloud feature extraction module as inputs, constructs a cross-modal feature interaction channel through a bidirectional cross-attention mechanism, calculates a semantic-geometric association weight matrix with image BEV features as queries and point cloud BEV features as keys, and simultaneously establishes a geometric-semantic reverse association matrix with point cloud BEV features as queries and image BEV features as keys. Through dynamic weight allocation, bidirectional complementarity of cross-modal features is achieved, global semantic-geometric association is constructed, and fusion is performed in the deep feature space to output fused BEV features.

[0010] The diffusion generation module takes the output of the cross-modal interaction and fusion module as input, injects Gaussian noise into the fused BEV features to simulate sensor noise and environmental interference in real-world scenarios, and generates noisy BEV features. Based on the denoising network, a conditional guidance mechanism is constructed, using the fused BEV features as constraints. Through multi-step iteration, noise interference is gradually eliminated, restoring the original feature distribution that was masked by noise while maintaining the consistency of the feature geometry, and finally outputting the denoised BEV features.

[0011] The temporal cross-attention fusion module takes the output of the diffusion generation module and the output of the historical temporal cross-attention fusion module as inputs to construct a temporal feature fusion mechanism. It uses the vehicle pose transformation matrix to realize the temporal alignment and motion compensation of historical features. It establishes the spatiotemporal correlation between the current frame BEV features and historical temporal features through a multi-head temporal cross-attention mechanism. It dynamically reconstructs the missing features in the region caused by the temporary occlusion of the target or the interruption of the sensor signal. It uses gated residual connection to fuse historical features and current features and outputs BEV features with spatiotemporal continuity.

[0012] The environmental perception decoding module takes the output of the temporal cross-attention fusion module as input, decodes the BEV features through the multi-task perception head, and outputs the semantic segmentation results, target detection results, and lane line recognition results of the scene, thereby realizing comprehensive perception and scene understanding of the vehicle's surrounding environment.

[0013] Furthermore, the image feature extraction module consists of three parts: an image feature extraction network, an image feature enhancement network, and a view transformation network.

[0014] The input to the image feature extraction network is multi-view RGB image data from the camera. Where N represents the total number of camera viewpoints. Indicates the first RGB image data from multiple perspectives and These represent the height and width of the image, respectively. This represents a three-dimensional tensor defined in real space for the image; to facilitate subsequent feature modeling and introduce local structure priors, each image is divided into several non-overlapping image blocks. Each image patch is flattened and mapped from the original pixel space to a fixed-dimensional feature space through a linear projection layer:

[0015]

[0016] in, For feature flattening operation, For linear projection operations, Indicates the first Feature vectors of image patches;

[0017] The Swin-Transformer backbone network consists of several stages, each stage consisting of multiple Transformer modules stacked sequentially. In the same stage, adjacent Transformer modules alternately use Window Multi-Head Self Attention (W-MSA) and Shifted Window Multi-Head Self Attention (SW-MSA).

[0018] The Transformer module employing W-MSA performs self-attention computation only within a local window of the image, capturing fine features within that local region. The Transformer module employing SW-MSA achieves cross-window information interaction through periodic movement of the window position. By alternately stacking W-MSA and SW-MSA modules within the layers, the network can gradually expand the feature perception range while maintaining local modeling efficiency, enabling feature representation to transition from local structure perception to broader contextual modeling. The calculation process for each Transformer module is as follows:

[0019]

[0020]

[0021] in, and These represent the input and output characteristics of the module, respectively. This represents a multi-head self-attention module, when it is... Even number For W-MSA, when When it is an odd number For SW-MSA, It is a multilayer perceptron. Presentation layer normalization operation;

[0022] The core of the Transformer module is the multi-head self-attention mechanism, where self-attention is calculated on a local window basis, i.e., W-MSA; for features within any window... The formula for calculating single-head attention is:

[0023]

[0024] in, , and The weight matrix is ​​a learnable matrix. This represents the query, key, and value generated from the input features. For window size, For single-head feature dimensions, For learnable relative position offsets, This is a normalization operation; multi-head self-attention is obtained by computing multiple attention heads in parallel, concatenating them, and then mapping them.

[0025]

[0026] in, This is the overall weight matrix;

[0027] A cyclic window offset strategy is introduced in adjacent Transformer modules to achieve cross-window information interaction; if the first If the first Transformer module uses W-MSA, then the second... Each Transformer module performs a cyclic offset operation along the spatial dimension on the input features, with the offset amount set to half the window size, so that the new window can cover the boundary region of the original window; then, multi-head self-attention calculation is performed on the window partition after offset, which can be represented as:

[0028]

[0029] in, and They represent the first Input and output characteristics of each Transformer module For cyclic window offset operators, and All are intermediate features; by alternately stacking Transformer modules using W-MSA and SW-MSA within the same layer, the network can gradually achieve cross-window feature interaction while maintaining local modeling efficiency, thereby effectively enhancing the contextual modeling capability of feature representation;

[0030] After stacking Transformer modules within a level, features in adjacent spatial domains are aggregated through a block merging operation to obtain the feature representation for the next level; for the th The position in the output features of each level is Features of the place Input features The features within the corresponding neighborhood are concatenated, and channel compression and feature fusion are achieved through convolutional mapping. The calculation process is as follows:

[0031]

[0032] in, , , and The neighborhood features of the input features For feature splicing operations, This involves a convolution operation; it reduces the feature space resolution while increasing the channel dimension, thereby progressively enhancing the semantic expressive power of the features; by repeating the above operation at different levels, the final output is multi-scale image features. Deep features encode global semantics, while shallow features preserve high-resolution details.

[0033] The image feature enhancement network takes the multi-layer output of the image feature extraction network as input, adopts an FPN structure, and fuses features from different scales; the four layers of multi-scale input features are unified in the number of channels of each layer through lateral convolution.

[0034]

[0035] in, For convolution operations on features at different scales, the number of channels in each layer is adjusted uniformly to output features. ;

[0036] From the deepest layer Initially, learnable deconvolution upsampling is used to gradually fuse shallow features:

[0037]

[0038] In the formula, For the first Fusion features at various scales; This is a deconvolution upsampling operation;

[0039] To preserve the characteristics of each level, a hierarchical attention weighting mechanism is introduced for the fused multi-layer features; for the first... Layer fusion features Through learnable attention weights The weighted adjustment is performed on it, and its calculation form is as follows:

[0040]

[0041] in, The fused features after applying hierarchical attention weights, For the range of values ​​within The learnable space weights within, This is an average pooling operation;

[0042] Finally, cross-level information aggregation is performed on the fused features:

[0043]

[0044] By using the above method, while maintaining the complementarity of multi-level features, adaptive enhancement of key information is achieved, and enhanced image features are output.

[0045] The view transformation network takes the output features of the image feature enhancement network as input and generates a pixel-wise depth probability distribution through the depth prediction head. :

[0046]

[0047] in, Represents the discrete depth probability distribution. and These represent the length and width of the image features, respectively. The number of discrete depth intervals;

[0048] Expanding the image features by the outer product of the depth probability generates 3D pseudo-point cloud features. :

[0049]

[0050] In the formula, Represents pixels Two-dimensional image features at the location, For depth interval indexing, 2D image features are uplifted to 3D space along the ray direction; using the camera intrinsic and extrinsic parameter matrices, pseudo-point clouds are mapped to BEV space, and voxel pooling is used to highly compress the mapped discrete point cloud features:

[0051] ,

[0052] In the formula, Indicates the spatial location of BEV At height voxel characteristics on For highly discrete layers, BEV spatial location BEV features at the location; spatial alignment and stitching of BEV features from multiple surround-view cameras to output the final image BEV features. ;

[0053] Furthermore, the point cloud feature extraction module consists of two parts: a point cloud voxelization network and a sparse convolutional feature extraction network.

[0054] The point cloud voxelization network takes the reflected signal from the lidar as input and maps the point cloud onto a voxel grid within a certain range around the vehicle based on its spatial location. Each voxel represents a small area in the space around the vehicle. This spatial partitioning distributes the original point cloud data into each voxel. Subsequently, the point features within each voxel are accumulated and averaged to obtain the voxel features.

[0055]

[0056] in, For the first Individual characteristics, This represents the number of points in the voxel. Indicates the first Features of each point;

[0057] The sparse convolutional feature extraction network takes voxelized features as input and extracts features from non-empty voxel regions through sparse convolutional layers; the update process of non-empty voxel features by a single sparse convolutional layer is defined as follows:

[0058]

[0059] in, This indicates a sparse convolution operator that is executed only within non-empty voxels and their neighborhoods. and These are the convolution parameters for the corresponding layer. For normalization operations, It is a non-linear activation function. and The first Input and output features of sparse convolution layers;

[0060] Based on this, the sparse convolutional feature extraction network is composed of several stacked sparse convolutional modules. Each module introduces residual connections and dilated convolutions on top of the above sparse convolutions, and its computational form is as follows:

[0061]

[0062] in, For expansion rate The sparse convolution with holes, the dilation rate As the network hierarchy increases;

[0063] Multi-scale fusion of sparse features from different output levels is performed. Feature weighting is achieved through upsampling and learningable channel attention weights to obtain fused sparse voxel features:

[0064]

[0065] in, For learnable channel attention weights, This indicates an upsampling operation; it combines the sparse features of the fused sample. Mapped to the BEV space, the dense BEV features are finally obtained:

[0066]

[0067] in, For point cloud BEV features, This is a feature mapping operation.

[0068] Furthermore, the cross-modal interaction and fusion module takes the image BEV features output by the image feature extraction module and the point cloud BEV features output by the point cloud feature extraction module as input, and performs global mean pooling and max pooling on the dual-modal features respectively to generate a feature prototype matrix:

[0069]

[0070] in, Represents image modality or point cloud modality. This represents the overall distribution feature matrix obtained by averaging statistical analysis of all spatial location features. This represents the salient region feature matrix obtained by taking the maximum response of all spatial location features. and For the length and width of the BEV space, and BEV spatial coordinates;

[0071] By adaptively weighting and fusing the average and salient features using learnable weight coefficients, the final global features for each modality are obtained.

[0072]

[0073] in, and These are learnable fusion weight coefficients used to adjust the proportions of overall distribution information and salient response information in the final global features;

[0074] To achieve fine-grained and effective interaction between image and point cloud features, in the attention path from image to point cloud, image BEV features are used as the query vector, and point cloud BEV features are used as the key and value. A spatial attention weight matrix is ​​calculated by scaling the dot product to dynamically capture the correlation between image semantic information and point cloud spatial geometric information.

[0075]

[0076] in, This is the attention weight matrix between the image and the point cloud. and The projection matrix is ​​learnable. For subspace dimension, For global features of the image;

[0077] By using the obtained attention weight matrix, point cloud geometric features relevant to image semantics are adaptively selected, and enhanced image feature representations are obtained through residual connections and layer normalization operations. :

[0078]

[0079] in, The projection matrix is ​​learnable. Represents any position in BEV space;

[0080] The point cloud to image attention path is similar to the image to point cloud path, using point cloud BEV features as queries and image BEV features as keys and values. It employs the same scaled dot product attention mechanism to dynamically mine fine-grained semantic associations between point cloud features and image features, and similarly obtains enhanced point cloud features through residual connections and layer normalization operations. To ensure the effectiveness of bidirectional interaction features, the image BEV features enhanced with bidirectional spatial attention are concatenated with the point cloud BEV features along the channel dimension. A convolutional network is then used to achieve consistent spatial fusion, resulting in unified cross-modal fused BEV features. :

[0081] ;

[0082] Through a two-way information flow mechanism, image BEV features inherit the precise spatial structure information provided by point cloud features, while point cloud BEV features integrate the rich semantic details carried by image features.

[0083] Furthermore, the diffusion generation module includes two stages: training and sampling. The training stage consists of three parts: a noise-adding network, a temporal embedding network, and a denoising network. The sampling stage is during the inference process, where the system takes the output of the cross-modal interaction and fusion module as input and generates denoised features through multiple rounds of diffusion.

[0084] The noise-adding network takes the output features of the cross-modal interaction and fusion module as input and first generates a random noise with the same shape as the input features. To simulate the uncertainties in real-world environments;

[0085] During the noise addition process, from a uniform distribution Obtain sampling time step To control the intensity of noise input; based on sampling time steps Calculate the noise signal-to-noise ratio parameter To determine the intensity of the noise:

[0086]

[0087] in, and These are the translation factor and scaling factor, used to adjust the nonlinear variation characteristics of the signal-to-noise ratio during diffusion, based on the signal-to-noise ratio parameter. The signal preservation coefficient, which allows for flexible control of noise injection intensity over time, can be further calculated. With noise figure :

[0088]

[0089] in, for Activation function, ensuring the result is in between;

[0090] By signal preservation coefficient With noise figure Obtain the fusion features after adding noise :

[0091]

[0092] The BEV features output by the cross-modal interaction and fusion module are conditionally concatenated with the fused features after adding noise, and the two are then used as input to the denoising network.

[0093]

[0094] in, The BEV characteristics after fusion conditions;

[0095] The time embedding network adds noise to the signal-to-noise ratio parameter of the network. As input, it is transformed into a high-dimensional feature representation and expanded to a shape consistent with the system input dimension; through a set of learnable frequency vectors The process involves calculating the frequency features at each time point to form a time embedding vector. :

[0096]

[0097] The obtained temporal embedding vector is fed into a multilayer perceptron for nonlinear transformation to obtain the temporal conditional embedding vector. :

[0098] ;

[0099] The denoising network takes the outputs of the temporal embedding network and the noise addition network as inputs, and is designed based on the U-Net network. The denoising network mainly consists of two parts: an encoder and a decoder. The encoder is responsible for extracting multi-scale features from the input data; this part consists of several convolutional modules, each extracting features at different scales. In each layer, the input features... Spatial features are extracted through convolution operations, and combined with temporal embedding vectors. This is then summed element-wise with the output features; in this way, temporal information is injected into each feature layer.

[0100]

[0101] in, These are the feature outputs generated by the encoder. Each feature output is stored as an intermediate feature for use by the decoder.

[0102] The decoder gradually recovers the original signal and removes noise by combining the features extracted by the encoder with temporal embedding information. Starting from the output of the last encoder layer, it upsamples layer by layer to recover the input features. During the decoding process, the features of each layer not only depend on the output of the previous decoder layer but also fuse the features of the corresponding layer in the encoder. High-resolution features are preserved through skip connections, improving the denoising effect.

[0103]

[0104] in, For upsampling operation, and For the input and output of the decoder, It is the feature output generated by the encoder;

[0105] After several rounds of upsampling operations, the denoised BEV features are obtained. Compared with the original features, the features are cleaned up by adding and removing noise, thereby improving the robustness of the system.

[0106] The sampling phase does not involve adding noise; the input is pure Gaussian noise with the same size as the fused BEV feature. The input features and noise are concatenated and fused using convolution to generate preliminary noise information. A denoising network gradually reduces the noise, utilizing temporal embedding information generated by a temporal embedding network to control the noise removal process. The final output is a denoised feature representation.

[0107]

[0108] in, For noise reduction networks.

[0109] Furthermore, the temporal cross-attention fusion module takes the output of the diffusion generation module and the retained historical BEV features as input, and first performs temporal alignment and motion compensation on the historical features; let... Indicates the interval with the current frame is Historical BEV characteristics, obtained through the vehicle's pose change matrix during that time interval. A spatial reprojection operation is performed on historical BEV features to map them to the current BEV space, and an effective region mask is generated by combining the point cloud coverage. The alignment process is represented as follows:

[0110]

[0111] in, This represents a spatial reprojection operation based on pose transformation. For effective region mask, For aligned historical BEV features;

[0112] By introducing temporal information to construct effective spatiotemporal dependencies, the BEV features of the current frame are analyzed. BEV features of historical frames Construct the query, key, and value for time-crossing attention separately:

[0113]

[0114] in, For time interval The corresponding temporal embedding features are used to distinguish the degree of influence of different historical frames in the temporal dimension; , and It is a learnable linear projection matrix that maps the features of the current frame and historical frames into a unified attention subspace;

[0115] The spatiotemporal correlation weights between the current feature and each historical frame are calculated using a multi-head attention mechanism; for the BEV feature at the current moment, in each attention head... Below, based on the query vector of the current frame Aligned history frame key vector The scaled dot product between them is used to calculate the attention weights at different historical time steps, which are expressed in the following form:

[0116]

[0117] in, For the number of attention heads, The total dimension of the features. For attention head Attention weights;

[0118] Subsequently, the attention points are spliced ​​together and weighted by different historical time steps. By aggregating historical frame features across time, a unified historical context feature is obtained:

[0119]

[0120] in, This is the value vector of historical frames. This represents the total number of historical frames.

[0121] Finally, the historical features and current features are fused into the final BEV feature representation through a gated residual connection mechanism. :

[0122]

[0123] in, For the gated weight matrix, for Activation functions are used to control the degree to which historical information influences the current feature update;

[0124] Furthermore, the environmental perception decoding module takes the output of the temporal cross-attention fusion module as input to construct specific perception branches for different tasks, including scene semantic segmentation branch, lane line detection branch, and target detection branch, so as to achieve a comprehensive understanding of the environmental information around the vehicle.

[0125] The scene semantic segmentation branch uses BEV features Using the convolution class header as input, the semantic category probability distribution corresponding to different locations in the BEV space is calculated to achieve pixel-level semantic understanding of different regions within the scene:

[0126]

[0127] in, This is the header for the semantic segmentation task. The probability value for the semantic category in the BEV space;

[0128] The lane detection branch uses a specific convolutional network and nonlinear activation function to predict the road structure and topological connectivity within the scene, in order to construct topological information on road connectivity and traffic network structure.

[0129]

[0130] in, For lane line detection head, for Activation function This is the result of lane segmentation;

[0131] The target detection branch uses continuous convolution and a feature regression head to regress the motion state of targets in the scene, outputting target state information such as speed, direction, and size, in order to achieve accurate prediction of the target state.

[0132]

[0133] in, For target detection head, For target detection results;

[0134] Each task branch is connected to an independent task head, and is trained and optimized according to the specific needs of the environmental perception task. The parameters of each task head are independent of each other to achieve collaborative modeling of multi-task environmental perception.

[0135] As a preferred approach, loss functions are defined for different tasks during training to guide the system to converge toward a stable and accurate environmental perception target: loss functions include semantic segmentation loss, lane detection loss, and object detection loss;

[0136] The semantic segmentation loss The difference between the scene semantic category predicted by the system and the true semantic label is measured using a pixel-level cross-entropy loss function, calculated as follows:

[0137]

[0138] in, BEV characteristics The number of pixels in For the number of semantic categories, Indicates the first The pixel in the actual annotation belongs to the _th _ ... Class indicator value, This represents the system's predicted probability of the corresponding category.

[0139] Lane detection loss The binary cross-entropy loss function is used to measure the difference between the system's predicted lane position and the actual lane position.

[0140]

[0141] in, This represents the number of prediction units in the lane detection task. For the first Real lane line labels at each location This corresponds to the predicted probability value;

[0142] The target detection loss is used to evaluate the accuracy of the system's predictions of the location, category, and size of traffic participants, and is derived from the heatmap loss. Classification loss and regression loss Common components:

[0143]

[0144]

[0145]

[0146]

[0147] in, The total loss for target detection, , These are the true and predicted values ​​of the heatmap, respectively. , These are the true label and predicted probability of the target category, respectively. , These are the ground truth bounding box and the predicted bounding box, respectively. The target number in the BEV space. , , These are the weighting coefficients for each loss term, used to balance the impact of different detection subtasks on the overall training process.

[0148] The beneficial effects of this invention are:

[0149] This invention discloses a highly robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environmental perception system. It designs a cross-modal spatial interactive fusion module, achieving complementary enhancement of BEV features from camera images and LiDAR point clouds through a bidirectional cross-attention mechanism. Simultaneously, it constructs a semantic-geometric bidirectional correlation weight matrix to dynamically allocate cross-modal feature weights, and further fuses the BEV features of images and point clouds using a convolutional network. By aligning cross-modal features to eliminate projection biases from different sensors in the BEV space, it resolves the heterogeneity contradictions of multi-source perception information, enhances the consistent expression of cross-modal features within the BEV space, and improves the multi-source perception information fusion capability of the BEV environmental perception system.

[0150] By using diffusion generation technology, Gaussian noise is injected into the fused BEV feature space to simulate multi-source noise interference in the real environment (such as rain and fog scattering noise, sensor thermal noise, and motion blur artifacts). A multi-scale denoising network based on the U-Net architecture is used to guide the system to separate the real environmental semantics from the noisy features, and gradually reconstruct the noise-contaminated BEV features into a pure feature representation. This can significantly suppress the feature confusion caused by environmental interference and sensor inherent noise, thereby improving the robustness of the intelligent vehicle BEV environmental perception system.

[0151] By constructing a multi-head temporal cross-attention mechanism, spatiotemporal alignment between historical BEV features and current BEV features is achieved, establishing a long-distance dependency between the current frame and historical time series, and extracting spatiotemporal context features from historical BEV feature sequences. This effectively reconstructs the features of occluded areas in scenarios where sensor data is missing or targets are continuously occluded, maintaining the spatiotemporal continuity of perception results and providing technical support for realizing continuous BEV environmental perception for intelligent vehicles. Attached Figure Description

[0152] Figure 1 This is a schematic diagram of the overall architecture of the present invention.

[0153] Figure 2 This is a schematic diagram of the image feature extraction module of the present invention.

[0154] Figure 3 This is a schematic diagram of the point cloud feature extraction module of the present invention.

[0155] Figure 4 This is a schematic diagram of the cross-modal interaction and fusion module of the present invention.

[0156] Figure 5 This is a schematic diagram of the diffusion generation module of the present invention.

[0157] Figure 6 This is a schematic diagram of the time-crossing attention fusion module of the present invention. Detailed Implementation

[0158] See Figure 1 As shown, the present invention provides a highly robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environment perception system, which mainly includes: an image feature extraction module, a point cloud feature extraction module, a cross-modal interaction fusion module, a diffusion generation module, a temporal cross-attention fusion module, and an environment perception decoding module.

[0159] (1) The image feature extraction module takes the camera image as input, extracts multi-scale hierarchical features through the pre-trained Swin-Transformer backbone network, and realizes multi-level fusion of shallow high-resolution details and deep semantic features based on the Feature Pyramid Network (FPN); constructs a BEV spatial projection model through the view transformation network, uses implicit depth distribution estimation to expand the perspective features along the pixel ray direction, and compresses and aggregates them in the BEV spatial height direction to generate image BEV features with rich semantic information;

[0160] (2) The point cloud feature extraction module takes the lidar point cloud as input, converts the unstructured point cloud into regular sparse voxels through the point cloud voxelization network, extracts multi-scale geometric features by combining the sparse convolution feature extraction network, and compresses the three-dimensional voxel features to the two-dimensional BEV plane through columnar sparse convolution to generate point cloud BEV features with rich geometric information.

[0161] (3) The cross-modal interaction and fusion module takes the outputs of the image feature extraction module and the point cloud feature extraction module as inputs, constructs a cross-modal feature interaction channel through a bidirectional cross-attention mechanism, calculates the semantic-geometric association weight matrix with image BEV features as queries and point cloud BEV features as keys; at the same time, it establishes a geometric-semantic reverse association matrix with point cloud BEV features as queries and image BEV features as keys, realizes bidirectional complementarity of cross-modal features through dynamic weight allocation, constructs global semantic-geometric association, and fuses in the deep feature space to output fused BEV features;

[0162] (4) The diffusion generation module takes the output of the cross-modal interaction and fusion module as input, injects Gaussian noise into the fused BEV features to simulate sensor noise and environmental interference in real scenarios, and generates noisy BEV features; a condition guidance mechanism is constructed based on the denoising network, and the fused BEV features are used as constraints to gradually eliminate noise interference through multi-step iterations. While maintaining the consistency of feature geometry, the original feature distribution covered by noise is restored, and finally the denoised BEV features are output.

[0163] (5) The temporal cross-attention fusion module takes the output of the diffusion generation module and the output of the historical temporal cross-attention fusion module as input to construct a temporal feature fusion mechanism. It uses the vehicle pose transformation matrix to realize the temporal alignment and motion compensation of historical features. It establishes the spatiotemporal correlation between the current frame BEV features and historical temporal features through the multi-head temporal cross-attention mechanism. It dynamically reconstructs the missing features in the region caused by the temporary occlusion of the target or the interruption of the sensor signal. It uses gated residual connection to fuse historical features and current features and outputs BEV features with spatiotemporal continuity.

[0164] (6) The environmental perception decoding module takes the output of the time-cross attention fusion module as input, decodes the BEV features through the multi-task perception head, and outputs environmental perception information such as the semantic segmentation result, target detection result and lane line recognition result of the scene, so as to realize the comprehensive perception and scene understanding of the vehicle's surrounding environment.

[0165] Further, see Figure 1 and Figure 2 The image feature extraction module consists of three parts: an image feature extraction network, an image feature enhancement network, and a view transformation network.

[0166] (1.1) The input to the image feature extraction network is multi-view RGB image data from the camera. Where N represents the total number of camera viewpoints. Indicates the first RGB image data from multiple perspectives and These represent the height and width of the image, respectively. This represents a three-dimensional tensor defined in real space for the image; to facilitate subsequent feature modeling and introduce local structure priors, each image is divided into several non-overlapping image blocks. Each image patch is flattened and mapped from the original pixel space to a fixed-dimensional feature space through a linear projection layer:

[0167] (1)

[0168] in, For feature flattening operation, For linear projection operations, Indicates the first Feature vectors of image patches;

[0169] The Swin-Transformer backbone network consists of several stages, each stage consisting of multiple Transformer modules stacked sequentially. In the same stage, adjacent Transformer modules alternately use Window Multi-Head Self Attention (W-MSA) and Shifted Window Multi-Head Self Attention (SW-MSA).

[0170] The Transformer module employing W-MSA performs self-attention computation only within a local window of the image, capturing fine features within that local region. The Transformer module employing SW-MSA achieves cross-window information interaction through periodic movement of the window position. By alternately stacking W-MSA and SW-MSA modules within the layers, the network can gradually expand the feature perception range while maintaining local modeling efficiency, enabling feature representation to transition from local structure perception to broader contextual modeling. The calculation process for each Transformer module is as follows:

[0171] (2) (3)

[0172] in, and These represent the input and output characteristics of the module, respectively. This represents a multi-head self-attention module, when it is... Even number For W-MSA, when When it is an odd number For SW-MSA, It is a multilayer perceptron. Presentation layer normalization operation;

[0173] The core of the Transformer module is the multi-head self-attention mechanism, where self-attention is calculated on a local window basis, i.e., W-MSA; for features within any window... The formula for calculating single-head attention is:

[0174] (4)

[0175] in, , and The weight matrix is ​​a learnable matrix. This represents the query, key, and value generated from the input features. For window size, For single-head feature dimensions, For learnable relative position offsets, This is a normalization operation; multi-head self-attention is obtained by computing multiple attention heads in parallel, concatenating them, and then mapping them.

[0176] (5)

[0177] in, This is the overall weight matrix;

[0178] To overcome the limitations of fixed-window perception, a cyclic window offset strategy is introduced in adjacent Transformer modules to achieve cross-window information interaction; if the first... If the first Transformer module uses W-MSA, then the second... Each Transformer module performs a cyclic offset operation along the spatial dimension on the input features, with the offset amount set to half the window size, so that the new window can cover the boundary region of the original window; then, multi-head self-attention calculation is performed on the window partition after offset, which can be represented as:

[0179] (6)

[0180] in, and They represent the first Input and output characteristics of each Transformer module For cyclic window offset operators, and All are intermediate features; by alternately stacking Transformer modules using W-MSA and SW-MSA within the same layer, the network can gradually achieve cross-window feature interaction while maintaining local modeling efficiency, thereby effectively enhancing the contextual modeling capability of feature representation;

[0181] After stacking Transformer modules within a level, features in adjacent spatial domains are aggregated through a block merging operation to obtain the feature representation for the next level; for the th The position in the output features of each level is Features of the place Input features The features within the corresponding neighborhood are concatenated, and channel compression and feature fusion are achieved through convolutional mapping. The calculation process is as follows:

[0182] (7)

[0183] in, , , and The neighborhood features of the input features For feature splicing operations, This involves a convolution operation; it reduces the feature space resolution while increasing the channel dimension, thereby progressively enhancing the semantic expressive power of the features; by repeating the above operation at different levels, the final output is multi-scale image features. Deep features encode global semantics, while shallow features preserve high-resolution details.

[0184] (1.2) The image feature enhancement network takes the multi-layer output of the image feature extraction network as input, adopts the FPN structure, and fuses features from different scales; the input four-layer multi-scale features are unified in the number of channels of each layer through lateral convolution:

[0185] (8)

[0186] in, For convolution operations on features at different scales, the number of channels in each layer is adjusted uniformly to output features. ;

[0187] From the deepest layer Initially, learnable deconvolution upsampling is used to gradually fuse shallow features:

[0188] (9)

[0189] In the formula, For the first Fusion features at various scales; This is a deconvolution upsampling operation;

[0190] To preserve the characteristics of each level, a hierarchical attention weighting mechanism is introduced for the fused multi-layer features; for the first... Layer fusion features Through learnable attention weights The weighted adjustment is performed on it, and its calculation form is as follows:

[0191] (10)

[0192] in, The fused features after applying hierarchical attention weights, For the range of values ​​within The learnable space weights within, This is an average pooling operation;

[0193] Finally, cross-level information aggregation is performed on the fused features:

[0194] (11)

[0195] By using the above method, while maintaining the complementarity of multi-level features, adaptive enhancement of key information is achieved, and enhanced image features are output.

[0196] (1.3) The view transformation network takes the output features of the image feature enhancement network as input and generates a pixel-wise depth probability distribution through the depth prediction head. :

[0197] (12)

[0198] in, Represents the discrete depth probability distribution. and These represent the length and width of the image features, respectively. The number of discrete depth intervals;

[0199] Expanding the image features by the outer product of the depth probability generates 3D pseudo-point cloud features. :

[0200] (13)

[0201] In the formula, Represents pixels Two-dimensional image features at the location, For depth interval indexing, this operation elevates 2D image features to 3D space along the ray direction; using the camera intrinsic and extrinsic parameter matrices, the pseudo-point cloud is mapped to the BEV space, and voxel pooling is used to highly compress the mapped discrete point cloud features:

[0202] (14)

[0203] In the formula, Indicates the spatial location of BEV At height voxel characteristics on For highly discrete layers, BEV spatial location BEV features at the location; spatial alignment and stitching of BEV features from multiple surround-view cameras to output the final image BEV features. .

[0204] Further, see Figure 1 and Figure 3 The point cloud feature extraction module consists of two parts: a point cloud voxelization network and a sparse convolutional feature extraction network.

[0205] (2.1) The point cloud voxelization network takes the reflected signal from the lidar as input and maps the point cloud to a voxel grid within a certain range around the vehicle based on its spatial location. Each voxel represents a small area in the space around the vehicle. Through this spatial division, the original point cloud data is distributed to each voxel. Subsequently, the point features within each voxel are accumulated and averaged to obtain the voxel features:

[0206] (15)

[0207] in, For the first Individual characteristics, This represents the number of points in the voxel. Indicates the first Features of each point;

[0208] (2.2) The sparse convolutional feature extraction network takes the voxelized features as input and extracts features from non-empty voxel regions through sparse convolutional layers; the update process of non-empty voxel features by a single sparse convolutional layer is defined as follows:

[0209] (16)

[0210] in, This indicates a sparse convolution operator that is executed only within non-empty voxels and their neighborhoods. and These are the convolution parameters for the corresponding layer. For normalization operations, It is a non-linear activation function. and The first Input and output features of sparse convolution layers;

[0211] Based on this, the sparse convolutional feature extraction network is composed of several stacked sparse convolutional modules. Each module introduces residual connections and dilated convolutions on top of the above sparse convolutions, and its computational form is as follows:

[0212] (17)

[0213] in, For expansion rate The sparse convolution with holes, the dilation rate As the network hierarchy increases;

[0214] Multi-scale fusion of sparse features from different output levels is performed. Feature weighting is achieved through upsampling and learningable channel attention weights to obtain fused sparse voxel features:

[0215] (18)

[0216] in, For learnable channel attention weights, This indicates an upsampling operation; it combines the sparse features of the fused sample. Mapped to the BEV space, the dense BEV features are finally obtained:

[0217] (19)

[0218] in, For point cloud BEV features, This is a feature mapping operation.

[0219] Further, see Figure 1 and Figure 4 The cross-modal interaction and fusion module takes the image BEV features output by the image feature extraction module and the point cloud BEV features output by the point cloud feature extraction module as input, and performs global mean pooling and max pooling on the dual-modal features respectively to generate a feature prototype matrix:

[0220] (20)

[0221] in, Represents image modality or point cloud modality. This represents the overall distribution feature matrix obtained by averaging statistical analysis of all spatial location features. This represents the salient region feature matrix obtained by taking the maximum response of all spatial location features. and For the length and width of the BEV space, and BEV spatial coordinates;

[0222] By adaptively weighting and fusing the average and salient features using learnable weight coefficients, the final global features for each modality are obtained.

[0223] (twenty one)

[0224] in, and These are learnable fusion weight coefficients used to adjust the proportions of overall distribution information and salient response information in the final global features;

[0225] To achieve fine-grained and effective interaction between image and point cloud features, in the attention path from image to point cloud, image BEV features are used as the query vector, and point cloud BEV features are used as the key and value. A spatial attention weight matrix is ​​calculated by scaling the dot product to dynamically capture the correlation between image semantic information and point cloud spatial geometric information.

[0226] (twenty two)

[0227] in, This is the attention weight matrix between the image and the point cloud. and The projection matrix is ​​learnable. For subspace dimension, For global features of the image;

[0228] By using the obtained attention weight matrix, point cloud geometric features relevant to image semantics are adaptively selected, and enhanced image feature representations are obtained through residual connections and layer normalization operations. :

[0229] (twenty three)

[0230] in, The projection matrix is ​​learnable. Represents any position in BEV space;

[0231] The point cloud to image attention path is similar to the image to point cloud path, using point cloud BEV features as queries and image BEV features as keys and values. It employs the same scaled dot product attention mechanism to dynamically mine fine-grained semantic associations between point cloud features and image features, and similarly obtains enhanced point cloud features through residual connections and layer normalization operations. To ensure the effectiveness of bidirectional interaction features, the image BEV features enhanced with bidirectional spatial attention are concatenated with the point cloud BEV features along the channel dimension. A convolutional network is then used to achieve consistent spatial fusion, resulting in unified cross-modal fused BEV features. :

[0232] (twenty four)

[0233] Through a two-way information flow mechanism, the image BEV features inherit the precise spatial structure information provided by the point cloud features, while the point cloud BEV features fuse the rich semantic details carried by the image features. This approach enables the fused BEV features to eliminate projection biases of different sensors in the BEV space, thereby improving the system's perception consistency in different scenarios.

[0234] Further, see Figure 1 and Figure 5 The diffusion generation module includes two stages: training and sampling. The training stage consists of three parts: a noise addition network, a temporal embedding network, and a denoising network. The sampling stage is during the inference process, where the system takes the output of the cross-modal interaction and fusion module as input and generates denoised features through multiple rounds of diffusion.

[0235] (4.1) The noise-adding network takes the output features of the cross-modal interaction and fusion module as input and first generates a random noise with the same shape as the input features. To simulate the uncertainties in real-world environments;

[0236] During the noise addition process, from a uniform distribution Obtain sampling time step To control the intensity of noise input; based on sampling time steps Calculate the noise signal-to-noise ratio parameter To determine the intensity of the noise:

[0237] (25)

[0238] in, and These are the translation factor and scaling factor, used to adjust the nonlinear variation characteristics of the signal-to-noise ratio during diffusion, based on the signal-to-noise ratio parameter. The signal preservation coefficient, which allows for flexible control of noise injection intensity over time, can be further calculated. With noise figure :

[0239] (26)

[0240] in, for Activation function, ensuring the result is in between;

[0241] By signal preservation coefficient With noise figure Obtain the fusion features after adding noise :

[0242] (27)

[0243] The BEV features output by the cross-modal interaction and fusion module are conditionally concatenated with the fused features after adding noise, and the two are then used as input to the denoising network.

[0244] (28)

[0245] (4.2) The time embedding network adds noise to the signal-to-noise ratio parameter in the network. The input is transformed into a high-dimensional feature representation and expanded to a shape consistent with the system input dimension; this is achieved through a set of learnable frequency vectors. The process involves calculating the frequency features at each time point to form a time embedding vector. :

[0246] (29)

[0247] The obtained temporal embedding vector is fed into a multilayer perceptron for nonlinear transformation to obtain the temporal conditional embedding vector. :

[0248] (30)

[0249] The obtained time information helps the system understand and process the changes in noise over time steps, thereby enhancing the system's noise reduction capability;

[0250] (4.3) The denoising network takes the output of the temporal embedding network and the output of the noise addition network as input, and is designed based on the U-Net network. The denoising network mainly consists of two parts: an encoder and a decoder. The encoder is responsible for extracting multi-scale features of the input data. This part consists of several convolutional modules, each of which extracts features at different scales. In each layer, the input features... Spatial features are extracted through convolution operations, and combined with temporal embedding vectors. This is then summed element-wise with the output features; in this way, temporal information is injected into each feature layer.

[0251] (31)

[0252] in, These are the feature outputs generated by the encoder. Each feature output is stored as an intermediate feature for use by the decoder.

[0253] The decoder gradually recovers the original signal and removes noise by combining the features extracted by the encoder with temporal embedding information. Starting from the output of the last encoder layer, features are upsampled layer by layer to recover them. During the decoding process, the features of each layer not only depend on the output of the previous decoder layer but also fuse the features of the corresponding layer in the encoder. High-resolution features are preserved through skip connections, improving the denoising effect.

[0254] (32)

[0255] in, For upsampling operation, and For the input and output of the decoder, It is the feature output generated by the encoder;

[0256] After several rounds of upsampling operations, the denoised BEV features are obtained. Compared with the original features, the features are cleaned up by adding and removing noise, thereby improving the robustness of the system.

[0257] The sampling phase does not involve adding noise; the input is pure Gaussian noise with the same size as the fused BEV feature. The input features and noise are concatenated and fused using convolution to generate initial noise information. Noise is gradually reduced through a denoising network, with temporal embedding information generated by a temporal embedding network controlling the noise removal process. The final output is a denoised feature representation. At each time step, the system uses temporal embedding information to control the noise weighting and removal process, thereby enhancing robustness to noise. Ultimately, the system generates a more accurate feature map through this progressive denoising approach.

[0258] (33)

[0259] in, For noise reduction networks.

[0260] Further, see Figure 1 and Figure 6 The temporal cross-attention fusion module takes the output of the diffusion generation module and the retained historical BEV features as input, and first performs temporal alignment and motion compensation on the historical features; let... Indicates the interval with the current frame is Historical BEV characteristics, obtained through the vehicle's pose change matrix during that time interval. A spatial reprojection operation is performed on historical BEV features to map them to the current BEV space, and an effective region mask is generated by combining the point cloud coverage. The alignment process is represented as follows:

[0261] (34)

[0262] in, This represents a spatial reprojection operation based on pose transformation. For effective region mask, For aligned historical BEV features;

[0263] By introducing temporal information to construct effective spatiotemporal dependencies, the BEV features of the current frame are analyzed. BEV features of historical frames Construct the query, key, and value for time-crossing attention separately:

[0264] (35)

[0265] in, For time interval The corresponding temporal embedding features are used to distinguish the degree of influence of different historical frames in the temporal dimension; , and It is a learnable linear projection matrix that maps the features of the current frame and historical frames into a unified attention subspace;

[0266] The spatiotemporal correlation weights between the current feature and each historical frame are calculated using a multi-head attention mechanism; for the BEV feature at the current moment, in each attention head... Below, based on the query vector of the current frame Aligned history frame key vector The scaled dot product between them is used to calculate the attention weights at different historical time steps, which are expressed in the following form:

[0267] (36)

[0268] in, For the number of attention heads, The total dimension of the features. For attention head Attention weights;

[0269] Each attention head independently captures spatiotemporal relationships at different scales and from different angles. The results from each attention head are then stitched together and combined with weighting coefficients from different historical time steps. By aggregating historical frame features across time, a unified historical context feature is obtained:

[0270] (37)

[0271] in, This is the value vector of historical frames. This represents the total number of historical frames.

[0272] Finally, the historical features and current features are fused into the final BEV feature representation through a gated residual connection mechanism. :

[0273] (38)

[0274] in, For the gated weight matrix, for The activation function controls the degree to which historical information affects the current feature update; historical frame data is extracted through a gating mechanism, enabling the system to adaptively select valuable historical information.

[0275] Further, see Figure 1 The environmental perception decoding module takes the output of the temporal cross-attention fusion module as input to construct specific perception branches for different tasks, so as to achieve a comprehensive understanding of the information about the vehicle's surrounding environment.

[0276] (6.1) Scene semantic segmentation branch, using BEV features Using the convolution class header as input, the semantic category probability distribution corresponding to different locations in the BEV space is calculated to achieve pixel-level semantic understanding of different regions within the scene:

[0277] (39)

[0278] in, This is the header for the semantic segmentation task. The probability value for the semantic category in the BEV space;

[0279] (6.2) Lane detection branch: This branch uses a specific convolutional network and nonlinear activation function to predict the road structure and topological connectivity within the scene, thereby constructing topological information on road connectivity and traffic network structure.

[0280] (40)

[0281] in, For lane line detection head, for Activation function This is the result of lane segmentation;

[0282] (6.3) The target detection branch uses continuous convolution and feature regression head to regress the motion state of targets in the scene, and outputs target state information such as the target's speed, direction and size, so as to achieve accurate prediction of the target state:

[0283] (41)

[0284] in, For target detection head, For target detection results;

[0285] Each task branch is connected to an independent task head, and is trained and optimized according to the specific needs of the environmental perception task. The parameters of each task head are independent of each other to achieve collaborative modeling of multi-task environmental perception.

[0286] Further, see Figure 1 The robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environmental perception system described in this invention requires defining a suitable loss function during training to guide the system towards a stable and accurate environmental perception target. The environmental perception decoding module involves multiple specific tasks; therefore, loss functions for different tasks need to be defined separately to achieve precise optimization and training for different perception tasks.

[0287] (7.1) Semantic segmentation loss The difference between the scene semantic category predicted by the system and the true semantic label is measured using a pixel-level cross-entropy loss function, calculated as follows:

[0288] (42)

[0289] in, BEV characteristics The number of pixels in For the number of semantic categories, Indicates the first The pixel in the actual annotation belongs to the _th _ ... Class indicator value, This represents the system's predicted probability of the corresponding category.

[0290] (7.2) Lane detection loss The binary cross-entropy loss function is used to measure the difference between the system's predicted lane position and the actual lane position.

[0291] (43)

[0292] in, This represents the number of prediction units in the lane detection task. For the first Real lane line labels at each location This corresponds to the predicted probability value;

[0293] (7.3) Target detection loss is used to evaluate the accuracy of the system's prediction of the location, category, and size of traffic participants, and is derived from heatmap loss. Classification loss and regression loss Common components:

[0294] (44) (45) (46) (47)

[0295] in, The total loss for target detection, , These are the true and predicted values ​​of the heatmap, respectively. , These are the true label and predicted probability of the target category, respectively. , These are the ground truth bounding box and the predicted bounding box, respectively. The target number in the BEV space. , , These are the weighting coefficients for each loss term, used to balance the impact of different detection subtasks on the overall training process.

Claims

1. A highly robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environmental perception system, characterized in that: include: The module includes an image feature extraction module, a point cloud feature extraction module, a cross-modal interaction fusion module, a diffusion generation module, a temporal cross-attention fusion module, and an environment perception decoding module. The image feature extraction module takes camera images as input, extracts multi-scale hierarchical features through a pre-trained Swin-Transformer backbone network, and achieves multi-level fusion of shallow high-resolution details and deep semantic features based on the Feature Pyramid Network (FPN). A BEV spatial projection model is constructed through a view transformation network, and the perspective features are unfolded along the pixel ray direction using implicit depth distribution estimation. The features are then compressed and aggregated in the BEV spatial height direction to generate image BEV features with rich semantic information. The point cloud feature extraction module takes LiDAR point cloud as input, converts unstructured point cloud into regular sparse voxels through point cloud voxelization network, extracts multi-scale geometric features by combining sparse convolution feature extraction network, and compresses three-dimensional voxel features to two-dimensional BEV plane through columnar sparse convolution to generate point cloud BEV features with rich geometric information. The cross-modal interaction and fusion module takes the outputs of the image feature extraction module and the point cloud feature extraction module as inputs, constructs a cross-modal feature interaction channel through a bidirectional cross-attention mechanism, calculates a semantic-geometric association weight matrix with image BEV features as queries and point cloud BEV features as keys, and simultaneously establishes a geometric-semantic reverse association matrix with point cloud BEV features as queries and image BEV features as keys. Through dynamic weight allocation, bidirectional complementarity of cross-modal features is achieved, global semantic-geometric association is constructed, and fusion is performed in the deep feature space to output fused BEV features. The diffusion generation module takes the output of the cross-modal interaction and fusion module as input, injects Gaussian noise into the fused BEV features to simulate sensor noise and environmental interference in real-world scenarios, and generates noisy BEV features. Based on the denoising network, a conditional guidance mechanism is constructed, using the fused BEV features as constraints. Through multi-step iteration, noise interference is gradually eliminated, restoring the original feature distribution that was masked by noise while maintaining the consistency of the feature geometry, and finally outputting the denoised BEV features. The temporal cross-attention fusion module takes the output of the diffusion generation module and the output of the historical temporal cross-attention fusion module as inputs to construct a temporal feature fusion mechanism. It uses the vehicle pose transformation matrix to realize the temporal alignment and motion compensation of historical features. It establishes the spatiotemporal correlation between the current frame BEV features and historical temporal features through a multi-head temporal cross-attention mechanism. It dynamically reconstructs the missing features in the region caused by the temporary occlusion of the target or the interruption of the sensor signal. It uses gated residual connection to fuse historical features and current features and outputs BEV features with spatiotemporal continuity. The environmental perception decoding module takes the output of the temporal cross-attention fusion module as input, decodes the BEV features through the multi-task perception head, and outputs the semantic segmentation results, target detection results, and lane line recognition results of the scene, thereby realizing comprehensive perception and scene understanding of the vehicle's surrounding environment.

2. The highly robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environmental perception system according to claim 1, characterized in that: The image feature extraction module consists of three parts: an image feature extraction network, an image feature enhancement network, and a view transformation network. The input to the image feature extraction network is multi-view RGB image data from the camera. Where N represents the total number of camera viewpoints. Indicates the first RGB image data from multiple perspectives and These represent the height and width of the image, respectively. This represents a three-dimensional tensor defined in real space for the image; to facilitate subsequent feature modeling and introduce local structure priors, each image is divided into several non-overlapping image blocks. Each image patch is flattened and mapped from the original pixel space to a fixed-dimensional feature space through a linear projection layer: in, For feature flattening operation, For linear projection operations, Indicates the first Feature vectors of image patches; The Swing-Transformer backbone network consists of several layers, each layer consisting of multiple Transformer modules stacked sequentially; within the same layer, adjacent Transformer modules alternately use window multi-head self-attention module W-MSA and moving window multi-head self-attention module SW-MSA. The Transformer module employing W-MSA performs self-attention computation only within a local window of the image, capturing fine features within that local region. The Transformer module employing SW-MSA achieves cross-window information interaction through periodic movement of the window position. By alternately stacking W-MSA and SW-MSA modules within the layers, the network can gradually expand the feature perception range while maintaining local modeling efficiency, enabling feature representation to transition from local structure perception to broader contextual modeling. The calculation process for each Transformer module is as follows: in, and These represent the input and output characteristics of the module, respectively. This indicates a multi-head self-attention module, when it is Even number For W-MSA, when When it is an odd number For SW-MSA, It is a multilayer perceptron. Presentation layer normalization operation; The core of the Transformer module is the multi-head self-attention mechanism, where self-attention is calculated on a local window basis, i.e., W-MSA; for features within any window... The formula for calculating single-head attention is: in, , and The weight matrix is ​​a learnable matrix. This represents the query, key, and value generated from the input features. For window size, For single-head feature dimensions, For learnable relative position offsets, This is a normalization operation; multi-head self-attention is obtained by computing multiple attention heads in parallel, concatenating them, and then mapping them. in, This is the overall weight matrix; A cyclic window offset strategy is introduced in adjacent Transformer modules to achieve cross-window information interaction; if the first If the first Transformer module uses W-MSA, then the second... Each Transformer module performs a cyclic offset operation along the spatial dimension on the input features, with the offset amount set to half the window size, so that the new window can cover the boundary region of the original window; then, multi-head self-attention calculation is performed on the window partition after offset, which can be represented as: in, and They represent the first Input and output characteristics of each Transformer module For cyclic window offset operators, and All are intermediate features; by alternately stacking Transformer modules using W-MSA and SW-MSA within the same layer, the network can gradually achieve cross-window feature interaction while maintaining local modeling efficiency, thereby effectively enhancing the contextual modeling capability of feature representation; After stacking Transformer modules within a level, features in adjacent spatial domains are aggregated through a block merging operation to obtain the feature representation for the next level; for the th The position in the output features of each level is Features of the place Input features The features within the corresponding neighborhood are concatenated, and channel compression and feature fusion are achieved through convolutional mapping. The calculation process is as follows: in, , , and The neighborhood features of the input features For feature splicing operations, This involves a convolution operation; it reduces the feature space resolution while increasing the channel dimension, thereby progressively enhancing the semantic expressive power of the features; by repeating the above operation at different levels, the final output is multi-scale image features. Deep features encode global semantics, while shallow features preserve high-resolution details; The image feature enhancement network takes the multi-layer output of the image feature extraction network as input, adopts an FPN structure, and fuses features from different scales; the four layers of multi-scale input features are unified in the number of channels of each layer through lateral convolution. in, For convolution operations on features at different scales, the number of channels in each layer is adjusted uniformly to output features. ; From the deepest layer Initially, learnable deconvolution upsampling is used to gradually fuse shallow features: In the formula, For the first Fusion features at various scales; This is a deconvolution upsampling operation; To preserve the characteristics of each level, a hierarchical attention weighting mechanism is introduced for the fused multi-level features; for the first level... Layer fusion features Through learnable attention weights The weighted adjustment is performed on it, and its calculation form is as follows: in, The fused features after applying hierarchical attention weights, For the range of values ​​within The learnable space weights within, This is an average pooling operation; Finally, cross-level information aggregation is performed on the fused features: By using the above method, while maintaining the complementarity of multi-level features, adaptive enhancement of key information is achieved, and enhanced image features are output. The view transformation network takes the output features of the image feature enhancement network as input and generates a pixel-wise depth probability distribution through the depth prediction head. : in, Represents the discrete depth probability distribution. and These represent the length and width of the image features, respectively. The number of discrete depth intervals; Expanding the image features by the outer product of the depth probability generates 3D pseudo-point cloud features. : In the formula, Represents pixels Two-dimensional image features at the location, For depth interval indexing, 2D image features are uplifted to 3D space along the ray direction; using the camera intrinsic and extrinsic parameter matrices, pseudo-point clouds are mapped to BEV space, and voxel pooling is used to highly compress the mapped discrete point cloud features: In the formula, Indicates the spatial location of BEV At height voxel characteristics on For highly discrete layers, BEV spatial location BEV features at the location; spatial alignment and stitching of BEV features from multiple surround-view cameras to output the final image BEV features. .

3. The highly robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environmental perception system according to claim 1, characterized in that: The point cloud feature extraction module consists of two parts: a point cloud voxelization network and a sparse convolutional feature extraction network. The point cloud voxelization network takes the reflected signal from the lidar as input and maps the point cloud onto a voxel grid within a certain range around the vehicle based on its spatial location. Each voxel represents a small area in the space around the vehicle. This spatial partitioning distributes the original point cloud data into each voxel. Subsequently, the point features within each voxel are accumulated and averaged to obtain the voxel features. in, For the first Individual characteristics, This represents the number of points in the voxel. Indicates the first Features of each point; The sparse convolutional feature extraction network takes voxelized features as input and extracts features from non-empty voxel regions through sparse convolutional layers; the update process of non-empty voxel features by a single sparse convolutional layer is defined as follows: in, This indicates a sparse convolution operator that is executed only within non-empty voxels and their neighborhoods. and For the convolution parameters of the corresponding layer, For normalization operations, It is a non-linear activation function. and The first Input and output features of sparse convolution layers; Based on this, the sparse convolutional feature extraction network is composed of several stacked sparse convolutional modules. Each module introduces residual connections and dilated convolutions on top of the above sparse convolutions, and its computational form is as follows: in, For expansion rate The sparse convolution with holes, the dilation rate As the network hierarchy increases; Multi-scale fusion of sparse features from different output levels is performed. Feature weighting is achieved through upsampling and the combination of learnable channel attention weights to obtain fused sparse voxel features: in, For learnable channel attention weights, This indicates an upsampling operation; it combines the sparse features of the fused sample. Mapped to the BEV space, the dense BEV features are finally obtained: in, For point cloud BEV features, This is a feature mapping operation.

4. The highly robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environmental perception system according to claim 1, characterized in that: The cross-modal interaction and fusion module takes the image BEV features output by the image feature extraction module and the point cloud BEV features output by the point cloud feature extraction module as input, and performs global mean pooling and max pooling on the dual-modal features respectively to generate a feature prototype matrix: in, Represents image modality or point cloud modality. This represents the overall distribution feature matrix obtained by averaging statistical analysis of all spatial location features. This represents the salient region feature matrix obtained by taking the maximum response of all spatial location features. and For the length and width of the BEV space, and BEV spatial coordinates; By adaptively weighting and fusing the average and salient features using learnable weight coefficients, the final global features for each modality are obtained. in, and These are learnable fusion weight coefficients used to adjust the proportions of overall distribution information and salient response information in the final global features; To achieve fine-grained and effective interaction between image and point cloud features, in the attention path from image to point cloud, image BEV features are used as the query vector, and point cloud BEV features are used as the key and value. A spatial attention weight matrix is ​​calculated by scaling the dot product to dynamically capture the correlation between image semantic information and point cloud spatial geometric information. in, This is the attention weight matrix between the image and the point cloud. and For a learnable projection matrix, For subspace dimension, For global features of the image; By using the obtained attention weight matrix, point cloud geometric features relevant to image semantics are adaptively selected, and enhanced image feature representations are obtained through residual connections and layer normalization operations. : in, For a learnable projection matrix, Represents any position in BEV space; In the point cloud to image attention path, point cloud BEV features are used as queries, and image BEV features are used as keys and values. The same scaling dot product attention mechanism used in the image to point cloud attention path is employed to dynamically mine fine-grained semantic associations between point cloud features and image features. Enhanced point cloud features are obtained through residual connections and layer normalization operations. Finally, the BEV features of the image enhanced with bidirectional spatial attention are concatenated with the BEV features of the point cloud along the channel dimension, and a convolutional network is used to achieve consistent fusion in the spatial dimension to obtain unified cross-modal fused BEV features. : 。 5. The highly robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environmental perception system according to claim 1, characterized in that: The diffusion generation module includes two phases: training and sampling. The training phase consists of three parts: a noise addition network, a temporal embedding network, and a denoising network. The sampling phase occurs during the inference process, where the system takes the output of the cross-modal interaction and fusion module as input and generates denoised features through multiple rounds of diffusion. The noise-adding network takes the output features of the cross-modal interaction and fusion module as input and first generates a random noise with the same shape as the input features. To simulate the uncertainties in real-world environments; During the noise addition process, from a uniform distribution Obtain sampling time step To control the intensity of noise input; based on sampling time steps Calculate the noise signal-to-noise ratio parameter To determine the intensity of the noise: in, and These are the translation factor and scaling factor, used to adjust the nonlinear variation characteristics of the signal-to-noise ratio during diffusion, based on the signal-to-noise ratio parameter. The signal preservation coefficient, which allows for flexible control of noise injection intensity over time, can be further calculated. With noise figure : in, for Activation function, ensuring the result is in between; By signal preservation coefficient With noise figure Obtain the fusion features after adding noise : ； The BEV features output by the cross-modal interaction and fusion module are conditionally concatenated with the fused features after adding noise, and the two are then used as input to the denoising network. in, The BEV characteristics after fusion conditions; The time embedding network adds noise to the signal-to-noise ratio parameter of the network. As input, it is transformed into a high-dimensional feature representation and expanded to a shape consistent with the system input dimension; through a set of learnable frequency vectors The process involves calculating the frequency features at each time point to form a time embedding vector. : The obtained temporal embedding vector is fed into a multilayer perceptron for nonlinear transformation to obtain the temporal conditional embedding vector. : ； The denoising network takes the outputs of the temporal embedding network and the noise addition network as inputs, and is designed based on the U-Net network. The denoising network mainly consists of two parts: an encoder and a decoder. The encoder is responsible for extracting multi-scale features from the input data; this part consists of several convolutional modules, each extracting features at different scales. In each layer, the input features... Spatial features are extracted through convolution operations, and combined with temporal embedding vectors. This is then summed element-wise with the output features; in this way, temporal information is injected into each feature layer. in, These are the feature outputs generated by the encoder. Each feature output is stored as an intermediate feature for use by the decoder. The decoder gradually recovers the original signal and removes noise by combining the features extracted by the encoder with temporal embedding information. Starting from the output of the last encoder layer, features are upsampled layer by layer to recover them. During the decoding process, the features of each layer not only depend on the output of the previous decoder layer but also fuse the features of the corresponding layer in the encoder. High-resolution features are preserved through skip connections, improving the denoising effect. in, For upsampling operation, and For the input and output of the decoder, It is the feature output generated by the encoder; After several rounds of upsampling operations, the denoised BEV features are obtained. .

6. The highly robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environmental perception system according to claim 5, characterized in that: The sampling phase does not involve adding noise; the input is pure Gaussian noise with the same size as the fused BEV feature. The input features and noise are concatenated and fused using convolution to generate preliminary noise information. A denoising network gradually reduces the noise, utilizing temporal embedding information generated by a temporal embedding network to control the noise removal process. The final output is a denoised feature representation. in, For noise reduction networks.

7. The highly robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environmental perception system according to claim 1, characterized in that: The temporal cross-attention fusion module takes the output of the diffusion generation module and the retained historical BEV features as input, and first performs temporal alignment and motion compensation on the historical features; let... Indicates the interval with the current frame is Historical BEV characteristics, obtained through the vehicle's pose change matrix during that time interval. A spatial reprojection operation is performed on historical BEV features to map them to the current BEV space, and an effective region mask is generated by combining the point cloud coverage. The alignment process is represented as follows: in, This represents a spatial reprojection operation based on pose transformation. For effective region mask, For aligned historical BEV features; By introducing temporal information to construct effective spatiotemporal dependencies, the BEV features of the current frame are analyzed. BEV features of historical frames Construct the query, key, and value for time-crossing attention separately: in, For time interval The corresponding temporal embedding features are used to distinguish the degree of influence of different historical frames in the temporal dimension; , and It is a learnable linear projection matrix that maps the features of the current frame and historical frames into a unified attention subspace; The spatiotemporal correlation weights between the current feature and each historical frame are calculated using a multi-head attention mechanism; for the BEV feature at the current moment, in each attention head... Below, based on the query vector of the current frame Aligned history frame key vector The scaled dot product between them is used to calculate the attention weights at different historical time steps, which are expressed in the following form: in, For the number of attention heads, The total dimension of the features. For attention head Attention weights; Subsequently, the results from each attention head are concatenated and weighted by coefficients from different historical time steps. By aggregating historical frame features across time, a unified historical context feature is obtained: in, This is the value vector of historical frames. The total number of historical frames is used; finally, the historical features and current features are fused into the final BEV feature representation through a gated residual connection mechanism. : in, For the gated weight matrix, for The activation function is used to control the degree to which historical information affects the current feature update.

8. The highly robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environmental perception system according to claim 1, characterized in that: The environmental perception decoding module takes the output of the temporal cross-attention fusion module as input and constructs specific perception branches for different tasks, including scene semantic segmentation branch, lane line detection branch, and target detection branch, in order to achieve a comprehensive understanding of the environmental information around the vehicle. The scene semantic segmentation branch uses BEV features Using the convolution class header as input, the semantic category probability distribution corresponding to different locations in the BEV space is calculated: in, This is the header for the semantic segmentation task. The probability value for the semantic category in the BEV space; The lane detection branch uses a specific convolutional network and nonlinear activation function to predict the road structure and topological connectivity within the scene: in, For lane line detection head, for Activation function This is the result of lane segmentation; The target detection branch uses continuous convolution and a feature regression head to regress the motion state of targets in the scene and outputs target state information: in, For target detection head, For target detection results; Each task branch is connected to an independent task head, which is trained and optimized according to the specific needs of the environmental perception task. The parameters of each task head are independent of each other to achieve collaborative modeling of multi-task environmental perception.

9. A highly robust intelligent vehicle bird's-eye view multi-source spatiotemporal fusion environmental perception system according to claim 8, characterized in that: During training, loss functions are defined for different tasks to guide the system to converge toward a stable and accurate environmental perception target: loss functions include semantic segmentation loss, lane detection loss, and object detection loss; The semantic segmentation loss The difference between the scene semantic category predicted by the system and the true semantic label is measured using a pixel-level cross-entropy loss function, calculated as follows: in, BEV characteristics The number of pixels in For the number of semantic categories, Indicates the first The pixel in the actual annotation belongs to the _th _ ... Class indicator value, This represents the system's predicted probability of the corresponding category. Lane detection loss The binary cross-entropy loss function is used to measure the difference between the system's predicted lane position and the actual lane position. in, This represents the number of prediction units in the lane detection task. For the first Real lane line labels at each location This corresponds to the predicted probability value; The target detection loss is used to evaluate the accuracy of the system's predictions of the location, category, and size of traffic participants, and is derived from the heatmap loss. Classification loss and regression loss Common components: in, The total loss for target detection, , These are the true and predicted values ​​of the heatmap, respectively. , These are the true label and predicted probability of the target category, respectively. , These are the ground truth bounding box and the predicted bounding box, respectively. The target number in the BEV space. , , These are the weighting coefficients for each loss term, used to balance the impact of different detection subtasks on the overall training process.

Images