Bird-eye view generation method based on multi-scale feature transformation and time sequence context

Abstract

The invention discloses a bird's-eye view generation method based on multi-scale feature transformation and time sequence context, and relates to the technical field of map generation. According to the method, the image and the laser radar data are fused, complementarity of the image and the laser radar data is fully utilized, the accuracy and robustness of aerial view generation are improved, and the aerial view can still be kept stable especially in severe weather; the multi-scale space conversion module extracts features of different scales, the feature expression ability is enhanced, and the aerial view is clearer and more accurate; time sequence information is introduced, current features are enhanced by using past time features, and the dynamic perception capability is improved; an advanced backbone network and a feature alignment and fusion module are adopted to ensure high efficiency and flexibility; through application of specific network structures such as Swinin-T, Point Pillars and a random inactivation layer, the accuracy and generalization ability are further improved.

Description

Technical Field

[0001] The present application belongs to the field of map generation technology, and in particular relates to a bird's-eye view generation method based on multi-scale feature transformation and temporal context. Background Art

[0002] Autonomous driving technology, a cutting-edge topic in today's technological landscape, has a clearly defined architecture consisting of three core components: perception, decision-making, and execution. Perception plays a crucial role, acting as the "eyes" and "ears" of the autonomous driving system. It efficiently integrates information from multiple sensors to accurately generate environmental perception results. These results provide solid data support for subsequent decision-making and execution, laying the foundation for the safe and stable operation of autonomous driving systems.

[0003] The Bird's-Eye-View (BEV) generation algorithm is a key component of autonomous driving perception algorithms. Its core goal is to deeply analyze the surrounding environment from multi-sensor data and generate a semantic map in real time. The generated map includes a rich set of elements, such as lane boundaries, lane markings, and crosswalks. These elements provide the vehicle with precise road and location information, enabling it to clearly understand its surroundings and make informed driving decisions.

[0004] Traditional map perception solutions rely primarily on analyzing perspective images, but this approach has numerous drawbacks. For one thing, perspective images can be subject to occlusion, potentially preventing the acquisition of crucial environmental information. Furthermore, the results obtained from perspective images are at the pixel level, deviating from real-world spatial relationships. Complex post-processing is required to establish a connection with the real world, which undoubtedly increases the difficulty of subsequent decision-making.

[0005] Most existing methods do not fully consider the impact of bumps on the bird's-eye view generation task. During actual driving, vehicles will inevitably encounter bumpy road conditions. Bumps will cause two problems: on the one hand, bumps will cause changes in camera extrinsics. Camera extrinsics describe the position and posture of the camera in the world coordinate system. Changes in extrinsics will cause misalignment in the projection of image features to BEV features, causing the generated bird's-eye view to deviate from the actual environment. On the other hand, since bumps have different impacts on different modal data, this will destroy the consistency between different modal features, resulting in feature misalignment during feature fusion. Under the combined influence of these two problems, existing bird's-eye view generation algorithms have low generation quality in bumpy scenarios and cannot meet the high-precision perception requirements of autonomous driving systems under complex road conditions. Summary of the Invention

[0006] The purpose of this application is to provide a bird's-eye view generation method based on multi-scale feature transformation and temporal context. By fusing image data and lidar data and utilizing the technical means of multi-scale feature transformation and temporal context information, the problems of feature extraction, alignment, fusion and temporal enhancement in bird's-eye view generation are solved, thereby improving the accuracy and robustness of bird's-eye view generation.

[0007] To achieve the above objectives, the present invention provides a method for generating a bird's-eye view image based on multi-scale feature transformation and temporal context, comprising: Obtaining original modal data, which includes image data and lidar data; The image data is processed through the image backbone network to obtain the forward-view image features, and the forward-view image features are transformed into multi-scale space to obtain the image BEV features. The lidar data is processed through the radar backbone network to obtain the lidar features, and the lidar features are compressed along the Z axis to obtain the lidar BEV features. The image BEV features and the lidar BEV features are aligned and fused to obtain the BEV fusion features. The BEV fusion features of the past n moments are introduced, and the BEV fusion features of the current moment are enhanced by the BEV fusion features of the past n moments to obtain the enhanced BEV fusion features. The enhanced BEV fusion features are processed by the semantic segmentation task head in the decoder to obtain the bird's-eye view generation result.

[0008] The above method according to the embodiment of the present application may also have the following additional technical features: Furthermore, Swin-T is used as the image backbone network to process the image data and output features of four different scales , the number of channel dimensions corresponding to the features of four different scales is ,right and To unify the dimensions, use bilinear interpolation Up-sample and convolute it with two layers of convolution and The fusion is performed to obtain the fused image features, and the fused image features are transformed into multi-scale space to obtain the image BEV features.

[0009] Furthermore, PointPillars is used as the radar backbone network to perform columnar feature processing on the lidar data to obtain pseudo-2D image features. The pseudo-2D image features are input into the 2D convolutional network to obtain features of three different scales. The features of the three different scales are upsampled to unify the spatial dimensions and the channel dimensions are spliced ​​to obtain the lidar BEV features.

[0010] Furthermore, a multi-scale spatial transformation module is constructed to perform multi-scale spatial transformation on the front view image features. The multi-scale spatial transformation module includes an encoder, a view relationship module, a multi-scale feature fusion module and a decoder connected in sequence. The encoder consists of two residual convolution modules connected together. Each residual convolution module performs two convolution operations and two times downsampling operations on the front view image features to obtain downsampled features. The view relationship module consists of two fully connected layers and a convolutional layer. The downsampled features are mapped through the two fully connected layers, and the channel dimension is reduced through a convolutional layer to obtain mapping features at different levels. The multi-scale feature fusion module consists of an offset prediction branch connected to a splicing layer. The offset prediction branch also includes two convolutional layers. The offset prediction branch splices the mapping features of different levels in the channel dimension, and the two convolutional layers perform offset prediction to obtain the offsets corresponding to the mapping features of different levels. Based on the offsets, the mapping features of different levels are sampled by interpolation and connected in the channel dimension through the splicing layer to obtain the preliminary image BEV features. The decoder uses two bilinear interpolation series residual convolution blocks to expand the spatial dimension of the preliminary image BEV features. The residual convolution block consists of two series-connected 3×3 convolution layer trunks and a 1×1 convolution layer branch to obtain the image BEV features.

[0011] Furthermore, a random inactivation layer is inserted between the two serially connected 3×3 convolutional layer backbones of the decoder, and 20% of the neurons are randomly set to 0 through the random inactivation layer.

[0012] Furthermore, the image BEV features and the lidar BEV features are aligned and fused by constructing a feature alignment and fusion module, which includes a deformable attention module, a first convolutional layer, and a channel attention module connected in parallel, and a triple attention module and a second convolutional layer connected in series. Among them, the image BEV features and the lidar BEV features are preliminarily spliced ​​and input into the first convolutional layer to obtain the preliminary fused BEV features. Then, the image BEV features and the preliminary fused BEV features are input into the deformable attention module together, and the channel dimension is superimposed to obtain the corrected features. Input the lidar BEV features into the channel attention module to obtain enhanced lidar BEV features; then concatenate the corrected features and enhanced lidar BEV features in the channel dimension to obtain preliminary BEV fusion features; The preliminary BEV fusion features are sequentially input into the triple attention module and the second convolutional layer, and the spatial and channel dimensions are modeled simultaneously to obtain the BEV fusion features.

[0013] Furthermore, feature alignment is performed on the BEV fusion features of the past n moments and the BEV fusion features of the current moment, and the BEV fusion features of the past n moments are converted to the vehicle coordinate system of the current moment according to the change of the vehicle position information. Feature aggregation is then performed on the BEV fusion features of the past n moments converted to the vehicle coordinate system of the current moment and the BEV fusion features of the current moment to obtain enhanced BEV fusion features.

[0014] Furthermore, before aligning the BEV fusion features at the past n moments with the BEV fusion features at the current moment, a Detach operation is performed on the BEV fusion features at the past n moments.

[0015] The bird's-eye view image generation method based on multi-scale feature transformation and temporal context provided by the embodiment of the present application has the following beneficial technical effects compared with the existing technology: The embodiments of the present application improve the accuracy of bird's-eye view generation by fusing image data and lidar data, making full use of the complementarity of the two modal data; image data provides rich visual information, while lidar data provides accurate distance and position information. The combination of the two can more comprehensively describe the environment around the vehicle; multimodal data fusion makes the method more robust in the face of different weather, lighting and other environmental conditions; for example, in severe weather conditions, image data may be seriously affected, but lidar data can still provide reliable distance information, thereby ensuring the stability of bird's-eye view generation.

[0016] The embodiment of the present application performs multi-scale spatial transformation on the forward-view image features through a multi-scale spatial transformation module. This method can extract feature information at different scales. This feature information contains both detailed and global information about the vehicle's surroundings, which helps to more comprehensively understand the environment around the vehicle. The multi-scale feature transformation can enhance the expressiveness of the features, making the generated bird's-eye view more accurate and clear. By fusing feature information at different scales, the complexity and diversity of the vehicle's surroundings can be better captured.

[0017] The embodiment of the present application enhances the BEV fusion features at the current moment by introducing the BEV fusion features of the past n moments, and makes full use of the time series information to improve the accuracy of the bird's-eye view generation; the time series information contains the dynamic change information of the vehicle's surrounding environment, which helps to more accurately predict the vehicle's surrounding environment at future moments; the time series context enhancement makes the method have stronger dynamic perception capabilities; by fusing the feature information of the past moments, it can better understand the dynamic changes of the vehicle's surrounding environment, thereby generating a more accurate and reliable bird's-eye view.

[0018] The embodiments of the present application adopt an advanced backbone network and feature alignment and fusion modules, which are highly efficient. These network structures can quickly process large amounts of data and extract useful feature information, thereby ensuring the real-time generation of bird's-eye views. The multi-scale spatial conversion module and the feature alignment and fusion module are both flexible. By adjusting the parameters and structures of these modules, they can adapt to different application scenarios and requirements, thereby improving the versatility and scalability of the method.

[0019] The embodiment of the present application uses Swin-T as the image backbone network to output features of four different scales, and obtains richer image BEV features through dimensional unification and fusion, which helps to improve the accuracy and clarity of bird's-eye view generation; using PointPillars as the radar backbone network can perform columnar feature processing on lidar data and obtain pseudo-2D image features, which helps to better fuse lidar data with image data and improve the accuracy of bird's-eye view generation; inserting a random deactivation layer between the two serially connected 3×3 convolutional layer backbones of the decoder can prevent overfitting and improve the generalization ability of the model, which helps to ensure the stability and reliability of the method on different data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a process for generating a bird's-eye view based on multi-scale feature transformation and temporal context according to an embodiment of the present application is shown; Figure 2 A schematic diagram of the model structure of a bird's-eye view generation method based on multi-scale feature transformation and temporal context according to an embodiment of the present application is shown; Figure 3 A schematic structural diagram of a multi-scale spatial conversion module according to an embodiment of the present application is shown; Figure 4 A schematic structural diagram of a multi-scale feature fusion module according to an embodiment of the present application is shown; Figure 5 A schematic structural diagram of a decoder according to an embodiment of the present application is shown; Figure 6 A schematic diagram of the structure of the feature alignment and fusion module in an embodiment of the present application is shown; Figure 7 A schematic diagram of the structure of a deformable attention module according to an embodiment of the present application is shown; Figure 8 A schematic diagram of the structure of the triple attention module according to an embodiment of the present application is shown; Figure 9 A schematic diagram of the structure of a time series fusion module based on vehicle position information according to an embodiment of the present application is shown; Figure 10 A comparison chart of the IoU indicators of features at different scales according to an embodiment of the present application is shown; Figure 11 A comparison chart of CPA indicators of different scale features of the embodiment of the present application is shown; Figure 12 It shows the effect diagram of different frame numbers fusion of the timing module of the embodiment of the present application; Figure 13 The following diagrams show the generation results of bird's-eye views of different modes according to an embodiment of the present application; Figure 14 A first comparison diagram of the bird's-eye view results under flat road conditions in an embodiment of the present application is shown; Figure 15 A second comparison diagram of the bird's-eye view results under flat road conditions according to an embodiment of the present application is shown; Figure 16 A first comparison diagram of the bird's-eye view results under slightly bumpy road conditions according to an embodiment of the present application is shown; Figure 17 A second comparison diagram of the bird's-eye view results under slightly bumpy road conditions according to an embodiment of the present application is shown; Figure 18 A first comparison diagram of the bird's-eye view results under bumpy road conditions according to an embodiment of the present application is shown; Figure 19 A second comparison diagram of the bird's-eye view results under bumpy road conditions of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some, rather than all, structures related to the present application are shown in the accompanying drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0022] As used herein, the terms "comprise," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0023] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0024] like Figure 1 As shown, the embodiment of the present application provides a method for generating a bird's-eye view based on multi-scale feature transformation and temporal context, comprising the following steps: Step 101: Acquire original modal data, which includes image data and lidar data.

[0025] In this step, acquiring raw modal data is the foundation of the entire bird's-eye view image generation method based on multi-scale feature transformation and temporal context. This step ensures sufficient and accurate data sources for subsequent processing. Specifically, raw modal data mainly includes image data and lidar data, both of which play a vital role in fields such as autonomous driving and intelligent transportation.

[0026] Image data, typically captured by on-board cameras, provides visual information about the vehicle's surroundings. During bird's-eye-view image generation, this image data is used to extract forward-view image features. These features then undergo a multi-scale spatial transformation to generate image BEV features. Factors such as image data quality, resolution, and capture angle directly impact the accuracy of subsequent feature extraction and BEV feature generation.

[0027] LiDAR data, acquired through a LiDAR sensor, provides precise three-dimensional information about the vehicle's surroundings. Compared to image data, LiDAR data offers greater accuracy and reliability in distance measurement, object recognition, and environmental modeling. During the bird's-eye view generation process, LiDAR data is used to extract LiDAR features, which are then compressed and transformed to generate LiDAR BEV features.

[0028] Acquiring high-quality, accurate raw modal data is key to ensuring the effectiveness of bird's-eye view generation methods. In practical applications, it is necessary to ensure the normal operation of cameras and lidar sensors, as well as the real-time transmission and storage of data.

[0029] Step 102: Process the image data through the image backbone network to obtain forward-view image features, perform multi-scale spatial transformation on the forward-view image features to obtain image BEV features; process the lidar data through the radar backbone network to obtain lidar features, compress the lidar features along the Z axis to obtain lidar BEV features.

[0030] like Figure 2 As shown in the figure, the embodiment of the present application uses Swin-T as the image backbone network, and the input image size is 256×704. This is an advanced architecture based on Transformer, which is particularly suitable for processing image data. Swin-T can effectively capture long-range dependencies in images while maintaining computational efficiency. After Swin-T processes the image data, it outputs four features of different scales. , and their channel dimensions are These features capture information at different levels of the image, from low-level textures to high-level semantics. Low-level features primarily contain the visual characteristics of the image (color, texture, etc.), while high-level features primarily contain the semantic characteristics of the objects. Compared to visual features, semantic features are more important for the bird's-eye view generation task. Therefore, higher-level features are selected as the input to the multi-scale spatial transformation module.

[0031] In order to perform multi-scale spatial transformation, it is necessary to and To unify the dimensions. Specifically, bilinear interpolation is used to Upsample to make its spatial dimension equal to Then, two layers of convolution are used to match and The fusion is performed to obtain the fused image features. This fusion method helps to retain the useful information in features of different scales while eliminating the problems caused by scale differences.

[0032] The fused image features are sent to the multi-scale spatial conversion module to generate image BEV features. The size of the image BEV feature map in the embodiment of the present application is 200×200, which is determined by the set perception range and voxel size.

[0033] After the image data has been processed by the image backbone network, the semantic features of the image have been extracted. For the bird's-eye view generation task, it is also necessary to perform spatial transformation on the front view image features to map the front view image features to the bird's-eye view space. However, road bumps will destroy the existing mapping relationship, resulting in deviations in the mapping from the front view to the bird's-eye view. Therefore, the embodiment of the present application designs a multi-scale spatial transformation module to construct a multi-level mapping relationship for multi-scale features, so as to enhance the stability and accuracy of the entire mapping process. Figure 3 As shown in Figure 3, the module includes an encoder, a view relation module, a multi-scale feature fusion module and a decoder.

[0034] The encoder consists of two residual convolution modules, each of which performs two convolution operations and two-fold downsampling on the input features, gradually extracting and compressing the features. The encoder further downsamples the front-view image features and obtains multi-scale high-level image features.

[0035] The View Relationship Module, or VRM, is essentially a multi-layer perceptron consisting of two fully connected layers and one convolutional layer. The VRM learns the mapping relationship between the front view and the bird's-eye view through the two fully connected layers, and reduces the channel dimension of the features through the subsequent convolutional layers, unifying the mapping results of image features at different levels in the channel dimension. Through the VRM, the model learns the relationship between any two points in the front view and the bird's-eye view, and maps and reduces the downsampled features through the fully connected layers and convolutional layers to obtain mapping features at different levels. These features capture information at different viewpoints and scales in the image, and represent the mapping relationship between the bird's-eye view and the front view image features. The formula is as follows:

[0036] in, Indicates the Layer front view image features; Indicates the Mapping features at different levels of the layer; Indicates the Layer front view image features and the first The correspondence between mapping features at different levels; express spatial feature dimension.

[0037] After obtaining the mapping features at different levels, they need to be fused to fully utilize the advantages of multi-scale features and eliminate projection bias. Since the VRM modules of front-view image features at different scales are different, the mapping relationships they learn are also different. Although this multi-scale mapping method learns more complex feature mapping relationships, when fusing mapping features at different levels, if only a simple channel dimension superposition method is used, the synergy between multi-scale features will not be fully utilized. It may even lead to degraded model performance due to feature misalignment between mapping features at different levels.

[0038] Therefore, if Figure 4 As shown, the embodiment of the present application proposes a multi-scale feature fusion module (MBF). Before fusing mapping features at different levels, MBF will perform fine adjustments in spatial dimensions and semantic levels based on the characteristics of features at each level through alignment operations between features.

[0039] Specifically, the embodiment of the present application first sets an offset prediction branch to As input, the mapping features at different levels are spliced ​​in channel dimension, and the offset is predicted through two convolutional layers to obtain the offset corresponding to the mapping features at different levels. ,in The two channel dimensions of represent the offset of the feature pixel in two different spatial dimensions. After that, it is necessary to align the mapping features at different levels. Represents the features to be aligned, and the alignment operation of the features to be aligned is represented by the following formula:

[0040] This process can be understood as using interpolation to treat the position of the alignment feature. Sampling is performed, where Represents the BEV features of the image after alignment operation; Indicates the characteristics The offset in the corresponding dimension; Indicates the characteristics The offset in the corresponding dimension.

[0041] Finally, it is necessary to concatenate the BEV features of the aligned image in the channel dimension through concat to obtain the preliminary image BEV features. The formula is as follows:

[0042] in, Represents preliminary image BEV features; represents the connection of channel dimension, After being processed by the MBF module, the bird's-eye view features of different scales are spatially and semantically adjusted, providing a basis for their full fusion.

[0043] The decoder is mainly used to expand the spatial dimension of the preliminary image BEV features, and finally obtain the image BEV features. The decoder uses two bilinear interpolation series residual convolution blocks to gradually expand the size of the preliminary image BEV features from (18×18) to (200×200). Linear interpolation simply selects the original pixel value closest to the target pixel position. It has no learnable parameters, cannot be adaptively adjusted according to the characteristics of the data, and cannot improve the semantic expression ability of the features. This method only obtains a feature map with smooth transitions, without further mining and fusion of features, and has poor flexibility. Therefore, the decoder designed in the embodiment of the present application combines the advantages of the former's reasonable pixel inference and the latter's dynamic learning and adjustment of features by connecting bilinear interpolation and residual convolution in series, so that the model has better adaptability to feature maps of different inputs and improves the robustness of the features.

[0044] Specifically, such as Figure 5 As shown, the backbone of the residual convolution block in the decoder is composed of two series-connected 3×3 convolutional layers, and the residual branch is processed by a 1×1 convolution. In addition, the embodiment of the present application also inserts a random deactivation layer between the two series-connected 3×3 convolutional layer backbones of the decoder to perform a random deactivation (Dropout) operation, randomly setting 20% ​​of the neurons to 0. After randomly discarding neurons, the input features received by the subsequent convolutional layers will change, which prompts the convolutional layers to learn different feature combinations and more diverse feature representations, thereby making the features learned by the network richer and more diverse, and improving the entire model's ability to understand and characterize data. In addition, the random deactivation operation causes the structure of the network to change dynamically during each training, which is equivalent to training multiple sub-networks with different structures, and these sub-networks share some parameters. In this way, the ability of the model to generalize is improved, and the impact of noise or blur caused by bumps on model performance is reduced.

[0045] After being processed by the decoder, the final output of the multi-scale spatial conversion module is obtained, namely the image BEV feature , where the size of the image BEV feature is By the set bird's eye view range and voxel size Jointly decided, the formula is as follows:

[0046] This embodiment of the application uses PointPillars as the radar backbone network, an architecture specifically designed for processing LiDAR data. PointPillars can effectively convert LiDAR point cloud data into pseudo-2D image features for subsequent processing.

[0047] PointPillars processes the lidar data into columnar features, generating pseudo-2D image features. These features capture the spatial distribution and intensity information in the point cloud data. These pseudo-2D image features are fed into a 2D convolutional network to generate features at three different scales. These features reflect the spatial structure of the point cloud data at different scales. To generate lidar BEV features, the three scale features are upsampled to a uniform spatial dimension of 200×200 and concatenated along the channel dimension. This processing helps preserve the multi-scale information in the point cloud data while converting it into a format suitable for BEV generation.

[0048] Step 103: perform feature alignment and fusion on the image BEV features and the lidar BEV features to obtain BEV fusion features, introduce the BEV fusion features of the past n moments, enhance the BEV fusion features of the current moment by using the BEV fusion features of the past n moments to obtain enhanced BEV fusion features, process the enhanced BEV fusion features by the semantic segmentation task head in the decoder, and obtain the bird's-eye view generation result.

[0049] After obtaining the image BEV features and the lidar BEV features respectively, they need to be fused to provide BEV fusion features for subsequent tasks. However, since the impact of turbulence on different modal data is not the same, different modal features will have feature misalignment problems when fusing features. For image BEV features, since the mapping from the front view to the bird's-eye view is essentially not separated from the assumption that the ground is flat, the turbulence problem will have a serious impact on the image BEV features, resulting in feature projection misalignment. At the same time, the lidar BEV features are less affected by turbulence and can still maintain a stable spatial structure under turbulence. Therefore, if Figure 6 As shown, the embodiment of the present application designs a feature alignment and fusion module, which utilizes the stable spatial structure of the LiDAR BEV feature to correct the projection misalignment of the image BEV feature, so that the two modal features can achieve spatial position matching, and uses the channel attention mechanism to self-enhance the LiDAR BEV feature to eliminate the interference of bumps on the point cloud data. In addition, when fusing two different modal features, it is also necessary to fully consider how to give full play to the respective advantages of the two modal features, such as the image BEV feature has denser semantic information and the LiDAR BEV feature has more stable spatial properties. Therefore, the model also uses a triple attention mechanism to process the preliminary BEV fusion features, and simultaneously models its space and channels, allowing the model to select more important features for different scenarios, so that the features of different modalities can give full play to their respective advantages.

[0050] This module contains several key components, and the specific process is as follows: Firstly, the image BEV feature and the lidar BEV feature are preliminarily spliced, and the spliced feature is input to the first convolutional layer for convolution operation to obtain a preliminary fusion BEV feature, and then the preliminary fusion BEV feature is input to the deformable attention module together with the image BEV feature. In the deformable attention module, the channel dimension is superimposed, and the features are weighted through the attention mechanism to obtain the corrected feature. This step helps to enhance the relevance between features and improve the fusion effect. At the same time, the lidar BEV feature is input to the channel attention module, and the channel attention module enhances the lidar BEV feature through the channel attention mechanism to obtain the enhanced lidar BEV feature.

[0051] Then, the corrected feature and the enhanced lidar BEV feature are spliced in the channel dimension to obtain a preliminary BEV fusion feature, and the preliminary BEV fusion feature is sequentially input to the triple attention module and the second convolutional layer for spatial and channel dimension modeling to finally obtain the BEV fusion feature.

[0052] Specifically, in the deformable attention module, in order to better utilize the lidar BEV feature to guide the spatial distribution of the image BEV feature, firstly, the channel dimension of the lidar BEV and the image BEV feature is superimposed, and two consecutive convolutional layers are used to preliminarily fuse the data of the two modalities, and the convolution kernel size is 3x3. After preliminary fusion, the number of feature channels is reduced from to , and the preliminary fusion BEV feature is taken as the query vector Q of the deformable attention module.

[0053] Taking the preliminary fusion BEV feature as the query vector can effectively improve the quality of the generated bird's eye view, while directly using the lidar BEV feature as the query vector will reduce the performance of the algorithm. On the one hand, the lidar BEV feature has the characteristic of sparsity, and using the lidar BEV feature alone as the query vector may result in many image BEV features not being corrected. On the other hand, although the spatial structure of the lidar BEV feature is stable, it lacks rich texture, color and other appearance information in the image. Using it alone as the query vector cannot solve the situation of accurately judging the object boundary and type relying on the appearance feature. Using the preliminary fusion BEV feature as the query vector can make full use of the advantages of both and make full use of these complementary information for feature correction. In addition, the preliminary fusion BEV feature obtains the internal relationship between the lidar BEV feature and the image BEV feature through convolution, and the deformable attention can adaptively adjust the weight of different regions based on this relevance to more reasonably correct the image BEV feature.

[0054] Specifically, the structure of the deformable attention module is as follows: Figure 7As shown in the figure, the image BEV features and the preliminary fused BEV features are used as input. In order to better utilize the preliminary fused BEV features to perform spatial correction on the image BEV features, the preliminary fused BEV features are first linearly projected to obtain the query vector Q. This is then input into the light quantum network to generate an offset, and the image BEV features and the offset are input into the offset function to obtain the preliminary corrected features. The preliminary corrected features are then linearly projected onto the key vector K and the value vector V. Finally, Q, K, and V are simultaneously input into the multi-head attention module to obtain the corrected features. The formula is as follows:

[0055] in, Indicates preliminary corrective features; Represents the BEV feature of the image; Indicates the offset; Indicates the preliminary fusion of BEV features; Indicates sampling of units near the offset point by linear interpolation.

[0056] In this process, the initially fused BEV features serve not only as input to the offset network but also as the query vector Q for the multi-head attention module, resulting in a two-stage correction of the image BEV features. In the first stage, the offset network performs a preliminary position correction based on the initially fused BEV features. In the second stage, the multi-head attention mechanism further optimizes and corrects the image BEV features at the feature association level, achieving a gradual refinement and optimization of the features.

[0057] Furthermore, the embodiment of the present application uses a channel attention module to perform feature self-enhancement on the lidar BEV features, suppressing the influence of the noise channel on the model and selectively enhancing features that are more critical to subsequent tasks. Specifically, the embodiment of the present application uses a variant of SE channel attention. For the lidar BEV features, first its spatial dimension is Perform global average pooling to compress the two-dimensional feature map on each channel into a value, that is, average the features of each channel in the spatial dimension to obtain a value of size The eigenvector of is as follows:

[0058] in, Represents the first The value corresponding to each channel; Indicates the first feature of LiDAR BEV The eigenvalues ​​at the channels.

[0059] For the feature vector after global average pooling, it is processed through a fully connected layer to obtain the final channel attention vector , channel attention Each element of corresponds to the corresponding weight of each channel in the feature map. Finally, Multiplying it with the lidar BEV feature, we get the enhanced lidar BEV feature. The formula is as follows:

[0060] in, Indicates that the LiDAR BEV feature is Characteristics of the location, Indicates enhanced LiDAR BEV features in Characteristics of the location.

[0061] Furthermore, the lidar BEV features and image BEV features are derived from inputs of different modal sensors, each containing unique and complementary information. Although simple convolution operations can fuse the two features, they do not fully exploit the advantages of different modal data; the use of channel attention or spatial attention alone to enhance or suppress the channel dimension or spatial dimension does not take into account the relationship between space and channel. For the spliced ​​BEV features, all spatial positions under the same channel or all channels at the same spatial position are not equally important. Therefore, for the spliced ​​features, such as Figure 8 As shown, the embodiment of the present application introduces a triple attention module to jointly model its spatial and channel dimensions, so that the model can adaptively enhance features of different modalities and different positions for different scenarios. The triple attention mechanism can assign different attention weights to different channels and different spatial positions based on the importance of the features. For example, in the detection of lane lines, the accurate spatial information in the lidar BEV features and the appearance information (such as color, shape, etc.) of the image BEV features should be highlighted. By introducing triple attention, the two different modal features can be better utilized, allowing subsequent modules to better focus on the truly critical information for accurate prediction.

[0062] Specifically, the correction feature and the enhanced lidar BEV feature are first spliced ​​in the channel dimension to obtain the preliminary BEV fusion feature , and adjust the dimensions of the preliminary BEV fusion features to obtain the feature dimensions as Then, average pooling and maximum pooling are performed along the last dimension of each feature map to obtain the feature representation of the feature map in the channel direction, horizontal direction, and vertical direction. Then, the average pooling and maximum pooling results of the features of different dimensions are spliced ​​on the last dimension and processed by two convolution kernels with a size of 3×3 and a Sigmoid activation function to obtain a dimension of Attention weight maps, which represent the spatial dimensions , channel-space and channel-space Attention weights on three different dimensions. Finally, the generated attention weight map is multiplied and dimensionally adjusted with the preliminary BEV fusion feature map, and the dimensions of the three adjusted feature maps are restored to , and add the three elements one by one to get the final output result, that is, the BEV fusion feature. The formula is as follows:

[0063] in, represents the preliminary BEV fusion feature map; Represents the feature map after dimension transformation; Represents the Sigmoid activation function; and Represents the convolution layer of three branches. The horizontal line above the parameter indicates the dimension adjustment of the feature, which restores the feature dimension to .

[0064] Furthermore, in autonomous driving scenarios, bumps can cause the vehicle to exhibit complex and continuous changes. A single moment of image or lidar data only reflects the instantaneous scene information, and this instantaneous information is easily disturbed by bumps, resulting in deviations and inaccuracies in the generated bird's-eye view. In order to solve the problem of deviation and inaccuracy of single-frame bird's-eye view caused by bumps, Figure 9 As shown, this embodiment of the application also proposes a time-series fusion module based on vehicle position information. Using vehicle position change information, it aligns the features of multiple frames of bird's-eye view to the coordinates of the current moment's bird's-eye view, correcting and enhancing the features at the current moment. This solves the problem of feature offset and misalignment at a single moment caused by turbulence. Furthermore, fusing features from multiple frames of bird's-eye view can also resolve issues with occlusion or missing data in a single frame, improving the integrity of the bird's-eye view.

[0065] It can be divided into two parts: feature alignment based on vehicle position information and time series feature fusion. Specifically, for BEV fusion features at different times, to further perform feature fusion, an alignment operation is first required. By changing the vehicle position information, all features at the previous time are converted to the vehicle coordinate system at the current time, so as to eliminate the feature differences caused by the different positions and postures of the vehicle at different times. Assume that the BEV fusion feature at the current time is , the BEV fusion characteristics of the first n moments are , based on the current BEV fusion characteristics and BEV fusion characteristics at time t-1 For example, to Align to First, we need to obtain the coordinates and angles of the vehicle in the world coordinate system at time t and time t-1, and then subtract them to obtain the relative position vector, the direction angle of the relative position vector, and the relative angle change of the vehicle in the world coordinate system at time t and time t-1. At this point, both the position change relationship and the angle change relationship are in the world coordinate system and need to be converted to the current vehicle's own vehicle coordinate system. In order to convert the relative position vector to the own vehicle coordinate system, we need to obtain the angle of the relative position vector in the own vehicle coordinate system at time t. The formula is as follows:

[0066] in, Represents the angle of the relative position vector in the vehicle coordinate system at time t; represents the vehicle angle at time t; Indicates the direction angle of the relative position vector.

[0067] After obtaining the angle of the relative position vector in the ego-vehicle coordinate system at time t, the position offset in the ego-vehicle coordinate system is expressed as follows:

[0068] in, and Respectively represent the offset on the x-axis and y-axis of the vehicle coordinate system; Represents the relative position vector.

[0069] At this point, the relationship between the vehicle coordinate systems at time t and time t-1 is established. The BEV fusion features at time t-1 are aligned to the BEV fusion features at the current time. The formula is as follows:

[0070] in, denotes a rotation transformation on the BEV fusion feature at t-1 time, which is used to adjust the feature direction difference caused by the change of vehicle angle; denotes a translation transformation on the BEV fusion feature at t-1 time, which aims to correct the feature position deviation caused by the movement of the vehicle position.

[0071] It should be noted that, for the aligned feature , the Detach operation needs to be performed on the features of the previous n time points, so that the previous time point features do not participate in the back propagation process in the training process, avoiding unnecessary interference of historical data on the gradient update of the current time point feature. In addition, attention also needs to be paid to whether the previous time point is in the same scene as the current time point. If not, the BEV fusion feature of the current time point is directly taken as the previous BEV fusion feature to participate in subsequent calculation.

[0072] After the alignment operation on the time sequence feature, the features of multiple time points need to be aggregated to realize the deep fusion and enhancement of the time sequence feature. Benefiting from the static characteristics of the map elements such as lanes and pedestrian crossings, it can be considered that they are in the same position in the aligned features at different time points. This provides a basis for the fusion of time sequence features.

[0073] First, the embodiment of the present application performs channel dimension splicing on the already aligned multi-frame BEV fusion features. Assuming that there are n time points of BEV fusion features, and the channel number of each feature is c, then the channel feature number of the spliced feature will become n×c. This splicing method can retain the independent information of each frame and provide rich input for subsequent fusion. Subsequently, the embodiment of the present application uses a convolution method to fuse the spliced features. Specifically, a convolution layer with a convolution kernel size of 3×3 is used to process the spliced features. The purpose of the convolution layer is to deeply fuse the multi-frame time sequence features, and through continuous convolution, normalization and activation operations, the channel dimension of the features is gradually reduced, and finally the enhanced BEV fusion feature is obtained, which is shown in the following formula:

[0074] wherein, denotes the BEV fusion feature of the previous time point after alignment; denotes the BEV fusion feature of the current time point; denotes the superposition of the channel dimension; denotes the convolution operation on the feature with a convolution kernel size of 3×3; denotes the enhanced BEV fusion feature at t time.

[0075] In addition, the enhanced BEV fusion features will not only be used for feature representation at the current moment, but will also participate in the feature enhancement process at subsequent moments. Specifically, at each subsequent moment, the embodiment of the present application will fuse the enhanced BEV fusion features obtained at the previous moment with the enhanced BEV fusion features at the current moment. This mechanism can implicitly introduce longer time series information for the features of the current moment without significantly increasing the computational complexity of the model, so that the effect of feature fusion can be accumulated and optimized over time. In this way, the embodiment of the present application can make better use of historical time series information and further enhance the ability to understand complex scenes. The accumulation of this time series information not only helps to reduce feature deviations and misalignment problems caused by factors such as vehicle bumps, but also improves the efficiency of utilizing time series information in the entire bird's-eye view generation process, ultimately generating a more accurate and stable bird's-eye view.

[0076] Finally, the enhanced BEV fusion features are processed by the semantic segmentation task head in the decoder to generate the bird's-eye view. The decoder typically contains multiple upsampling and convolutional layers to gradually restore the enhanced BEV fusion features to the spatial resolution of the original image. The semantic segmentation task head then generates the final bird's-eye view.

[0077] In this embodiment, the bird's-eye view image is set to a range of [0, 30]m × [-15, 15]m, with a voxel resolution of 0.15m. Within this range, the model uses IoU, CPA, and LPIPS as evaluation metrics to evaluate the performance of the method in this embodiment. The LPIPS metric is primarily used in comparative experiments. To fully verify the effectiveness of the proposed method, this embodiment designed an ablation experiment to verify the effectiveness of each module of the proposed method.

[0078] First, the method proposed in this embodiment utilizes a Multi-Scale Spatial Transformation Module (MSCM), a Feature Alignment and Fusion Module (AFAM), and a Temporal Fusion Module (TFM-EVLI) based on ego vehicle position information to mitigate the impact of turbulence on bird's-eye view image generation. To verify the impact of each module on the model, eight baseline networks were constructed to verify the contribution of each module to the model, and ablation analysis was performed on each of the three modules. Specifically, the following results were used: a) using only the base network without any of the aforementioned modules; b) using the base network and the Multi-Scale Spatial Transformation Module; c) using the base network and the Feature Alignment and Fusion Module; d) using the base network and the Temporal Fusion Module based on ego vehicle position information; e) using the base network, the Multi-Scale Spatial Transformation Module, and the Feature Alignment and Fusion Module; f) using the base network, the Multi-Scale Spatial Transformation Module, and the Temporal Fusion Module based on ego vehicle position information; g) using the base network, the Feature Alignment and Fusion Module, and the Temporal Fusion Module based on ego vehicle position information; and h) using the complete network model. The parameters of the baseline network in the above ablation experiments remain unchanged during training and testing. Divider, Ped, and Boundary in the table represent lane dividers, crosswalks, and lane boundaries, respectively. The results of this ablation experiment are shown in the following table:

[0079] First, let's examine the impact of individual modules on model performance. The results for a), b), c), and d) show that the addition of the multi-scale spatial conversion module, the feature alignment and fusion module, and the ego-vehicle position-based temporal fusion module all improve model performance. Overall, the temporal fusion module achieves the greatest performance improvement, with lane dividers experiencing increases in IoU and CPA by 1.8% and 1.3%, crosswalks by 3.5% and 8.3%, and lane edges by 2.3% and 7.3%, respectively.

[0080] In principle, the time series fusion module based on ego-vehicle position information introduces more additional information than other modules. Furthermore, since the position of map elements in world coordinates does not change, the features provided by this additional information can almost always be considered accurate, provided the sensor extrinsic parameters are accurate. Therefore, the time series fusion module can significantly improve model performance.

[0081] Then the effects of different combinations of these three modules on the model performance are observed. For the combinations e), f), and g) containing two modules, their performance is better than the performance of the corresponding modules introduced separately. Specifically, the performance of the model that combines the multi-scale spatial conversion module and the feature alignment and fusion module is slightly worse, which may be because both modules involve the correction of image feature projections. On the whole, the performance of the model containing the timing module is better, among which the model that combines the feature alignment and fusion module with the timing fusion module has the best comprehensive performance. Finally, by comparing the performance of the model that combines two modules and that that combines three modules, it can be seen that the complete model has the best performance, which effectively proves that there is a synergistic relationship between the three modules designed in the embodiment of the present application.

[0082] In summary, the three modules proposed in this embodiment all improve the performance of the bird's-eye view generation algorithm. For individual modules, the time series fusion module based on ego-vehicle position information achieves better results than the other two modules. For multiple modules, the model combining all three modules achieves the best performance in the bird's-eye view generation task.

[0083] The multi-scale spatial conversion module can be divided into two steps: the first step is to obtain multi-scale image BEV features, and the second step is to fuse the image BEV features at different levels. First, to explore the impact of different scale features on algorithm performance, this embodiment of the application inputs image features at scales of {1 / 16, 1 / 32, 1 / 64} into the multi-scale spatial conversion module and performs a performance test without changing other modules.

[0084] like Figure 10 and Figure 11 As shown, It means that the features of three different scales, 1 / 16, 1 / 32, and 1 / 64, are fused. By observing the experimental results, we can see that when a single scale feature is selected, the feature of scale 1 / 32 can achieve the best performance. The performance of the model with features of scale 1 / 16 or 1 / 64 is relatively poor. This may be because the features of scale 1 / 16 fail to capture enough high-level features, while the features of scale 1 / 64 are too abstract and not conducive to subsequent tasks. At the same time, the fusion of features of three different scales is used. The model achieved the best results in both IoU and CPA, fully demonstrating the effectiveness of multi-scale feature fusion. The multi-scale approach can leverage the advantages of features at each scale and compensate for the shortcomings of features at a single scale, thereby significantly improving the overall performance of the model.

[0085] In addition, in order to explore the impact of different fusion methods on algorithm performance, the embodiment of the present application conducted an ablation analysis on the multi-scale feature fusion method, and set up three cases. a) Feature fusion through the Add+CNN method; b) Feature fusion through the Concat+CNN method; c) Feature fusion through the MBF module proposed in the embodiment of the present application. For the first two methods, the number of convolution layers is set to 2, and the convolution kernel sizes are set to 1×1 and 3×3 respectively. Compared with the first two methods, the fusion module proposed in the embodiment of the present application has better performance. The experimental results are shown in the following table:

[0086] In the feature alignment and fusion module proposed in the embodiment of this application, it is necessary to utilize the spatial stability of the lidar BEV features to perform feature alignment operations on the image BEV features to resolve the possible projection misalignment problem of the image BEV features. To explore the effectiveness of the feature alignment operation, the embodiment of this application set up an ablation analysis, with a total of three cases: a) no feature alignment; b) direct use of the lidar BEV features to correct the position of the image BEV features; c) use of the BEV features after preliminary fusion to correct the position of the image BEV features. The experimental results are shown in the following table:

[0087] Experimental results show that using preliminarily fused features to correct image BEV features can achieve superior results. Compared to the model without feature alignment, the model's IoU for lane dividers, crosswalks, and lane edges increased by 0.8%, 1.5%, and 0.9%, respectively, and CPA increased by 0.9%, 2.8%, and 2%, respectively. Furthermore, directly using lidar BEV features to correct image BEV features resulted in a significant decrease in all metrics except lane edges. This is because direct use of lidar BEV features for correction fails to account for the differences between image and lidar features. Compared to image features, lidar features lack sufficient semantic information, making them difficult to match with image BEV features. In contrast, preliminarily fused features not only integrate the lidar's high-precision spatial information but also fully preserve the rich semantic and visual features of camera images, providing a more accurate and comprehensive basis for correcting image BEV features. Therefore, using fused features can achieve relatively better results.

[0088] In the in-depth study of the bird's-eye view generation model, the temporal fusion module plays a key role in improving the model performance. To this end, the embodiment of the present application conducts a comprehensive ablation experiment specifically on the number of past frames fused by this module, where the frame number 0 means no temporal information is used, and the frame numbers 3 and 5 mean using the information of the past 3 frames or the past 5 frames. The experimental results are as follows: Figure 12 and the following table:

[0089] Comparing the experimental results with 0 and 3 frames, it can be seen that introducing temporal information can significantly improve model performance. The IoU and CPA of lane dividers increased by 3.2% and 1.8% respectively, the IoU and CPA of crosswalks increased by 3.3% and 6.4% respectively, and the IoU and CPA of lane edges increased by 2.4% and 5.1% respectively. All indicators have been significantly improved.

[0090] Furthermore, further observation of the experimental results for fusing the past three frames and the past five frames reveals that, judging by the magnitude of improvement, model performance does not improve significantly with increasing the number of fused frames. This may be related to the data used in the experiments of the present embodiment. The experiments of the present embodiment only used the forward-looking image and the corresponding point cloud information. For constructing a bird's-eye view focused on the vehicle's front, the scene in front of the vehicle changes relatively continuously, and only information from the adjacent moments is critical for the model to judge the state of the environment in front of the vehicle. Past frame information beyond a certain time range is of limited help to the model in judging the current state of the bird's-eye view in front of the vehicle due to the large scene changes. This experimental result indicates that there is a relatively reasonable range of frame numbers to select when using temporal information to improve model performance. For the scenario involved in this experiment, where the bird's-eye view is constructed based on forward-looking images and point cloud information, the information from the past five frames already provides the model with sufficient temporal clues. Further increasing the number of fused frames does not lead to a significant performance improvement for the model. Instead, it increases the model's complexity, causing the model to consume more resources during training and inference, reducing operational efficiency.

[0091] By observation Figure 12 The visualization results show that the bird's-eye view generated with time series information is more complete and performs better than the one without. This is because single-frame information can be significantly affected by abnormal conditions, such as bumps and occlusions. Incorporating time series information can effectively smooth out these anomalies, improving the model's robustness to these conditions.

[0092] In order to explore the impact of multimodal input on the algorithm, the embodiment of this application designed experiments for different modal inputs. According to the different types of input data, they can be divided into three different types, namely: a) camera input only; b) lidar input only; c) camera and lidar input at the same time. It is worth noting that the lack of radar data input will cause the feature alignment and fusion module to fail to operate normally, so the experiment is based on the use of a model that includes a multi-scale spatial conversion module and a temporal fusion module. The experimental results are shown in Figure 2. Figure 13 and the following table:

[0093] According to Table 5 and Figure 13 The visualization of the experimental results clearly shows the impact of different modal inputs on model performance. When using only camera modality data as input, the model achieved the worst results among the three experiments. This may be because although camera data can theoretically provide richer visual features than lidar data, such as the color, shape, and texture of objects, this information plays an important role in the model's understanding of scene content. However, in practical applications, complex and ever-changing scenes pose many challenges to cameras. In complex scenes, image features are more easily affected by factors such as occlusion and lighting. These factors will interfere with the model's extraction and analysis of object features in the image, leading to poor overall performance of the model.

[0094] When using only lidar input, the algorithm's various metrics improved significantly compared to using only camera input. This is likely because lidar data performs well within the specific distance range of 0-30m, with no significant sparsity issues. Furthermore, compared to image data, lidar data can capture depth information about objects. This means that within this distance range, lidar can more densely collect information about the surrounding environment, providing the model with rich and accurate features. Therefore, in the 0-30m bird's-eye view generation task, the model using only lidar data achieved better results than the model using only camera data.

[0095] Finally, the algorithm achieved the best results when using both modal data as input, and even accurately segmented some areas that were not clearly segmented using either single-modal model. This fully demonstrates the complementary nature of camera and lidar data. The rich texture information provided by the camera combined with the precise distance information provided by the lidar provides the algorithm with more comprehensive and accurate features, significantly improving the model's performance on the bird's-eye view generation task.

[0096] In the bird's-eye view generation task, the 2D backbone network is responsible for extracting rich and valuable features from camera images, laying the foundation for accurate bird's-eye view generation. Compared to laser point clouds, images provide more dense and detailed semantic information, such as color and shape. An effective image feature extraction network can provide high-quality features for subsequent steps, thereby improving the model's overall performance in the bird's-eye view generation task.

[0097] This application example uses four different feature extraction networks: ResNet101, EfficientNet-b0, EfficientNet-b3, and Swin-T to extract image features. The experimental results are shown in the following table:

[0098] The table shows that compared to ResNet-101, EfficientNet-b3 achieves better performance with fewer parameters. Meanwhile, Swin-T achieves higher performance with a similar parameter count compared to EfficientNet-b3. This is likely due to Swin-T's Transformer architecture, whose self-attention mechanism is able to capture long-range dependencies in images. Map elements such as lane lines and road edges inherently have global correlations, allowing Swin-T to better understand the semantic information and spatial relationships between bird's-eye view elements.

[0099] In summary, since the Swin-T backbone network performs outstandingly in the bird's-eye view generation task, the model selects Swin-T as the backbone network for extracting image features.

[0100] In order to better verify the effectiveness of the bird's-eye view generation method based on multi-scale feature transformation and temporal context proposed in the embodiment of the present application, the embodiment of the present application conducted a comparative experiment on the NuScenes dataset. First, a comparative experiment of the overall algorithm performance was carried out to verify the performance advantage of the algorithm. Then, comparative experiments were carried out under three different road conditions: flat road, slightly bumpy road and bumpy road, to verify the robustness of the algorithm. In these experiments, not only the model proposed in the embodiment of the present application was thoroughly tested, but also fully compared with other bird's-eye view generation algorithms to evaluate the pros and cons of different algorithms. LSS, PointPillars, HDMapNet, BEVFusion, BevFusion, and SuperFusion were selected for the comparative test.

[0101] LSS achieves a change in perspective from perspective to bird's-eye view and performs specific detection or segmentation tasks through four steps: feature extraction, "lift" (mapping image feature points to a 3D camera coordinate system to form a point cloud based on a depth probability distribution), "splat" (converting the camera point cloud to the vehicle coordinate system and "flattening" it into a BEV feature map), and "shoot" (performing specific tasks such as bird's-eye view generation). PointPillars is a point cloud processing network that balances speed and performance. By generating a pseudo-2D map, it significantly reduces the performance overhead of traditional point cloud networks. HDMapNet focuses on bird's-eye view generation, designing three task heads: semantic segmentation, instance vector prediction, and direction prediction. It then generates an instanced bird's-eye view through post-processing. BEVFusion, aiming to improve algorithm robustness, provides a unified feature space for BEV feature fusion, enabling the model to maintain good performance even when a single sensor fails. BEVFusion innovates in feature fusion, unifying multimodal features from cameras and lidar into a shared bird's-eye view representation space while preserving both geometric and semantic information. Optimized BEV pooling operations address the efficiency bottleneck of view transformation, enabling support for diverse perception tasks. SuperFusion fuses features at multiple levels and generates a dense depth map through bilinear interpolation, providing more effective depth supervision for depth estimation, improving depth prediction accuracy, and significantly enhancing the performance of the bird's-eye view generation algorithm.

[0102] Table 7 is the test results of the model on the NuScenes dataset, where Mod represents the modality used by the model, C represents the camera modality, L represents the laser point cloud modality, and C+L represents the simultaneous use of the camera and laser point cloud. According to the test results, for the two different indicators of IOU and CPA for three different targets, namely lane dividers, crosswalks, and road boundaries, the algorithm proposed in the embodiment of the present application has achieved the best results in all indicators except the CPA indicator of lane dividers. Specifically, the IoU and CPA corresponding to the crosswalk increased by 1.7% and 0.5% respectively, the IoU corresponding to the lane divider increased by 0.5%, and the IoU and CPA corresponding to the lane edge increased by 1.3% and 0.9% respectively. At the same time, observing the LPIPS indicator, it can be seen that the results generated by the algorithm proposed in the embodiment of the present application also have the best performance for the visual perception system, as shown in the following table:

[0103] Furthermore, a detailed analysis of various metrics reveals that SuperFusion achieved suboptimal results. This demonstrates, on the one hand, that providing explicit depth supervision can effectively improve algorithm performance, while inaccurate depth estimation can severely disrupt algorithm performance. It also demonstrates that the potential connections between multimodal data remain to be explored and hold significant potential. Furthermore, experimental results indicate that algorithms that utilize both cameras and laser point clouds generally perform better than algorithms that utilize only a single sensor. This demonstrates that a single sensor often has limitations, and that combining camera and laser point cloud features allows the model to have a more comprehensive and accurate understanding of the target, resulting in better performance across various metrics.

[0104] To more intuitively demonstrate the advantages of the bird's-eye view generation method based on multi-scale feature transformation and temporal context in addressing road bumps, this embodiment compares the performance of the method proposed in this embodiment with other algorithms in different scenarios from the NuScenes dataset. Specifically, Z-axis acceleration is used as the basis for evaluating whether the road surface is bumpy. When the vehicle is stationary, the Z-axis acceleration is approximately 9.8 m / s. This embodiment carefully classifies road conditions based on the severity of the Z-axis acceleration curve, dividing them into three different types: flat, slightly bumpy, and bumpy. If the Z-axis acceleration fluctuation range is between 9.5 m / s and 10.5 m / s, the road condition is flat; if the Z-axis acceleration fluctuation range is between 9.0 m / s and 10.5 m / s, the road condition is slightly bumpy; if the Z-axis acceleration fluctuation exceeds this range and the curve oscillates strongly, the road condition is bumpy. For each type of road condition, 20 scenarios were selected, totaling 800 frames of data, for the comparison experiment.

[0105] The experimental results of the algorithm in the flat road scene are shown in Table 8, and the visualization results are shown in Figure 14 , Figure 15 As shown:

[0106] The experimental results show that the LSS and PointPillars results exhibit different trends from those in Table 7. In Table 7, PointPillars outperforms LSS, while in Table 8, LSS outperforms PointPillars in all metrics except lane edges. This demonstrates that in favorable driving conditions, camera modality data provides richer information. In adverse driving conditions, such as bumpy conditions, laser point cloud modality data is less susceptible to interference than camera modality data, providing more reliable information for generating bird's-eye views.

[0107] In addition, under flat road conditions, after comparing the performance of the algorithm proposed in the embodiment of the present application with that of many other algorithms, it was found that although the performance of the algorithm proposed in the embodiment of the present application was similar to that of SuperFusion and did not show a clear advantage, it was still significantly better than other algorithms. Specifically, in the recognition of crosswalks, the algorithm proposed in the embodiment of the present application achieved the best results in terms of IoU and CPA indicators. At the same time, in terms of IoU, CPA indicators of lane dividers and lane edges and the overall LPIPS indicator, the algorithm proposed in the embodiment of the present application achieved suboptimal results, which fully demonstrated that the method proposed in the embodiment of the present application also has excellent performance under flat road conditions.

[0108] It can be seen from the visualization results that under flat road conditions, most algorithms can achieve good results. Compared with other algorithms, the algorithm proposed in the embodiment of the present application has better overall performance. It is worth noting that other algorithms often lose information when dealing with distant and obscured areas. The algorithm proposed in the embodiment of the present application can achieve better results when processing this part of the area, which may benefit from the multi-scale spatial conversion module. When facing distant and obscured areas, large-scale features can capture the overall outline and approximate position, while small-scale features focus on resolving details. The two complement each other, allowing the model to reasonably predict the area with limited information, thereby effectively avoiding the information loss problem of the bird's-eye view generation result.

[0109] The experimental results under mild road bumps are shown in Table 9, and the visualization results are shown in Figure 16 、 Figure 17 As shown:

[0110] Comparing Tables 8 and 9, we can see that all algorithms experience a certain degree of performance degradation on slightly bumpy roads compared to flat roads. This phenomenon may be due to slightly bumpy roads causing a certain degree of deviation in the data collected by the vehicle's cameras and lidar, which in turn affects the algorithm's processing and analysis of the data, ultimately resulting in a decrease in algorithm performance.

[0111] Although the performance of all algorithms is reduced in the light bump road, SuperFusion and the algorithm proposed in the embodiment of the application still achieves suboptimal and optimal results. This fully shows that the two algorithms have stronger adaptability and stability than other algorithms in dealing with light bump roads. It is worth noting that compared with SuperFusion, the performance of the algorithm proposed in the embodiment of the application has a smaller decline. Specifically, the IoU indicators of the road separation line and pedestrian crossing of SuperFusion decrease by 5.3% and 10.8% respectively, the CPA indicators decrease by 6.1% and 14.4% respectively, and the LPIPS indicator increases by 0.0193, while the IoU indicators of the algorithm proposed in the embodiment of the application decrease by 4.3% and 10.1% respectively, the CPA indicators decrease by 4.8% and 12.6% respectively, and the LPIPS increases by 0.0098. In addition, the IoU and CPA indicators of the road edge are basically not affected and do not decrease. This may be because the road edge information mainly depends on the laser point cloud input, and the light bump has little effect on the laser radar, so the corresponding indicators do not decrease. At the same time, the visualization results also show that the bird's eye view generated by the algorithm proposed in the embodiment of the application has better effect.

[0112] The experimental results of the algorithm in the bump road are shown in Table 10, and the visualization results are shown in Figure 18 、 Figure 19

[0113] From the experimental results, it can be known that in the bump road, the algorithm proposed in the embodiment of the application shows obvious advantages compared with other algorithms. Specifically, compared with the suboptimal algorithm, the IoU and CPA indicators of the road separation line of the algorithm proposed in the embodiment of the application are improved by 2.5% and 2.9% respectively, the corresponding indicators of the pedestrian crossing are improved by 1.4% and 1.8% respectively, the corresponding indicators of the lane edge are improved by 1.8% and 0.7% respectively, and the LPIPS indicator is reduced by 0.0064. This series of data shows that in the bump road, the algorithm in the embodiment of the application has stronger stability and accuracy than other algorithms.

[0114] From the visualization results Figure 18 、 Figure 19 ​In the figure, we can see the advantages of the algorithm of the embodiment of the present application more intuitively. The lines of road dividers, crosswalks and lane edges in the bird's-eye view generated by the algorithm of the embodiment of the present application are clearer, continuous and accurate, and have a higher degree of match with the actual scene. However, in bird's-eye views generated by other algorithms, these elements will appear blurry, broken or positionally deviated under bumpy road conditions. These experimental results and visual comparisons fully and powerfully illustrate that the method proposed in the embodiment of the present application has unique advantages in dealing with bumpy road conditions, and can effectively solve the problem of reduced quality of bird's-eye view generation due to bumpy road conditions.

[0115] In summary, the bird's-eye view generation model proposed in the embodiment of the present application shows significant advantages in comparison with other models, and demonstrates excellent performance and strong adaptability in three different road conditions: flat road, slightly bumpy road, and bumpy road, proving the effectiveness and robustness of the algorithm.

[0116] It should be noted that, in the present application, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0117] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for generating a bird's-eye view image based on multi-scale feature transformation and temporal context, characterized in that: The method comprises: Acquiring raw modal data, where the raw modal data includes image data and lidar data; The image data is processed by an image backbone network to obtain forward-view image features, and the forward-view image features are subjected to multi-scale spatial transformation to obtain image BEV features; the lidar data is processed by a radar backbone network to obtain lidar features, and the lidar features are compressed along the Z axis to obtain lidar BEV features; The image BEV features and the lidar BEV features are aligned and fused to obtain BEV fusion features. The BEV fusion features of the past n moments are introduced. The BEV fusion features of the current moment are enhanced by the BEV fusion features of the past n moments to obtain enhanced BEV fusion features. The enhanced BEV fusion features are processed by the semantic segmentation task head in the decoder to obtain a bird's-eye view generation result.

2. The method for generating a bird's-eye view image based on multi-scale feature transformation and temporal context according to claim 1, wherein: The method comprises: Swin-T is used as the image backbone network to process the image data and output features of four different scales , the channel dimensions corresponding to the features of the four different scales are ,right To unify the dimensions, use bilinear interpolation Up-sample and convolute it with two layers of convolution and The fusion is performed to obtain the fused image features, and the fused image features are transformed into multi-scale space to obtain the image BEV features.

3. The method for generating a bird's-eye view image based on multi-scale feature transformation and temporal context according to claim 1 or 2, wherein: The method comprises: PointPillars is used as the radar backbone network to perform columnar feature processing on the lidar data to obtain pseudo-2D image features. The pseudo-2D image features are input into a 2D convolutional network to obtain features at three different scales. The three features at different scales are upsampled to unify the spatial dimensions and then spliced ​​in the channel dimension to obtain the lidar BEV features.

4. The method for generating a bird's-eye view image based on multi-scale feature transformation and temporal context according to claim 1 or 2, wherein: The method comprises: Performing multi-scale spatial transformation on the front view image features by constructing a multi-scale spatial transformation module, wherein the multi-scale spatial transformation module includes an encoder, a view relationship module, a multi-scale feature fusion module and a decoder connected in sequence; The encoder is composed of two residual convolution modules connected together, each of which performs two convolution operations and two times downsampling operations on the front view image features to obtain downsampled features; The view relationship module consists of two fully connected layers and a convolutional layer. The downsampled features are mapped by the two fully connected layers, and the channel dimension is reduced by a convolutional layer to obtain mapping features at different levels. The multi-scale feature fusion module consists of an offset prediction branch and a splicing layer. The offset prediction branch also includes two convolutional layers. The offset prediction branch splices the mapping features of different levels in the channel dimension, and the two convolutional layers perform offset prediction to obtain the offsets corresponding to the mapping features of different levels. Based on the offsets, the mapping features of the different levels are sampled by interpolation and connected in the channel dimension through the splicing layer to obtain the preliminary image BEV features. The decoder expands the spatial dimension of the preliminary image BEV features by using two bilinear interpolation series residual convolution blocks. The residual convolution block consists of two series-connected 3×3 convolution layer trunks and a 1×1 convolution layer branch to obtain the image BEV features.

5. The method for generating a bird's-eye view image based on multi-scale feature transformation and temporal context according to claim 4, wherein: The method comprises: A random dropout layer is inserted between the two serially connected 3×3 convolutional layer backbones of the decoder, through which 20% of the neurons are randomly set to 0.

6. The method for generating a bird's-eye view image based on multi-scale feature transformation and temporal context according to claim 1, wherein: The method comprises: Performing feature alignment and fusion on the image BEV features and the lidar BEV features by constructing a feature alignment and fusion module, wherein the feature alignment and fusion module includes a deformable attention module, a first convolutional layer, and a channel attention module connected in parallel, and a triple attention module and a second convolutional layer connected in series therewith; The image BEV features and the lidar BEV features are preliminarily concatenated and input into the first convolutional layer to obtain a preliminary fused BEV feature. The image BEV features and the preliminarily fused BEV features are then input into the deformable attention module together for channel dimension superposition to obtain a corrected feature. Input the lidar BEV feature into the channel attention module to obtain an enhanced lidar BEV feature; then concatenate the corrected feature and the enhanced lidar BEV feature channel dimension to obtain a preliminary BEV fusion feature; The preliminary BEV fusion features are sequentially input into the triple attention module and the second convolutional layer, and the spatial and channel dimensions are modeled simultaneously to obtain the BEV fusion features.

7. The method for generating a bird's-eye view image based on multi-scale feature transformation and temporal context according to claim 1 or 6, wherein: The method comprises: The BEV fusion features of the past n moments and the BEV fusion features of the current moment are feature aligned, the BEV fusion features of the past n moments are converted to the vehicle coordinate system of the current moment according to the change of the vehicle position information, and the BEV fusion features of the past n moments converted to the vehicle coordinate system of the current moment and the BEV fusion features of the current moment are feature aggregated to obtain enhanced BEV fusion features.

8. The method for generating a bird's-eye view image based on multi-scale feature transformation and temporal context according to claim 7, wherein: The method comprises: Before aligning the BEV fusion features at the past n moments with the BEV fusion features at the current moment, a Detach operation is performed on the BEV fusion features at the past n moments.

Images