Bird's eye view generation method based on multi-scale feature transformation and temporal context

Abstract

The application discloses an aerial view generation method based on multi-scale feature transformation and time sequence context, and relates to the technical field of map generation. The method fully utilizes the complementarity of image and laser radar data by fusing the image and the laser radar data, improves the accuracy and robustness of aerial view generation, and can still remain stable under bad weather; a multi-scale space conversion module extracts different scale features, enhances the feature expression capability, and makes the aerial view clearer and more accurate; time sequence information is introduced, past time features are used to enhance current features, and dynamic perception capability is improved; an advanced backbone network and a feature alignment and fusion module are adopted, so that high efficiency and flexibility are ensured; and specific network structures such as Swin-T, PointPillars and random inactivation layers are applied, so that the accuracy and generalization capability are further improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of map generation, and particularly relates to a bird's eye view generation method based on multi-scale feature transformation and time sequence context. BACKGROUND

[0002] As a cutting-edge hotspot in today's technology field, the overall architecture of automatic driving technology can be clearly divided into three core parts: perception, decision-making, and execution. Among them, the perception part plays a crucial role, as it is like the "eyes" and "ears" of the automatic driving system. Through efficient fusion of information from multiple sensors, accurate environment perception results are generated. These perception results provide a solid data support for the subsequent decision-making and execution parts, and are the basis for the safe and stable operation of the automatic driving system.

[0003] The bird's eye view (BEV) generation algorithm is a key component of the automatic driving perception algorithm. The core goal of this algorithm is to deeply analyze the surrounding environment information from multi-sensor data and generate a semantic map in real time. The generated map covers a wide range of elements, such as lane boundaries, lane lines, pedestrian crossings, etc. These elements provide accurate road information and location information for the vehicle, allowing it to clearly understand the traffic environment it is in and make reasonable driving decisions.

[0004] Traditional map perception schemes mainly rely on the analysis of perspective view images, but this approach has many drawbacks. On the one hand, the perspective view has occlusion problems, which can result in some important environmental information being inaccessible. On the other hand, the results obtained from the perspective view are in the pixel plane, which deviates from the spatial relationship in the real world, and complex post-processing is needed to establish a connection with the real world, which undoubtedly increases the difficulty of subsequent decision-making tasks.

[0005] Existing methods do not fully consider the impact of bumps on bird's eye view generation tasks. In actual driving, vehicles inevitably encounter bumpy road conditions. Bumps can cause two problems: on the one hand, bumps can cause changes in camera extrinsic parameters. Camera extrinsic parameters describe the position and attitude of the camera in the world coordinate system, and changes in extrinsic parameters can cause misalignment of image features to BEV features, resulting in a deviation between the generated bird's eye view and the actual environment. On the other hand, since bumps have different effects on different modal data, this can destroy the consistency between different modal features, causing misalignment of features during feature fusion. Under the combined influence of these two problems, existing bird's eye view generation algorithms have the problem of low generation quality in bumpy scenarios, which cannot meet the high-precision perception needs of the automatic driving system in complex road conditions. SUMMARY

[0006] The purpose of the present application is to provide a bird's eye view generation method based on multi-scale feature transformation and temporal context, which solves the problems of feature extraction, alignment, fusion and temporal enhancement in bird's eye view generation by fusing image data and lidar data and using the technical means of multi-scale feature transformation and temporal context information, thereby improving the accuracy and robustness of bird's eye view generation.

[0007] In order to achieve the above purpose, the embodiment of the present application provides a bird's eye view generation method based on multi-scale feature transformation and temporal context, comprising:

[0008] Obtaining original modal data, the original modal data including image data and lidar data;

[0009] Processing the image data through an image backbone network to obtain front view image features, performing multi-scale spatial conversion on the front view image features to obtain image BEV features; processing the lidar data through a radar backbone network to obtain lidar features, compressing the lidar features along the Z axis to obtain lidar BEV features;

[0010] Aligning and fusing the image BEV features and the lidar BEV features to obtain BEV fusion features, introducing BEV fusion features of the past n time points, enhancing the BEV fusion features of the current time point through the BEV fusion features of the past n time points to obtain enhanced BEV fusion features, processing the enhanced BEV fusion features through a semantic segmentation task head in a decoder to obtain a bird's eye view generation result.

[0011] According to the above method of the embodiment of the present application, the following additional technical features can also be provided:

[0012] Further, the Swin-T is used as the image backbone network to process the image data, and four different scale features are output The channel dimension numbers corresponding to the four different scale features are The dimensions of and are unified, the bilinear interpolation is used to up-sample , and the two layers of convolution are used to fuse and to obtain the fused image features, and the multi-scale spatial conversion is performed on the fused image features to obtain the image BEV features.

[0013] Further, PointPillars is adopted as the radar backbone network, columnar features of the lidar radar data are processed to obtain pseudo 2D image features, the pseudo 2D image features are input into a 2D convolution network to obtain three features of different scales, the three features of different scales are up-sampled, the spatial dimensions are unified, and the channel dimensions are spliced to obtain lidar BEV features.

[0014] Further, a multi-scale spatial conversion module is constructed to perform multi-scale spatial conversion on the front-view image features, the multi-scale spatial conversion module comprises an encoder, a view relationship module, a multi-scale feature fusion module and a decoder connected in sequence;

[0015] The encoder is composed of two residual convolution modules connected in sequence, each residual convolution module performs twice convolution operation and twice down-sampling operation on the front-view image features to obtain down-sampled features.

[0016] The view relationship module is composed of two fully connected layers and a convolution layer connected in sequence, the two fully connected layers are used to map the down-sampled features, and the convolution layer is used to reduce the channel dimension to obtain mapping features of different levels.

[0017] The multi-scale feature fusion module is composed of an offset prediction branch and a splicing layer connected in sequence, the offset prediction branch further comprises two convolution layers, the offset prediction branch is used to splice the mapping features of different levels in the channel dimension, and the two convolution layers are used to predict the offset to obtain the offset corresponding to the mapping features of different levels; based on the offset, the mapping features of different levels are sampled by interpolation, and the splicing layer is used to connect the channel dimensions to obtain preliminary image BEV features.

[0018] The decoder is used to expand the spatial dimension of the preliminary image BEV features by twice bilinear interpolation in series with a residual convolution block, the residual convolution block is composed of two 3x3 convolution layer trunks connected in series and a 1x1 convolution layer branch, and the image BEV features are obtained.

[0019] Further, a random inactivation layer is inserted between the two 3x3 convolution layer trunks of the decoder, and 20% of the neurons are randomly set to 0 by the random inactivation layer.

[0020] Further, a feature alignment and fusion module is constructed to perform feature alignment and fusion on the image BEV features and the lidar BEV features, the feature alignment and fusion module comprises a deformable attention module, a first convolution layer and a channel attention module connected in parallel, and a triple attention module and a second convolution layer connected in series therewith.

[0021] The image BEV feature and the lidar BEV feature are preliminarily spliced, and a preliminary fusion BEV feature is obtained by inputting the preliminary fusion BEV feature into a first convolutional layer; the image BEV feature and the preliminary fusion BEV feature are input into a deformable attention module, channel dimensions are superposed, and a correction feature is obtained;

[0022] The lidar BEV feature is input into a channel attention module to obtain an enhanced lidar BEV feature; the correction feature and the enhanced lidar BEV feature are spliced in the channel dimension to obtain a preliminary BEV fusion feature;

[0023] The preliminary BEV fusion feature is sequentially input into a triple attention module and a second convolutional layer, and spatial and channel dimensions are modeled to obtain a BEV fusion feature.

[0024] Further, the BEV fusion features of the past n time points and the BEV fusion feature of the current time point are aligned, the BEV fusion features of the past n time points are converted to the vehicle coordinate system at the current time point through the change of the vehicle position information, and the BEV fusion features of the past n time points converted to the vehicle coordinate system at the current time point and the BEV fusion feature of the current time point are aggregated to obtain an enhanced BEV fusion feature.

[0025] Further, before the BEV fusion features of the past n time points and the BEV fusion feature of the current time point are aligned, the BEV fusion features of the past n time points are subjected to a detach operation.

[0026] Compared with the prior art, the bird's eye view generation method based on multi-scale feature transformation and time sequence context provided by the embodiments of the present application has the following beneficial technical effects:

[0027] The embodiments of the present application fuse image data and lidar data, fully utilize the complementarity of the two modal data, and improve the accuracy of bird's eye view generation; the image data provides rich visual information, while the lidar data provides accurate distance and position information, and the combination of the two can more comprehensively describe the environment around the vehicle; multi-modal data fusion makes the method have stronger robustness in the face of different environmental conditions such as weather and light; for example, in severe weather conditions, image data may be severely affected, but lidar data can still provide reliable distance information, thereby ensuring the stability of bird's eye view generation.

[0028] The embodiment of the application can convert the front-view image features through a multi-scale space conversion module, which can extract feature information of different scales; these feature information contains detailed information and global information of the environment around the vehicle, which helps to more comprehensively understand the environment around the vehicle; the multi-scale feature conversion can enhance the expression ability of the features, so that the generated bird's eye view is more accurate and clear; by fusing feature information of different scales, the complexity and diversity of the environment around the vehicle can be better captured.

[0029] The embodiment of the application can enhance the BEV fusion features of the current time by introducing the BEV fusion features of the past n times, fully utilize the time sequence information to improve the accuracy of the bird's eye view generation; the time sequence information contains dynamic change information of the environment around the vehicle, which helps to more accurately predict the environment around the vehicle at future time; the time sequence context enhancement makes the method have stronger dynamic perception ability; by fusing feature information of past time, the dynamic change of the environment around the vehicle can be better understood, so that a more accurate and reliable bird's eye view can be generated.

[0030] The embodiment of the application adopts advanced backbone networks and feature alignment and fusion modules, which are efficient; these network structures can quickly process a large amount of data and extract useful feature information, thereby ensuring the real-time performance of the bird's eye view generation; the multi-scale space conversion module and the feature alignment and fusion module are flexible, by adjusting the parameters and structures of these modules, different application scenarios and requirements can be adapted, thereby improving the universality and scalability of the method.

[0031] The embodiment of the application adopts Swin-T as the image backbone network to output four different scale features, and through dimension unification and fusion, more rich image BEV features are obtained, which helps to improve the accuracy and clarity of the bird's eye view generation; PointPillars is adopted as the radar backbone network to process columnar features of lidar data and obtain pseudo-2D image features, which helps to better fuse lidar data and image data, and improve the accuracy of bird's eye view generation; inserting a random inactivation layer between the two serial 3x3 convolution layers of the decoder can prevent overfitting and improve the generalization ability of the model, which helps to ensure the stability and reliability of the method on different data sets. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The flowchart of the bird's eye view generation method based on multi-scale feature transformation and time sequence context of the embodiment of the application is shown;

[0033] Figure 2 The model structure diagram of the bird's eye view generation method based on multi-scale feature transformation and time sequence context of the embodiment of the application is shown;

[0034] Figure 3 A structural diagram of a multi-scale space conversion module of an embodiment of the present application is shown.

[0035] Figure 4 A structural diagram of a multi-scale feature fusion module of an embodiment of the present application is shown.

[0036] Figure 5 A structural diagram of a decoder of an embodiment of the present application is shown.

[0037] Figure 6 A structural diagram of a feature alignment and fusion module of an embodiment of the present application is shown.

[0038] Figure 7 A structural diagram of a deformable attention module of an embodiment of the present application is shown.

[0039] Figure 8 A structural diagram of a triple attention module of an embodiment of the present application is shown.

[0040] Figure 9 A structural diagram of a timing fusion module based on self-vehicle position information of an embodiment of the present application is shown.

[0041] Figure 10 An IoU index comparison diagram of different scale features of an embodiment of the present application is shown.

[0042] Figure 11 A CPA index comparison diagram of different scale features of an embodiment of the present application is shown.

[0043] Figure 12 A diagram showing the effect of a timing module fusing different frame numbers of an embodiment of the present application is shown.

[0044] Figure 13 A diagram showing the generation results of different modal bird's eye views of an embodiment of the present application is shown.

[0045] Figure 14 A first comparison diagram of bird's eye view results under flat road conditions of an embodiment of the present application is shown.

[0046] Figure 15 A second comparison diagram of bird's eye view results under flat road conditions of an embodiment of the present application is shown.

[0047] Figure 16 A first comparison diagram of bird's eye view results under light jolt road conditions of an embodiment of the present application is shown.

[0048] Figure 17 A second comparison diagram of bird's eye view results under light jolt road conditions of an embodiment of the present application is shown.

[0049] Figure 18A first comparison chart showing the bird's eye view result of the bump road condition of the embodiment of the present application is shown.

[0050] Figure 19 A second comparison chart showing the bird's eye view result of the bump road condition of the embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, in order to facilitate the description, only the parts related to the present application are shown in the drawings, not all the structures. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0052] The terms "comprising" and "having" and any variations thereof in the present application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to these processes, methods, products or devices.

[0053] In the present application, the phrase "embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment to other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described in the present application can be combined with other embodiments.

[0054] As Figure 1 The embodiment of the present application provides a bird's eye view generation method based on multi-scale feature transformation and time sequence context, which comprises the following steps:

[0055] Step 101, obtaining original modal data, the original modal data including image data and laser radar data.

[0056] In this step, obtaining the original modal data is the basis of the entire bird's eye view generation method based on multi-scale feature transformation and time sequence context. This step ensures that there is enough and accurate data source in the subsequent processing flow. Specifically, the original modal data mainly includes image data and laser radar data, and these two kinds of data play a crucial role in the field of autonomous driving, intelligent transportation, etc.

[0057] Image data is usually captured by on-board cameras, which provide visual information about the environment surrounding the vehicle. In the bird's eye view generation process, image data is used to extract front-view image features, which are then transformed in a multi-scale space to generate image BEV features. The quality, resolution, and capture angle of image data directly affect the accuracy of subsequent feature extraction and BEV feature generation.

[0058] Lidar data is acquired by lidar sensors, which provide precise three-dimensional information about the environment surrounding the vehicle. Compared with image data, lidar data has higher precision and reliability in distance measurement, object recognition, and environment modeling. In the bird's eye view generation process, lidar data is used to extract lidar features, which are then compressed and transformed to generate lidar BEV features.

[0059] Acquiring high-quality and accurate raw modality data is crucial to ensure the effectiveness of the bird's eye view generation method. In practical applications, it is necessary to ensure the normal operation of cameras and lidar sensors, as well as the real-time transmission and storage of data.

[0060] In step 102, image data is processed by an image backbone network to obtain front-view image features, which are then transformed in a multi-scale space to obtain image BEV features. Lidar data is processed by a radar backbone network to obtain lidar features, which are then compressed along the Z-axis to obtain lidar BEV features.

[0061] As shown in Figure 2 , the embodiment of the present application adopts Swin-T as the image backbone network, and the input image size is 256x704. Swin-T is an advanced architecture based on Transformer, which is particularly suitable for processing image data. Swin-T can effectively capture long-range dependencies in images while maintaining computational efficiency. After processing image data, Swin-T outputs four features of different scales , with channel dimensions of , respectively. These features capture different levels of information in the image, from low-level textures to high-level semantics. Low-level features mainly contain visual features of the image (color, texture, etc.), while high-level features mainly contain semantic features of objects. Compared with visual features, semantic features are more important for the bird's eye view generation task. Therefore, higher-level features are selected as the input of the multi-scale space transformation module.

[0062] To perform multi-scale space transformation, the dimensions of and need to be unified. Specifically, bilinear interpolation is used to upsample to make its spatial dimensions consistent with Matching. Then, using two convolutional layers... and The images are then fused to obtain fused image features. This fusion method helps retain useful information from features at different scales while eliminating problems caused by scale differences.

[0063] The fused image features are fed into a multi-scale spatial transformation module to generate image BEV features. In this embodiment, the size of the image BEV feature map is 200×200, which is determined by the set perception range and voxel size.

[0064] After processing by the image backbone network, the semantic features of the image data have been extracted. For the bird's-eye view generation task, spatial transformation of the foreground image features is still required to map them to the bird's-eye view space. However, road bumps can disrupt the existing mapping relationship, causing deviations in the mapping from the foreground view to the bird's-eye view. Therefore, this application's embodiments design a multi-scale spatial transformation module to construct multi-level mapping relationships for multi-scale features, thereby enhancing the stability and accuracy of the entire mapping process. Figure 3 As shown, this module includes an encoder, a view relationship module, a multi-scale feature fusion module, and a decoder.

[0065] The encoder consists of two residual convolutional modules. Each module performs two convolution operations and a double downsampling operation on the input features, progressively extracting and compressing the features. The encoder's role is to further downsample the foreground image features and obtain high-level image features at multiple scales.

[0066] The View Relationship Module (VRM) is essentially a multilayer perceptron, consisting of two fully connected layers and one convolutional layer. The VRM learns the mapping relationship between the front view and the bird's-eye view through the two fully connected layers, and then uses the subsequent convolutional layer to reduce the dimensionality of the features across different channels, unifying the mapping results of image features at different levels. Through the VRM, the model learns the relationship between any two points in the front view and the bird's-eye view. The fully connected and convolutional layers map and reduce the dimensionality of downsampled features, obtaining mapping features at different levels. These features capture information from different viewpoints and scales in the image, representing the mapping relationship between the bird's-eye view and the front view image features, as shown in the following formula:

[0067]

[0068] in, Indicates the first Layer front view image features; Indicates the first Mapping features at different levels of the layer; Indicates the first layer front-view image features and the first corresponding relationship between mapping features of different levels of layers; represent spatial feature dimensions.

[0069] After obtaining mapping features of different levels, fusion operation is needed to give full play to the advantages of multi-scale features and eliminate projection bias. Since the VRM modules of front-view image features of different scales are different, the mapping relationships learned by them are also different. Although this multi-scale mapping method learns more complex feature mapping relationships, when fusing mapping features of different levels, if only the channel dimension stacking method is used, the synergistic effect of multi-scale features cannot be fully played, and even the model performance may be reduced due to misalignment of features between mapping features of different levels.

[0070] Therefore, as Figure 4 shown, the embodiment of the present application proposes a multi-scale feature fusion module (MBF), which will perform alignment operation between features before fusing mapping features of different levels, and make fine adjustments in spatial dimensions and semantic levels according to the characteristics of features of each level.

[0071] Specifically, the embodiment of the present application first sets an offset prediction branch, which takes as input, concatenates the channel dimensions of mapping features of different levels, and predicts the offset through two convolutional layers to obtain the offset of mapping features of different levels, where The two channel dimensions respectively represent the offset of feature pixels in two different spatial dimensions. After obtaining the offset , alignment operation needs to be performed on mapping features of different levels. For convenience of description, the embodiment of the present application uses to represent the feature to be aligned, and the alignment operation of the feature to be aligned is represented, as shown in the following formula:

[0072]

[0073] This process can be understood as sampling the position of the feature to be aligned by interpolation, where represents the image BEV feature after alignment operation; represents the offset of the feature in the corresponding dimension of ; and represents the offset of the feature in the corresponding dimension of .

[0074] Finally, the image BEV features after the alignment operation need to be connected in the channel dimension through concatenation to obtain preliminary image BEV features, as shown in the following formula:

[0075]

[0076] wherein, denotes the preliminary image BEV features; denotes the connection in the channel dimension, is the alignment operation. After processing by the MBF module, the bird's eye view features of different scales are adjusted in space and semantics, providing a basis for their full fusion.

[0077] The decoder is mainly used for expanding the preliminary image BEV features in the spatial dimension to finally obtain the image BEV features. The decoder adopts the way of twice bilinear interpolation in series with residual convolution blocks to gradually expand the size of the preliminary image BEV features from (18x18) to (200x200). Linear interpolation is simply to select the original pixel value closest to the target pixel position. It has no learnable parameters and cannot adaptively adjust according to the characteristics of the data, so it cannot improve the semantic expression ability of the features. This way only gets a transitional smooth feature map without further mining and fusion of the features, and has poor flexibility. Therefore, the decoder designed in the embodiments of the present application combines the advantages of reasonable pixel prediction of the former and dynamic learning and adjustment of the features of the latter by the way of bilinear interpolation in series with residual convolution, so that the model has good adaptability to different input feature maps, and the robustness of the features is improved.

[0078] Specifically, as shown in Figure 5 , the main part of the residual convolution block in the decoder is composed of two 3x3 convolution layers in series, and the residual branch is processed by a 1x1 convolution. In addition, the embodiments of the present application also insert a random inactivation layer between the two 3x3 convolution layer main parts of the decoder to perform a dropout operation, randomly setting 20% of the neurons to 0. After randomly discarding the neurons, the input features received by the subsequent convolution layers will change, which prompts the convolution layers to learn different feature combinations and more diverse feature representations, so that the features learned by the network are more rich and diverse, and the understanding and representation ability of the entire model for data is improved. In addition, the random inactivation operation makes the structure of the network dynamically change every time it is trained, which is equivalent to training multiple sub-networks with different structures, and these sub-networks share some parameters. In this way, the generalization ability of the model is improved, and the influence of noise or blur caused by bumps on the performance of the model is reduced.

[0079] After processing by the decoder, the final output result of the multi-scale spatial conversion module, i.e., the image BEV features The size of the image BEV feature here is determined by the set bird's eye view range and the voxel size together, as shown in the following formula:

[0080]

[0081] The embodiment of the present application adopts PointPillars as the radar backbone network, which is an architecture specially designed for processing lidar data. PointPillars can effectively convert lidar point cloud data into pseudo 2D image features, which are convenient for subsequent processing.

[0082] After the PointPillars processes the lidar data into columnar features, pseudo 2D image features are obtained. These features capture the spatial distribution and intensity information in the point cloud data. The pseudo 2D image features are input into a 2D convolutional network to obtain three features of different scales. These features reflect the spatial structure of the point cloud data at different scales. In order to obtain the lidar BEV feature, the three features of different scales are upsampled, the spatial dimensions are unified to 200x200, and the channel dimensions are spliced. This processing method helps to preserve the multi-scale information in the point cloud data while converting it into a format suitable for BEV generation.

[0083] In step 103, the image BEV feature and the lidar BEV feature are aligned and fused to obtain a BEV fusion feature. The BEV fusion features of the past n time points are introduced to enhance the BEV fusion feature of the current time point to obtain an enhanced BEV fusion feature. The enhanced BEV fusion feature is processed by a semantic segmentation task head in the decoder to obtain a bird's eye view generation result.

[0084] When the image BEV feature and the lidar BEV feature are obtained respectively, they need to be fused to provide a BEV fusion feature for subsequent tasks. However, since the influence of bumps on different modal data is not the same, there will be a feature misalignment problem when different modal features are fused. For the image BEV feature, since the mapping from the front view to the bird's eye view does not essentially deviate from the assumption of flat ground, the bump problem will have a serious impact on the image BEV feature, resulting in feature projection misalignment. At the same time, the lidar BEV feature is less affected by bumps and can still maintain a stable spatial structure under bump conditions. Therefore, as Figure 6As shown, the embodiment of the present application designs a feature alignment and fusion module. By using the spatial structure stability of the laser radar BEV feature, the projection misplacement of the image BEV feature is corrected, so that the two modal features can be matched in space position. The channel attention mechanism is used to self-enhance the laser radar BEV feature, and the interference of bumps on the point cloud data is eliminated. In addition, when fusing two different modal features, it is necessary to fully consider how to give full play to the respective advantages of the two modal features, such as the more dense semantic information of the image BEV feature and the more stable spatial properties of the laser radar BEV feature. Therefore, the model also uses a triple attention mechanism to process the preliminary BEV fusion feature, models the space and channel at the same time, lets the model face different scenes and selects more important features, so that the features of different modalities can give full play to their respective advantages.

[0085] The module includes multiple key components, and the specific process is as follows:

[0086] First, the image BEV feature and the laser radar BEV feature are preliminarily spliced, and the spliced feature is input to the first convolutional layer for convolution operation to obtain the preliminary fusion BEV feature. The preliminary fusion BEV feature and the image BEV feature are input to the deformable attention module at the same time. In the deformable attention module, the channel dimension is superimposed, and the features are weighted through the attention mechanism to obtain the corrected feature. This step helps to enhance the correlation between features and improve the fusion effect. At the same time, the laser radar BEV feature is input to the channel attention module, and the channel attention module enhances the laser radar BEV feature through the channel attention mechanism to obtain the enhanced laser radar BEV feature.

[0087] Then, the corrected feature and the enhanced laser radar BEV feature are spliced in the channel dimension to obtain the preliminary BEV fusion feature. The preliminary BEV fusion feature is input to the triple attention module and the second convolutional layer in turn to model the spatial and channel dimensions, and finally the BEV fusion feature is obtained.

[0088] Specifically, in the deformable attention module, in order to better utilize the spatial distribution of the laser radar BEV feature to guide the image BEV feature, first, the channel dimension of the laser radar BEV and the image BEV feature is superimposed, and two consecutive convolutional layers are used to preliminarily fuse the data of the two modalities. The convolution kernel size is 3x3, and the channel number of the preliminary fusion feature is reduced from to , and the preliminary fusion BEV feature is taken as the query vector Q of the deformable attention module.

[0089] The preliminary fused BEV feature as a query vector can effectively improve the quality of generating bird's eye view, while directly using the lidar BEV feature as a query vector can reduce the performance of the algorithm. On the one hand, the lidar BEV feature has the characteristic of sparsity, and using the lidar BEV feature alone as a query vector can cause many image BEV features to fail to be corrected. On the other hand, although the spatial structure of the lidar BEV feature is stable, it lacks the appearance information such as texture and color in the image. Using it alone as a query vector cannot solve the situation of accurately judging the boundary and type of the object relying on the appearance feature. Using the preliminary fused BEV feature as a query vector can make full use of the advantages of both and make full use of the complementary information for feature correction. In addition, the preliminary fused BEV feature obtains the internal relationship between the lidar BEV feature and the image BEV feature through convolution, and the deformable attention can adaptively adjust the weight of different regions based on this correlation, and more reasonably correct the image BEV feature.

[0090] Specifically, the deformable attention module structure is as shown in Figure 7 The image BEV feature and the preliminary fused BEV feature are input. In order to better utilize the preliminary fused BEV feature to correct the image BEV feature in space, the preliminary fused BEV feature is first linearly projected to obtain the query vector Q. Then it is input into the lightweight subnetwork to generate the offset, and the image BEV feature and the offset are input into the offset function to obtain the preliminary corrected feature. Then the preliminary corrected feature is linearly projected onto the key vector K and the value vector V. Finally, Q, K and V are input into the multi-head attention module to obtain the corrected feature, as shown in the following formula:

[0091]

[0092] Wherein, represents the preliminary corrected feature; represents the image BEV feature; represents the offset; represents the preliminary fused BEV feature; represents the sampling of the unit near the offset point by linear interpolation.

[0093] In the above process, the preliminary fused BEV feature is not only the input of the offset network, but also the query vector Q of the multi-head attention module, which corrects the image BEV feature in two stages. The first stage corrects the image BEV feature based on the preliminary fused BEV feature through the offset network, and the second stage further optimizes and corrects the image BEV feature from the feature correlation level by using the multi-head attention mechanism, which realizes the effect of gradually refining and optimizing the feature.

[0094] Furthermore, this embodiment employs a channel attention module to perform feature self-enhancement on the LiDAR BEV features, suppressing the influence of noisy channels on the model and selectively enhancing features more critical to subsequent tasks. Specifically, this embodiment uses a variant of SE channel attention. For LiDAR BEV features, firstly, its spatial dimension... Global average pooling is performed to compress the two-dimensional feature map of each channel into a single value, that is, to average the features of each channel in the spatial dimension to obtain a value of size . The eigenvectors of are given by the following formula:

[0095]

[0096] in, This represents the i-th element after global average pooling. The values ​​corresponding to each channel; The first characteristic in the BEV feature of lidar The feature values ​​at each channel.

[0097] The feature vector after global average pooling is processed through a fully connected layer to obtain the final channel attention vector. Channel attention Each element corresponds to the weight of each channel in the feature map. Ultimately, [the weight will be...]. Multiplying the LiDAR BEV feature by the enhanced LiDAR BEV feature yields the following formula:

[0098]

[0099] in, Indicating the BEV characteristics of lidar in Location characteristics, This indicates that the enhanced BEV feature of lidar is in Characteristics of location.

[0100] Furthermore, both LiDAR BEV features and image BEV features originate from inputs from different modal sensors, each containing unique and complementary information. While simple convolution operations can fuse these two types of features, they don't fully leverage the advantages of different modal data. Simply using channel attention or spatial attention to enhance or suppress channel or spatial dimensions doesn't consider the relationship between space and channels. For the stitched BEV features, not all spatial locations within the same channel or all channels within the same spatial location are equally important. Therefore, for the stitched features, such as... Figure 8As shown, the embodiments of the present application introduce a triple attention module to jointly model its spatial and channel dimensions, so that the model can adaptively enhance the features of different modalities and different positions for different scenarios. The triple attention mechanism can assign different attention weights to different channels and different spatial positions according to the importance of the features. For example, in the detection of lane lines, the accurate spatial information in the laser radar BEV feature and the appearance information (such as color, shape, etc.) of the image BEV feature should be highlighted. By introducing the triple attention, the two different modal features can be better utilized, and the subsequent modules can better focus on the truly key information for accurate prediction.

[0101] Specifically, first, the rectified features are concatenated with the enhanced laser radar BEV features in the channel dimension to obtain preliminary BEV fusion features , and the preliminary BEV fusion features are adjusted in dimension to obtain feature maps with feature dimensions of respectively. Then, average pooling and maximum pooling are performed along the last dimension of each feature map to obtain the feature representations of the feature maps in the channel direction, the horizontal direction, and the vertical direction. Then, the average pooling and maximum pooling results of the features with different dimensions are concatenated in the last dimension, and processed by two convolution kernels with a size of 3x3 and a Sigmoid activation function to obtain attention weight maps with dimensions of The attention weight maps represent the attention weights in the spatial dimension , the channel-spatial dimension , and the channel-spatial dimension respectively. Finally, the generated attention weight maps are multiplied with the preliminary BEV fusion feature maps and adjusted in dimension, and the dimensions of the three adjusted feature maps are restored to , and the three are added element by element to obtain the final output result, i.e., the BEV fusion feature, as shown in the following formula:

[0102]

[0103] wherein, represents the preliminary BEV fusion feature map; represents the feature map after dimension transformation; represents the Sigmoid activation function; and represent the convolution layers of the three branches, and the horizontal line above the parameters represents dimension adjustment of the features to restore the feature dimension to .

[0104] Further, in the automatic driving scene, the bumping will make the vehicle present complex and continuous changes. The image or laser radar data at a single moment only reflects the instantaneous scene information, and such instantaneous information is extremely susceptible to the interference brought by the bumping, resulting in the deviation and inaccuracy of the generated bird's eye view. In order to solve the single-frame bird's eye view deviation and inaccuracy problem caused by the bumping, as shown in Figure 9 , the embodiment of the present application also proposes a time sequence fusion module based on the self-vehicle position information. Through the vehicle position change information, the previous multi-frame bird's eye view features are aligned to the bird's eye view coordinates at the current moment, the features at the current moment are corrected and enhanced, and the single-moment feature deviation and misplacement problem caused by the bumping is solved. In addition, the fusion of multi-frame bird's eye view features can also solve the occlusion or missing problem of single-frame data, and improve the integrity of the bird's eye view.

[0105] It can be divided into two parts of feature alignment based on vehicle position information and time sequence feature fusion. Specifically, for the BEV fusion features at different moments, further feature fusion is needed. First, through the change of the vehicle position information, the features at the previous moments are converted to the vehicle coordinate system at the current moment, so as to eliminate the feature difference caused by the different positions and postures of the vehicle at different moments. Assuming that the BEV fusion feature at the current moment is , the BEV fusion features at the previous n moments are , the BEV fusion feature at the current moment is , and the BEV fusion feature at the t-1 moment is , for example, the is aligned to , first, the coordinates and angles of the vehicle at the t moment and the t-1 moment in the world coordinate system are obtained, then the difference between the two is obtained, and the relative position vector of the vehicle at the t moment and the t-1 moment in the world coordinate system, the direction angle of the relative position vector and the relative angle change are obtained. At this time, whether it is the position change relationship or the angle change relationship is in the world coordinate system, it needs to be converted to the self-vehicle coordinate system of the current vehicle. In order to convert the relative position vector to the self-vehicle coordinate system, the angle of the relative position vector in the self-vehicle coordinate system at the t moment is needed, and the formula is as follows:

[0106]

[0107] , wherein represents the angle of the relative position vector in the self-vehicle coordinate system at the t moment; represents the vehicle angle at the t moment; represents the direction angle of the relative position vector.

[0108] After obtaining the angle of the relative position vector in the ego vehicle coordinate system at time t, the position offset in the ego vehicle coordinate system is represented as follows:

[0109]

[0110] wherein, and denote the offset in the x-axis and y-axis of the ego vehicle coordinate system, respectively; denotes the relative position vector.

[0111] At this point, the relationship between the vehicle coordinate system at time t and the vehicle coordinate system at time t-1 is established, and the BEV fusion feature at time t-1 is aligned to the BEV fusion feature at the current time, as follows:

[0112]

[0113] wherein, denotes the rotation transformation of the BEV fusion feature at time t-1, which is used to adjust the difference in feature direction caused by the change in vehicle angle; denotes the translation transformation of the BEV fusion feature at time t-1, which is used to correct the feature position deviation caused by the movement of the vehicle.

[0114] It should be noted that for the aligned feature , the Detach operation needs to be performed on the features of the previous n time points, so that the previous time point features do not participate in the backpropagation process in the training process, avoiding unnecessary interference of historical data on the feature gradient update at the current time. In addition, attention should be paid to whether the previous time point is in the same scene as the current time. If not, the BEV fusion feature at the current time is directly taken as the previous BEV fusion feature to participate in subsequent calculations.

[0115] After the alignment operation on the time sequence features, the features at multiple time points need to be aggregated to realize the deep fusion and enhancement of the time sequence features. Benefiting from the static characteristics of map elements such as lanes and pedestrian crossings, it can be considered that they are in the same position in the aligned features at different times. This provides a basis for the fusion of time sequence features.

[0116] Firstly, the embodiment of the present application splices the channel dimension of the multi-frame BEV fusion features that have been aligned. Assuming that there are n BEV fusion features at different time points, and each feature has c channels, the number of feature channels after splicing will become n x c. This splicing method can retain the independent information of each frame and provide rich input for subsequent fusion. Subsequently, the embodiment of the present application fuses the spliced features using a convolution method. Specifically, a convolution layer with a kernel size of 3 x 3 is used to process the spliced features. The purpose of the convolution layer is to deeply fuse the multi-frame time sequence features, and gradually reduce the channel dimension of the features through consecutive convolution, normalization and activation operations, and finally obtain enhanced BEV fusion features, as shown in the following formula:

[0117]

[0118] wherein, represents the BEV fusion feature at the previous time point after alignment; represents the BEV fusion feature at the current time point; represents the superposition of the channel dimension; represents the convolution operation on the feature with a kernel size of 3 x 3; represents the enhanced BEV fusion feature at time t.

[0119] In addition, the enhanced BEV fusion feature will not only be used for the feature representation at the current time point, but also will participate in the feature enhancement process at the subsequent time point. Specifically, at each subsequent time point, the embodiment of the present application will fuse the enhanced BEV fusion feature obtained at the previous time point with the enhanced BEV fusion feature at the current time point. This mechanism can implicitly introduce longer time sequence information into the feature at the current time point without significantly increasing the model calculation complexity, so that the effect of feature fusion can be continuously accumulated and optimized over time. In this way, the embodiment of the present application can better utilize historical time sequence information and further improve the understanding ability of complex scenes. The accumulation of this time sequence information not only helps to reduce the feature deviation and misplacement caused by factors such as vehicle bumping, but also improves the utilization efficiency of time sequence information in the entire bird's eye view generation process, and finally generates more accurate and stable bird's eye view.

[0120] Finally, the enhanced BEV fusion feature is processed by the semantic segmentation task head in the decoder to obtain the bird's eye view generation result. The decoder usually includes multiple up-sampling layers and convolution layers to gradually restore the enhanced BEV fusion feature to the spatial resolution of the original image, and generate the final bird's eye view through the semantic segmentation task head.

[0121] The embodiment of the application sets the bird's eye view range to [0, 30] m x [-15, 15] m, and the voxel resolution is 0.15 m. In the set range, the model uses IoU, CPA and LPIPS as evaluation indexes to evaluate the performance of the method of the embodiment of the application. Among them, the LPIPS index is mainly used in the comparative experiment. In order to fully verify the effectiveness of the proposed method, the ablation experiment is designed to verify the effectiveness of each module of the method proposed in the paper.

[0122] Firstly, the method proposed in the embodiment of the application adopts a multi-scale space conversion module (MSCM), a feature alignment and fusion module (AFAM) and a time sequence fusion module based on ego vehicle location information (TFM-EVLI) to reduce the influence of bumps on the generation of bird's eye view. In order to verify the influence degree of each module of the embodiment of the application on the model, the embodiment of the application constructs 8 kinds of baseline networks to verify the contribution of each module to the model, and the ablation analysis of the above three modules is performed. Specifically as follows: a) only use the basic network, do not add any of the above modules; b) use the basic network and the multi-scale space conversion module c) use the basic network and the feature alignment and fusion module d) use the basic network and the time sequence fusion module based on ego vehicle location information e) use the basic network, the multi-scale space conversion module and the feature alignment and fusion module; f) use the basic network, the multi-scale space conversion module and the time sequence fusion module based on ego vehicle location information; g) use the basic network, the feature alignment and fusion module and the time sequence fusion module based on ego vehicle location information; h) use the complete network model. The parameter settings of the baseline networks of the above ablation experiments remain unchanged during training and testing, and the Divider, Ped and Boundary in the table represent lane dividers, pedestrian crossings and lane boundaries respectively. The results of the ablation experiment are as shown in the following table:

[0123]

[0124] Firstly, the influence of a single module on the performance of the model is observed. It can be known from the observation of the results corresponding to a), b), c) and d) that whether it is the multi-scale space conversion module, the feature alignment and fusion module or the time sequence fusion module based on ego vehicle location information, the model performance is improved after adding. Overall, the greatest performance improvement can be obtained after adding the time sequence fusion module, the IoU and CPA indexes of the lane dividers increase by 1.8% and 1.3%, the IoU and CPA indexes of the pedestrian crossing increase by 3.5% and 8.3% respectively, and the IoU and CPA indexes of the lane edge increase by 2.3% and 7.3% respectively.

[0125] In principle, the temporal fusion module based on vehicle location information introduces more additional information compared to other modules. Furthermore, since the positions of map elements in world coordinates do not change, the features provided by this additional information can almost always be considered accurate, provided the sensor extrinsic parameters are accurate. Therefore, the temporal fusion module can significantly improve model performance.

[0126] Next, the impact of different combinations of these three modules on model performance was observed. For combinations e), f), and g), which include two modules, the performance was better than that of introducing the corresponding module alone. Specifically, the model combining the multi-scale spatial transformation module and the feature alignment and fusion module performed slightly worse, possibly because both modules involve correcting image feature projections. Overall, the model including the temporal module performed better, with the model combining the feature alignment and fusion module and the temporal fusion module exhibiting the best overall performance. Finally, comparing the performance of models combining two modules and models combining three modules shows that the complete model has the best performance, effectively demonstrating the synergistic effect among the three modules designed in this embodiment.

[0127] In summary, the three modules proposed in this application can improve the performance of the bird's-eye view generation algorithm. For a single module, the temporal fusion module based on vehicle location information can achieve better results than the other two modules; for multiple modules, the model combining the three modules has the best performance in the bird's-eye view generation task.

[0128] The multi-scale spatial transformation module can be divided into two steps: the first step is to acquire multi-scale image BEV features, and the second step is to fuse image BEV features at different levels. Firstly, to explore the impact of different scale features on algorithm performance, this embodiment of the application inputs image features at scales {1 / 16, 1 / 32, 1 / 64} into the multi-scale spatial transformation module and performs performance tests without modifying other modules.

[0129] like Figure 10 and Figure 11 As shown, where This indicates the fusion of features at three different scales: 1 / 16, 1 / 32, and 1 / 64. Experimental results show that, when selecting a single scale feature, the 1 / 32 scale feature achieves the best performance. Models using 1 / 16 or 1 / 64 scale features perform relatively poorly. This may be because the 1 / 16 scale feature fails to capture enough high-level details, while the 1 / 64 scale feature is too abstract for subsequent tasks. Furthermore, the use of features fused from three different scales demonstrates this approach. The model of the application achieves the optimal result in both IoU and CPA, fully illustrating the effectiveness of multi-scale feature fusion. The multi-scale method can take advantage of each scale of feature and make up for the deficiency of single scale feature, thereby significantly improving the comprehensive performance of the model.

[0130] In addition, in order to explore the influence of different fusion methods on the performance of the algorithm, the application embodiment performs ablation analysis on the multi-scale feature fusion method, and three cases are set.a) Feature fusion through Add+CNN; b) Feature fusion through Concat+CNN; c) Feature fusion through the MBF module proposed in the application embodiment. For the first two methods, the number of convolution layers is set to 2, and the convolution kernel size is set to 1x1 and 3x3 respectively. Compared with the first two methods, the fusion module proposed in the application embodiment has better performance, and the experimental results are shown in the following table:

[0131]

[0132] In the feature alignment and fusion module proposed in the application embodiment, the spatial property stability of the laser radar BEV feature is used to perform feature alignment operation on the image BEV feature, to solve the possible projection misplacement problem of the image BEV feature. In order to explore the effectiveness of the feature alignment operation, the application embodiment sets up an ablation analysis, and three cases are set.a) No feature alignment; b) Directly use the laser radar BEV feature to correct the position of the image BEV feature; c) Use the preliminarily fused BEV feature to correct the position of the image BEV feature. The experimental results are shown in the following table:

[0133]

[0134] It can be seen from the observation of the test results that the correction of the image BEV feature by using the preliminarily fused feature can make the algorithm obtain better results. Compared with not performing feature alignment, the model increases by 0.8%, 1.5% and 0.9% in IoU of lane dividers, pedestrian crossings and lane edges respectively, and increases by 0.9%, 2.8% and 2% in CPA respectively. In addition, directly using the laser radar BEV feature to correct the image BEV feature will cause other indicators except lane edges to decrease significantly, because the direct correction method using the laser radar BEV feature does not consider the difference between the image feature and the laser radar feature. Compared with the image feature, the laser radar feature lacks sufficient semantic information, and it is difficult to match the image BEV feature. In contrast, the preliminarily fused feature not only integrates the high-precision spatial information of the laser radar, but also fully retains the rich semantics and visual features of the camera image, and can provide more accurate and rich basis for the correction of the image BEV feature. Therefore, using the fused feature can obtain relatively better results.

[0135] In the in-depth study of the bird's eye view generation model, the time sequence fusion module plays a key role in improving the performance of the model. Therefore, the embodiments of the present application specially carry out comprehensive ablation experiments on the number of frames fused by the module, wherein the frame number 0 represents that no time sequence information is used, the frame number 3 and 5 represent that the information of the past 3 frames or the past 5 frames is used, and the experimental results are as shown in the following table and the following table: Figure 12

[0136]

[0137] It can be seen from the comparison of the experimental results of the frame number 0 and the frame number 3 that the introduction of the time sequence information can significantly improve the performance of the model. The IoU and CPA of the lane dividers increase by 3.2% and 1.8% respectively, the IoU and CPA of the pedestrian crossing increase by 3.3% and 6.4% respectively, and the IoU and CPA of the lane edge increase by 2.4% and 5.1% respectively, and each indicator is obviously improved.

[0138] ​Furthermore, further observation of the experimental results of fusing the past 3 frames and the past 5 frames reveals that, in terms of the improvement, the model performance did not significantly improve with the increase in the number of fused frames. This may be related to the data used in the experiments of this application embodiment. The experiments of this application embodiment only used the front view image and the corresponding point cloud information. For the construction of a bird's-eye view focusing on the front of the vehicle, the changes in the scene in front of the vehicle are relatively continuous, and only the information of the nearest moment is most critical for the model to judge the state of the environment in front of the vehicle. Information from past frames beyond a certain time range has limited help in judging the current state of the bird's-eye view in front of the vehicle due to the large changes in the scene. This experimental result shows that there is a relatively reasonable range of frame number selection in terms of using temporal information to improve model performance. For the scenario of constructing a bird's-eye view based on the front view image and point cloud information involved in this experiment, the information of the past 5 frames is already sufficient to provide the model with relatively sufficient temporal clues. Continuing to increase the number of fused frames will not result in a significant performance improvement for the model. On the contrary, it will make the model too complex, causing the model to consume more resources during training and inference, and reducing the running efficiency.

[0139] Through observation Figure 12 The visualization results show that the bird's-eye view generated by incorporating temporal information is more complete and of better quality than the result without temporal information. This is because single-frame information can be significantly affected by abnormal conditions, such as bumps or occlusions. Incorporating temporal information can effectively smooth out these abnormalities, improving the model's robustness to abnormal conditions.

[0140] To explore the impact of multimodal inputs on the algorithm, this application designed experiments for different modal inputs. Based on the type of input data, they can be divided into three different types: a) camera input only; b) LiDAR input only; c) simultaneous camera and LiDAR input. It is worth noting that since the lack of LiDAR data input will cause the feature alignment and fusion module to malfunction, the experiments were conducted using a model that includes a multi-scale spatial transformation module and a temporal fusion module. The experimental results are as follows... Figure 13 And as shown in the table below:

[0141]

[0142] Through Table 5 and as shown in the figure Figure 13The visualization of the experimental results can clearly observe the influence of different modal inputs on the model performance. When only using camera modal data as input, the model achieves the worst performance in the three groups of experiments. This may be because although camera data can theoretically provide more abundant visual features than lidar data, such as object color, shape, texture, etc., these information is important for the model to understand the scene content. However, in practical applications, complex and variable scenes pose many challenges to cameras. In complex scenes, image features are more susceptible to factors such as occlusion and lighting, which can interfere with the model's extraction and analysis of object features in the image, thus leading to poor overall performance of the model.

[0143] When only using lidar input, the indicators of the algorithm have improved compared to only camera input. This may be because in the specific distance range of 0-30m, the data characteristics of the lidar perform well and there is no obvious sparsity problem. And compared with image data, lidar data can obtain the depth information of the object. This means that within this distance range, lidar can densely collect information about the surrounding environment, providing the model with rich and accurate features. Therefore, in the bird's eye view generation task within the range of 0-30m, the model using only lidar data achieves better results than the model using only camera data.

[0144] Finally, when using both modal data as input, the algorithm achieves the best results, and some areas that could not be clearly segmented in the two single-modal models can also be accurately divided. This fully demonstrates the complementarity of camera and lidar data, the rich texture information provided by the camera combined with the accurate distance information provided by the lidar can provide more comprehensive and accurate features for the algorithm, thus significantly improving the model's performance in the bird's eye view generation task.

[0145] In the bird's eye view generation task, the 2D backbone network is responsible for extracting rich and valuable features from the camera image, laying the feature foundation for accurate bird's eye view generation. Compared with lidar point cloud, image can provide more dense and detailed semantic information, such as color, shape, etc. An effective image feature extraction network can provide high-quality features for subsequent steps, thus improving the overall performance of the model in the bird's eye view generation task.

[0146] The present application embodiment selects a total of 4 different feature extraction networks: ResNet101, EfficientNet-b0, EfficientNet-b3, Swin-T to extract image features. The experimental results are shown in the following table:

[0147]

[0148] From the table, it can be seen that compared with ResNet-101, EfficientNet-b3 network has better performance with less parameter amount. At the same time, Swin-T has higher performance than EfficientNet-b3 with close parameter amount. This may be due to the fact that Swin-T is based on the Transformer architecture, and its self-attention mechanism can capture long-distance dependencies in images, while lane lines, road edges and other map elements themselves have global correlation. Using Swin-T can better understand the semantic information and spatial relationship between the elements of the bird's eye view.

[0149] In summary, since the Swin-T backbone network performs outstandingly in the bird's eye view generation task, the model selects Swin-T as the backbone network for extracting image features.

[0150] In order to better verify the effectiveness of the bird's eye view generation method based on multi-scale feature transformation and time sequence context proposed in the embodiments of the present application, the embodiments of the present application perform comparative experiments on the NuScenes dataset. First, comparative experiments of overall algorithm performance are performed to verify the performance advantages of the algorithm. Then, comparative experiments are performed under three different road conditions of flat road, lightly bumpy road and bumpy road to verify the robustness of the algorithm. In these experiments, not only the model proposed in the embodiments of the present application is tested in detail, but also other bird's eye view generation algorithms are compared fully to evaluate the advantages and disadvantages of different algorithms. The comparative experiments respectively select LSS, PointPillars, HDMapNet, BEVFusion, BevFusion and SuperFusion.

[0151] LSS realizes the perspective change from perspective view to bird's eye view and performs specific detection or segmentation tasks through four steps of feature extraction, "Lift" (mapping image feature points to 3D camera coordinate system to form a point cloud based on a deep probability distribution), "Splat" (converting camera point cloud to ego coordinate system and "flattening" to BEV feature map) and "Shoot" (performing specific tasks such as bird's eye view generation). PointPillars is a point cloud processing network that balances speed and performance. It greatly reduces the performance consumption of traditional point cloud networks by generating pseudo 2D maps. HDMapNet focuses on the generation of bird's eye view and designs three task heads: semantic segmentation, instance vector prediction and direction prediction. The instantiated bird's eye view result is obtained through post-processing. BEVFusion improves the robustness of the algorithm and provides a unified feature space for BEV feature fusion, so that the model can still have good performance when a single sensor fails. BEVFusion innovates in feature fusion, unifying the multi-modal features of cameras and lidar into a shared bird's eye view representation space while preserving geometric and semantic information. It also solves the efficiency bottleneck of view transformation through optimized BEV pooling operations, supporting different perception tasks. SuperFusion performs multi-level fusion of features and generates dense depth maps through bilinear interpolation, providing more effective depth supervision for depth estimation, improving the accuracy of depth prediction and effectively improving the performance of bird's eye view generation algorithms.

[0152] Table 7 is the test results of the model on the NuScenes dataset, where Mod represents the modality used by the model, C represents the camera modality, L represents the laser point cloud modality, and C+L represents the simultaneous use of camera and laser point cloud. According to the test results, for the IOU and CPA of three different targets of lane marking, pedestrian crossing and road boundary, the algorithm proposed in the embodiment of the application achieves the best results for all indicators except the CPA of lane marking. Specifically, the IoU and CPA corresponding to the pedestrian crossing are improved by 1.7% and 0.5% respectively, the IoU corresponding to the lane marking is improved by 0.5%, and the IoU and CPA corresponding to the lane edge are improved by 1.3% and 0.9% respectively. At the same time, observing the LPIPS indicator, the results generated by the algorithm proposed in the embodiment of the application also have the best performance for visual perception systems, as shown in the following table:

[0153]

[0154] In addition, detailed analysis of each index shows that SuperFusion achieves suboptimal results, which on the one hand indicates that providing display depth supervision can effectively improve algorithm performance, and the inaccuracy of depth estimation can seriously interfere with algorithm performance, and on the other hand shows that the potential link between multi-modal data needs to be explored and has great potential. In addition, by observing the experimental results, it can be found that the algorithm using both cameras and laser point clouds usually has better performance than the algorithm using only a single sensor. This fully shows that a single sensor often has certain limitations, and by combining camera and laser point cloud features, the model can have a more comprehensive and accurate understanding of the target, thereby achieving better results in various indicators.

[0155] In order to more intuitively observe the advantages of the bird's eye view generation method based on multi-scale feature transformation and time sequence context proposed in the embodiments of the present application in dealing with the problems caused by bumps, the embodiments of the present application compare the performance of the method proposed in the embodiments of the present application and other algorithms in different scenes of the NuScenes dataset. Specifically, here the Z-axis acceleration is used as the basis for evaluating whether the road surface is bumpy, and under the condition that the vehicle is stationary, the Z-axis acceleration is greater than 9.8 m / s. According to the degree of change of the Z-axis acceleration curve, the embodiments of the present application carefully classify the road conditions, and divide the road conditions into three different types: flat, slight bump and bump. If the fluctuation range of the Z-axis acceleration is in the interval of 9.5 m / s to 10.5 m / s, it is a flat road; if the fluctuation range of the Z-axis acceleration is in the interval of 9.0 m / s to 10.5 m / s, it is a slightly bumpy road; if the fluctuation of the Z-axis acceleration exceeds this range and the curve oscillates strongly, it is a bumpy road. For each type of road condition, 20 scenes are selected, totaling 800 frames of data for comparative experiments.

[0156] The experimental results of the algorithm in the flat road surface scene are shown in Table 8, and the visualization results are shown in Figure 14 , Figure 15

[0157]

[0158] By observing the experimental results, it can be found that the results of LSS and PointPillars show different trends from Table 7. In Table 7, the performance of PointPillars is higher than that of LSS, while in Table 8, the performance of LSS is higher than that of PointPillars in all indicators except the lane edge, which shows that in good driving environments, camera modal data can provide more rich information. When the vehicle is in a bumpy or other bad driving environment, the laser point cloud modal data is less disturbed than the camera modal, and can provide more reliable information for the generation of bird's eye view.

[0159] ​In addition, in the case of flat road conditions, after comparing the performance of the algorithm proposed in the embodiments of the present application with that of other numerous algorithms, it is found that although the performance of the algorithm proposed in the embodiments of the present application is similar to that of SuperFusion and does not show obvious advantages, it is still significantly better than other algorithms. Specifically, in the recognition of pedestrian crossings, the algorithm proposed in the embodiments of the present application achieves the best results in the IoU and CPA indicators. At the same time, in the IoU, CPA indicators and overall LPIPS indicators of lane dividers and lane edges, the algorithm proposed in the embodiments of the present application all achieves sub-optimal results, fully proving that the method proposed in the embodiments of the present application also has excellent performance in flat road conditions.

[0160] From the visual results, it can be seen that in flat road conditions, most algorithms can achieve good results. Compared with other algorithms, the algorithm proposed in the embodiments of the present application has better comprehensive performance. It is worth noting that other algorithms often lose information when processing areas at a long distance and under occlusion. However, the algorithm proposed in the embodiments of the present application can achieve good results when processing this part of the area, which may benefit from the multi-scale spatial conversion module. When facing areas at a long distance and under occlusion, large-scale features can capture the overall contour and approximate position, while small-scale features focus on detailed resolution. The two complement each other, allowing the model to reasonably predict the area through limited information, thereby effectively avoiding the information loss problem of the bird's eye view generation result.

[0161] The experimental results in the case of light bumpy road conditions are shown in Table 9, and the visual results are shown in Figs. 9a-9d: Figure 16 Figure 17

[0162]

[0163] By comparing Table 8 and Table 9, it can be seen that compared with flat road conditions, the performance of all algorithms in light bumpy road conditions has declined to some extent. This phenomenon may be due to the fact that light bumpy road conditions can cause the data collected by the camera and laser radar carried by the vehicle to deviate to some extent, thereby affecting the processing and analysis of the data by the algorithm, and ultimately reflecting as a decline in algorithm performance.

[0164] ​​Although the performance of all algorithms is reduced in the light bump road, SuperFusion and the algorithm proposed in the embodiment of the present application still achieves suboptimal and optimal results. This fully shows that the two algorithms have stronger adaptability and stability than other algorithms in dealing with light bump roads. It is worth noting that compared with SuperFusion, the performance of the algorithm proposed in the embodiment of the present application has a smaller decline. Specifically, the IoU indicators of the road separation line and pedestrian crossing of SuperFusion decrease by 5.3% and 10.8% respectively, the CPA indicators decrease by 6.1% and 14.4% respectively, and the LPIPS indicator increases by 0.0193, while the IoU indicators of the algorithm proposed in the embodiment of the present application decrease by 4.3% and 10.1% respectively, the CPA indicators decrease by 4.8% and 12.6% respectively, and the LPIPS increases by 0.0098. In addition, the IoU and CPA indicators of the road edge are basically not affected and do not decrease. This may be because the road edge information mainly depends on the laser point cloud input, and the light bump has little effect on the laser radar, so the corresponding indicators do not decrease. At the same time, the visualization results also show that the bird's eye view generated by the algorithm proposed in the embodiment of the present application has better effect.

[0165] The experimental results of the algorithm in the bump road are shown in Table 10, and the visualization results are shown in Figure 18 、 Figure 19

[0166]

[0167] From the experimental results, it can be seen that in the bump road, the algorithm proposed in the embodiment of the present application shows obvious advantages compared with other algorithms. Specifically, compared with the suboptimal algorithm, the IoU and CPA indicators of the road separation line of the algorithm proposed in the embodiment of the present application are improved by 2.5% and 2.9% respectively, the corresponding indicators of the pedestrian crossing are improved by 1.4% and 1.8% respectively, the corresponding indicators of the lane edge are improved by 1.8% and 0.7% respectively, and the LPIPS indicator is reduced by 0.0064. This series of data shows that in the bump road, the algorithm proposed in the embodiment of the present application has stronger stability and accuracy than other algorithms.

[0168] From the visualization results Figure 18 、 Figure 19 ​In this way, the advantages of the algorithm of the embodiments of the present application can be more intuitively seen. The lines of the road separation line, the pedestrian crossing and the lane edge in the bird's eye view generated by the algorithm of the embodiments of the present application are clearer, continuous and accurate, and have a higher matching degree with the actual scene. However, in the bird's eye view generated by other algorithms, these elements will appear fuzzy, broken or deviated in position and other problems under bumpy road conditions. These experimental results and visual comparisons fully and powerfully illustrate that the method proposed in the embodiments of the present application has unique advantages in processing bumpy road conditions, and can effectively solve the problem of the decline in the generation quality of the bird's eye view caused by the bumpy road surface.

[0169] In summary, the bird's eye view generation model proposed in the embodiments of the present application has shown significant advantages in comparison with other models, and has exhibited excellent performance and strong adaptability under three different road conditions of flat road, slightly bumpy road and bumpy road, proving the effectiveness and robustness of the algorithm.

[0170] It should be noted that in the present application, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element. In addition, it should be pointed out that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but can also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0171] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above specific embodiments, and the above specific embodiments are only illustrative and not limiting, and those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. A method for generating an aerial view based on multi-scale feature transformation and temporal context, characterized in that, The method comprises the following steps: acquiring original modal data, wherein the original modal data comprises image data and lidar data; processing the image data through an image backbone network to obtain front-view image features, performing multi-scale spatial conversion on the front-view image features to obtain image BEV features, processing the lidar data through a radar backbone network to obtain lidar features, compressing the lidar features along the Z axis to obtain lidar BEV features; performing feature alignment and fusion on the image BEV features and the lidar BEV features through a feature alignment and fusion module, wherein the feature alignment and fusion module comprises a deformable attention module, a first convolutional layer and a channel attention module connected in parallel, and a triple attention module and a second convolutional layer connected in series with the feature alignment and fusion module; wherein the image BEV features and the lidar BEV features are preliminarily spliced and input into the first convolutional layer to obtain preliminary fusion BEV features, and then the image BEV features and the preliminary fusion BEV features are input into the deformable attention module to perform stacking in the channel dimension to obtain correction features; the lidar BEV features are input into the channel attention module to obtain enhanced lidar BEV features; then the correction features and the enhanced lidar BEV features are spliced in the channel dimension to obtain preliminary BEV fusion features; the preliminary BEV fusion features are sequentially input into the triple attention module and the second convolutional layer to simultaneously perform modeling in the spatial and channel dimensions to obtain BEV fusion features, the BEV fusion features of the past n time points are introduced, the BEV fusion features of the past n time points and the BEV fusion features of the current time point are aligned, the BEV fusion features of the past n time points are converted to the vehicle coordinate system at the current time through the change of the vehicle position information, and the BEV fusion features of the past n time points converted to the vehicle coordinate system at the current time and the BEV fusion features of the current time are aggregated to obtain enhanced BEV fusion features, and the enhanced BEV fusion features are processed through a semantic segmentation task head in a decoder to obtain an aerial view generation result.

2. The multi-scale feature transform and temporal context based bird's eye view generation method of claim 1, wherein, The method comprises the following steps: Swin-T is adopted as the image backbone network to process the image data, and four different scale features are output , and the channel dimension numbers corresponding to the four different scale features are , the dimensions of and are unified, the bilinear interpolation is adopted to up-sample , and the fusion of and is performed through two layers of convolution to obtain the front view image feature, and the multi-scale spatial conversion is performed on to obtain the image BEV feature.

3. The multi-scale feature transform and temporal context based bird's eye view generation method of claim 1 or 2, wherein, The method comprises the following steps: adopting PointPillars as the radar backbone network to perform columnar feature processing on the lidar data to obtain pseudo 2D image features, inputting the pseudo 2D image features into a 2D convolutional network to obtain three features of different scales, performing upsampling on the three features of different scales to unify the spatial dimension and perform channel dimension splicing to obtain lidar BEV features.

4. The multi-scale feature transform and temporal context based bird's eye view generation method of claim 1 or 2, wherein, The method comprises the following steps: performing multi-scale spatial conversion on the front-view image features through a multi-scale spatial conversion module, wherein the multi-scale spatial conversion module comprises an encoder, a view relationship module, a multi-scale feature fusion module and a decoder connected in sequence; wherein the encoder is composed of two residual convolution modules connected in series, and each residual convolution module performs twice convolution operation and twice downsampling operation on the front-view image features to obtain down-sampled features; The view relationship module is connected by two fully connected layers and a convolution layer, the two fully connected layers are used for mapping the down-sampling features, and the convolution layer is used for reducing the channel dimension to obtain mapping features at different levels; The multi-scale feature fusion module is connected by an offset prediction branch and a splicing layer, the offset prediction branch further includes two convolution layers, the offset prediction branch is used for splicing the mapping features at different levels in the channel dimension, and the two convolution layers are used for offset prediction to obtain offsets corresponding to the mapping features at different levels; based on the offsets, the mapping features at different levels are sampled in an interpolation manner, and the splicing layer is used for connecting in the channel dimension to obtain preliminary image BEV features; The decoder is used for expanding the spatial dimension of the preliminary image BEV features in a manner of twice bilinear interpolation in series with a residual convolution block, the residual convolution block is connected by two 3*3 convolution layer trunks in series and a 1*1 convolution layer branch, and image BEV features are obtained.

5. The multi-scale feature transform and temporal context based bird's eye view generation method of claim 4, wherein, It comprises: A random inactivation layer is inserted between the two 3*3 convolution layer trunks in series of the decoder, and 20% of neurons are randomly set to 0 through the random inactivation layer.

6. The multi-scale feature transform and temporal context based bird's eye view generation method of claim 1, wherein, The method comprises: Before the BEV fusion features of the past n time points and the BEV fusion features of the current time point are aligned, the BEV fusion features of the past n time points are subjected to a detach operation.

Images