Bird's eye view generation method based on feature mutual enhancement and map prior

Abstract

The application discloses an aerial view generation method based on feature mutual enhancement and map prior, and relates to the technical field of aerial view generation. The method improves generation accuracy by fusing image, radar and map information; features are obtained through Swin-T and PointPillars backbone network preprocessing, bidirectional feature projection and enhancement; the features are enhanced through a cross attention mechanism, and a result is obtained through decoder upsampling; rasterized map features are fused with image and radar features to improve model robustness. The method solves the problem of modal mismatch, enriches feature basis, optimizes feature processing through an encoder-decoder structure, improves resolution and detail performance, and introduces map prior knowledge to make the generated aerial view more consistent with the actual road scene.

Description

Technical Field

[0001] The present application belongs to the technical field of bird's-eye view generation, and in particular relates to a bird's-eye view generation method based on feature mutual enhancement and map prior. Background Art

[0002] As a key component of autonomous driving perception algorithms, the Bird's-Eye-View (BEV) generation algorithm plays an irreplaceable role. Its core goal is to deeply analyze the surrounding environment from multi-sensor data and generate a semantic map in real time. This map details key elements such as lane boundaries, lane markings, and crosswalks, providing the vehicle with precise road and location information. This acts as a real-time "navigation map" for safer and more accurate driving.

[0003] While existing bird's-eye view map generation methods have achieved some success, they still face significant deficiencies in long-range map generation accuracy. In long-range scenarios, due to factors such as sensor signal attenuation and environmental interference, the generated bird's-eye view maps are prone to blurring and distortion, resulting in a significant decrease in map accuracy, seriously impacting the safety and reliability of autonomous driving.

[0004] To address the low accuracy of long-range bird's-eye view image generation, existing methods mostly focus on introducing additional information, failing to fully leverage the advantages of multi-source information. The potential synergies between modal information remain unexplored. For example, the correlations and complementary relationships between different sensor data are not fully utilized, resulting in suboptimal fusion of information. Summary of the Invention

[0005] The purpose of this application is to provide a bird's-eye view generation method based on feature mutual enhancement and map prior, which solves the technical problem of being unable to accurately and efficiently generate high-quality bird's-eye view in long-distance bird's-eye view generation tasks through feature mutual enhancement and map prior.

[0006] To achieve the above objectives, the present invention provides a method for generating a bird's-eye view image based on feature mutual enhancement and map priors, including:

[0007] Obtaining raw image data and raw point cloud data, and preprocessing the raw image data and raw point cloud data to obtain front view image features and radar BEV features;

[0008] Perform bidirectional feature enhancement on the front view image features and the radar BEV features to obtain the front view image fusion features and the lidar fusion features;

[0009] Perform perspective conversion on the front view image fusion features to obtain the image BEV features;

[0010] The image BEV feature and the lidar fusion feature are enhanced and completed respectively by using the front view image fusion feature to obtain the enhanced image BEV feature and enhanced radar BEV feature.

[0011] Obtain the original map, process the original map, and obtain rasterized map features;

[0012] The enhanced image BEV features and enhanced radar BEV features are downsampled by the BEV encoder and fused with the rasterized map features to obtain fused BEV features.

[0013] The fused BEV features are upsampled through the decoder to obtain the bird's-eye view generation result.

[0014] The above method according to the embodiment of the present application may also have the following additional technical features:

[0015] Furthermore, the original image data and original point cloud data are preprocessed, including:

[0016] The original image data is processed through the Swin-T backbone network to obtain the front view image features; the original point cloud data is processed through the PointPillars backbone network to obtain the radar BEV features;

[0017] The process of processing the original image data by the Swin-T backbone network is divided into four stages. As the number of stages increases, the size of the front view image features decreases continuously.

[0018] The process of processing raw point cloud data by the PointPillars backbone network is divided into three stages, including PointNet feature extraction, pseudo-imaging and 2D convolution processing.

[0019] Furthermore, bidirectional feature enhancement is performed on the front view image features and the radar BEV features to obtain the front view image fusion features and the lidar fusion features, including:

[0020] Bidirectional feature projection is performed on the image features obtained in the first stage of processing the original image data by the Swin-T backbone network and the lidar features obtained in the PointNet feature extraction stage of processing the original point cloud data by the PointPillars backbone network to obtain the mapping relationship between the two modal features. The features in different feature spaces are then fused and the channel dimensions are integrated to obtain the front view image fusion features and lidar fusion features, respectively.

[0021] Furthermore, in bidirectional feature projection, the mapping of lidar features to front view image features includes:

[0022] The point cloud coordinates in the laser radar feature are converted from the laser radar coordinate system to the ego vehicle coordinate system under the laser radar timestamp, the global coordinate system, the ego vehicle coordinate system under the camera timestamp, the camera coordinate system, and the pixel coordinate system in sequence, and the camera intrinsic matrix is reduced to 1 / 4 of the original size, to complete the projection of the point cloud coordinates to the image coordinates;

[0023] After completing the projection of the point cloud coordinates to the image coordinates, the laser radar feature to image feature mapping is constructed through the Scatter operation, and the formula is as follows:

[0024]

[0025] wherein, indicates the feature of the laser radar feature mapped to the image plane; indicates the image coordinate point set corresponding to the laser radar coordinate; indicates the laser radar feature.

[0026] Further, in the bidirectional feature projection, the mapping of the front view image feature to the laser radar feature includes:

[0027] The feature of the 2D position projected by the 3D point is sampled through bilinear interpolation to obtain the mapping of the front view image feature to the laser radar feature, and the formula is as follows:

[0028]

[0029] wherein, indicates the feature of the image feature mapped to the laser radar; indicates the image coordinate point set corresponding to the laser radar coordinate; indicates the front view image feature.

[0030] Further, the image BEV feature and the laser radar fusion feature are enhanced and completed through the front view image fusion feature, to obtain the enhanced image BEV feature and the enhanced radar BEV feature, including:

[0031] The image BEV feature and the laser radar fusion feature are reduced in dimension through an encoder to obtain intermediate features, and the intermediate features include intermediate image features and intermediate radar features;

[0032] The intermediate features and the front view image fusion feature are adjusted in dimension through cross attention, the intermediate features are converted into query vectors Q, and the front view image fusion feature is converted into key vectors K and value vectors V; the relationship matrix between the intermediate features and the front view image fusion feature is obtained through the inner product operation of the query vectors Q and the key vectors K;

[0033] The relationship matrix is ​​normalized by the softmax activation function, and the normalized relationship matrix is ​​weighted and aggregated with the value vector V to obtain the aggregated BEV feature. The aggregated BEV feature includes the image aggregated BEV feature and the radar aggregated BEV feature.

[0034] Convolution operation is performed on the aggregated BEV features to reduce the channel dimension. The intermediate features and aggregated BEV features are connected in the channel dimension through the Concat operation. The connected features are then convolved, normalized, and activated to obtain preliminary enhanced BEV features.

[0035] The decoder recovers and reconstructs the preliminary enhanced BEV features to obtain enhanced BEV features, which include enhanced image BEV features and enhanced radar BEV features.

[0036] Furthermore, the encoder consists of 3 convolutional modules and 3 pooling layers. The convolutional module consists of a convolutional layer with a convolution kernel size of 3×3, as well as normalization and activation operations.

[0037] The decoder consists of 3 convolutional modules and 3 inverse max pooling layers.

[0038] Furthermore, the original map is obtained and processed to obtain rasterized map features, including:

[0039] The road structure information in the original map is extracted and rasterized to obtain a rasterized map. VGG-13 is used as the feature extraction network to extract features from the rasterization and obtain rasterized map features of five different sizes, with corresponding downsampling ratios of {2, 4, 8, 16, 32}.

[0040] Furthermore, the enhanced image BEV features and enhanced radar BEV features are downsampled by the BEV encoder and fused with the rasterized map features to obtain fused BEV features, including:

[0041] The BEV encoder uses a 4-level feature pyramid to downsample the enhanced image BEV features and enhanced radar BEV features. It selects the features of the 1st, 2nd, and 3rd stages as the first fused features, and the rasterized map features with downsampling ratios of 2, 4, and 8 as the second fused features. The first and second fused features are fused and dimensionally restored through two convolution blocks and an upsampling operation to obtain the fused BEV features.

[0042] The convolution block consists of two serially connected 3×3 convolutional layers and an activation function; the upsampling operation uses bilinear interpolation.

[0043] Furthermore, the method includes:

[0044] The first fused feature and the second fused feature are connected to form a feature association block. In the feature association block, the first fused feature and the second fused feature are fused. The second fused feature is used as the query vector Q, and the first fused feature is used as the key vector K and the value vector V. The formula is as follows:

[0045]

[0046] in, Represent multi-scale features; Represent the mapping matrices of Q, K, and V of the i-th layer respectively; represents the channel dimension of Q and K;

[0047] Bilinear interpolation is used to upsample the features of size 1 / 8 and 1 / 4 in the multi-scale features, and match the features of size 1 / 2 to obtain the multi-scale aggregated features. The formula is as follows:

[0048]

[0049] in, Represents multi-scale aggregate features; Indicates the feature with a size of 1 / 8 in the multi-scale feature; Indicates a feature with a size of 1 / 4 in the multi-scale feature; Indicates the feature with size 1 / 2 in the multi-scale feature;

[0050] The first-stage features in the first fusion feature are connected with the multi-scale aggregation features by using a jump connection method to obtain the enhanced image BEV features. The same method is used to obtain the enhanced radar BEV features.

[0051] The enhanced image BEV features and enhanced radar BEV features are fused and dimensionally restored to obtain fused BEV features.

[0052] The bird's-eye view image generation method based on feature mutual enhancement and map prior provided in the embodiments of the present application has the following beneficial technical effects compared with the existing technology:

[0053] The embodiments of the present application fully utilize the complementarity between different data sources by fusing multiple information such as images, radars, and maps, thereby improving the accuracy of bird's-eye view generation; at the same time, through mutual enhancement and completion of features, as well as fine processing of fused features, the robustness of the model is further enhanced, enabling it to cope with complex and changing scenarios.

[0054] The embodiment of the present application proposes a bidirectional feature projection and enhancement mechanism, which can effectively map and fuse image features and radar features, solving the mismatch problem between different modal features; this multimodal feature fusion method enables the model to understand scene information more comprehensively, providing a richer feature basis for the generation of bird's-eye view.

[0055] The embodiment of the present application adopts an encoder-decoder structure, and performs operations such as dimensionality reduction, upsampling and reconstruction on features through components such as convolution modules, pooling layers, and inverse maximum pooling layers, thereby achieving effective feature processing and optimization; this processing method not only improves the expressiveness and utilization of features, but also makes the generated bird's-eye view have higher resolution and detail expression.

[0056] In the embodiment of the present application, the original map is processed to obtain rasterized map features, which are then integrated with image and radar features. The introduction of prior map knowledge provides additional constraints and guidance for the generation of a bird's-eye view, making the generated bird's-eye view more consistent with actual road scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flow chart of a method for generating a bird's-eye view based on feature mutual enhancement and map priors according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some, rather than all, structures related to the present application are shown in the accompanying drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0059] As used herein, the terms "comprise," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0060] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0061] like Figure 1 As shown, the embodiment of the present application provides a method for generating a bird's-eye view based on feature mutual enhancement and map prior, comprising the following steps:

[0062] Step 101 : Acquire original image data and original point cloud data, and pre-process the original image data and original point cloud data to obtain front view image features and radar BEV features.

[0063] In terms of backbone network selection, the embodiment of the present application processes the original image data through the Swin-T backbone network to obtain the front view image features; and processes the original point cloud data through the PointPillars backbone network to obtain the radar BEV features.

[0064] Specifically, for the camera branch, the Swin-T backbone network processes raw image data in four stages. As the number of stages increases, the size of the front view image features decreases, and the larger the receptive field of the features, the more blurred their spatial structure relationships become. For the radar branch, the PointPillars backbone network processes raw point cloud data in three stages: PointNet feature extraction, pseudo-imaging, and 2D convolution. During pseudo-imaging, the PointPillars backbone network performs average pooling on all point clouds belonging to the same voxel to accommodate the subsequent 2D convolution process. However, this operation inevitably results in the loss of specific point cloud coordinate information.

[0065] Most existing multimodal bird's-eye view (BEV) algorithms independently extract BEV features from images and point clouds. However, this approach has not been very effective for long-range BEV generation. For one thing, the camera branch requires pixel-by-pixel depth estimation to convert front-view image features into BEV features. However, as distance increases, object features such as texture and outline blur, making it extremely difficult to accurately infer the depth of each pixel based on the image's inherent features. Depth estimation often relies on clear and discernible visual cues, but distant objects occupy very few pixels in an image, resulting in limited features that contribute to depth estimation. Relying solely on such blurred image features for pixel-by-pixel depth estimation inevitably leads to significant errors, which in turn affects the accuracy of the conversion from front-view image features to BEV features. For another, for the lidar branch, point clouds exhibit significant sparsity as the sensing distance increases. This sparsity destroys the structural information in the point cloud, forcing it to rely more on features in the reflection intensity dimension, making it difficult to effectively identify distant objects. Since the model fails to obtain high-quality features on different branches, it is difficult to achieve good results using the fused features for the long-distance bird's-eye view generation task.

[0066] Therefore, an embodiment of the present application proposes a bidirectional feature projection method, which performs bidirectional feature projection on the image features obtained in the first stage of processing the original image data by the Swin-T backbone network and the lidar features obtained in the PointNet feature extraction stage of processing the original point cloud data by the PointPillars backbone network, to obtain a mapping relationship between the two modal features, and then fuses the features in different feature spaces and integrates the channel dimensions to obtain the front view image features and radar BEV features, respectively.

[0067] Specifically, we first need to map the lidar features to the front view image features. It is worth noting that since the sampling frequency of the radar and the camera are inconsistent, we must consider not only the spatial conversion but also the influence of the time factor.

[0068] Therefore, the embodiment of the present application converts the point cloud coordinates in the lidar feature from the lidar coordinate system to the vehicle coordinate system at the lidar timestamp, the global coordinate system, the vehicle coordinate system at the camera timestamp, and the camera coordinate system in sequence. This process is a rigid body transformation and can be achieved through translation and rotation operations. The formula is as follows:

[0069]

[0070] Where r represents the rotation matrix; t represents the translation matrix; P represents the point cloud coordinates in the laser radar coordinate system; Represents the point cloud coordinates in the camera coordinate system.

[0071] The conversion of point cloud coordinates in the camera coordinate system to point cloud coordinates in the pixel coordinate system needs to be realized with the help of camera intrinsic parameters. The formula is as follows:

[0072]

[0073] in, Represents the point cloud coordinates in the pixel coordinate system; K represents the camera intrinsic parameter.

[0074] However, since the image feature size in the embodiment of this application is 1 / 4 of the input image, this scale change will cause the camera intrinsic parameters to change accordingly. Therefore, it is necessary to process the camera intrinsic parameter matrix. The original camera intrinsic parameter formula is as follows:

[0075]

[0076] in, represents the focal length in the x direction; Indicates the focal length in the y direction; and Represents the optical center position. After the size is reduced to 1 / 4 of the original, the formula of the adjusted camera intrinsic parameter is as follows:

[0077]

[0078] Therefore, the conversion formula from the point cloud coordinates in the camera coordinate system to the point cloud coordinates in the pixel coordinate system can be obtained as follows:

[0079]

[0080] At this point, the projection of point cloud coordinates to image coordinates is completed. After obtaining the projection relationship between the two, the relationship between the point cloud coordinates and their corresponding image coordinates can be expressed as follows:

[0081]

[0082] in, Represents a set of laser radar coordinate points; Represents the corresponding image coordinate point set; represents the projection matrix; Represents a projective transformation.

[0083] The mapping from lidar features to front view image features is constructed through the Scatter operation. The formula is as follows:

[0084]

[0085] in, Features representing the LiDAR feature map to the image plane; Represents a lidar feature.

[0086] For the mapping of front view image features to lidar features, the embodiment of the present application samples the features of the 2D position projected by the 3D point through bilinear interpolation to obtain the mapping of the front view image features to the lidar features. The formula is as follows:

[0087]

[0088] in, Indicates the mapping of image features to lidar features; Represents the front view image features.

[0089] Step 102 : performing bidirectional feature enhancement on the front view image features and the lidar fusion features to obtain the front view image fusion features and the lidar fusion features.

[0090] After obtaining the mapping relationship between the two modal features, it is necessary to fuse the features in different feature spaces and integrate the channel dimensions to obtain the fused features.

[0091] For the image branch, feature fusion is performed using channel-dimensional splicing and convolution. Specifically, a convolutional layer with a kernel size of 3×3 is used to process the spliced ​​features. The convolutional layer here fuses the features mapped from the LiDAR feature map to the image plane with the front view image features and reduces the channel dimension of the spliced ​​features to achieve channel-dimensional matching. The formula is as follows:

[0092]

[0093] in, Represents the superposition of channel dimensions; Indicates a convolution operation with a convolution kernel size of 3×3; Represents the front view image fusion feature. The front view image fusion feature is not only richer in feature expression, but also achieves dimensional matching with the subsequent feature extraction network.

[0094] For the point cloud branch, the channel dimension splicing and convolution are also used to perform feature fusion. It is worth noting that after the channel dimension splicing, since the image features are mapped to the lidar features and the lidar features are in three-dimensional space at this time, they need to be pseudo-imaged and converted into two-dimensional features. The model here follows the PointPillars method, dividing the point cloud space into Q cylinders of infinite height, and performing average pooling on the point cloud features belonging to the same cylinder. This method reduces the complexity of the data while retaining the approximate information of the point cloud features of the same cylinder. After average pooling, each cylinder is represented by a unified feature vector. This feature vector integrates the feature information of all point clouds in the cylinder. At this time, the feature size is ,in and Corresponding respectively and The number of feature channels.

[0095] In order to map the above features into the bird's-eye view space and convert them into a 2D pseudo image, it is necessary to find the size of the bird's-eye view space based on the size of the cylinder, and find the position of each cylinder in the bird's-eye view space based on its coordinates, and convert the feature dimension into ,in represents the feature size of the bird's-eye view, which is related to the set perception range and the size of the voxel. In this embodiment, the size is 100×300.

[0096] The model obtains 2D pseudo-image features. Finally, to further fuse the two features and unify the channel dimensions, a convolution layer with a convolution kernel size of 3×3 is used to process the 2D pseudo-image, and finally the lidar fusion feature is obtained. The formula is as follows:

[0097]

[0098] in, Represents the superposition of channel dimensions; Indicates pseudo-image processing; Indicates a convolution operation with a convolution kernel size of 3×3; Represents the lidar fusion feature.

[0099] Step 103 : Perform perspective conversion on the front view image fusion features to obtain image BEV features.

[0100] After feature extraction by the backbone network, the front view image fusion feature and the lidar fusion feature are obtained. In order to obtain the image BEV feature, the front view image fusion feature needs to be converted from a perspective. The embodiment of the present application adopts the LSS method to explicitly estimate the depth of each pixel to obtain the depth distribution, and performs outer product operations and voxelization processing on the front view image fusion feature and the depth distribution to obtain the image BEV feature.

[0101] Step 104 : Enhance and complete the image BEV feature and the lidar fusion feature respectively using the front view image fusion feature to obtain an enhanced image BEV feature and an enhanced radar BEV feature.

[0102] After the features of the two modalities are obtained through the processing of the backbone network, the image BEV features and the lidar fusion features need to be converted into the bird's-eye view space. For the camera branch, the mapping process from the image BEV features to the bird's-eye view features will result in feature loss and blurring due to errors in depth estimation and information compression in the subsequent 3D to 2D pooling process. For the lidar branch, although there is no feature loss due to spatial conversion, its own structural characteristics determine that the farther the object is from the lidar, the fewer point clouds are scanned, resulting in the lidar fusion feature missing features for distant objects. At the same time, compared with the image BEV features, the front view image fusion features have more complete and rich semantic features; compared with the lidar fusion features, the front view image fusion features can obtain richer semantic information under long-distance conditions.

[0103] Therefore, the embodiment of the present application uses the front view image fusion feature to enhance and complete the image BEV feature and the lidar fusion feature respectively, so as to improve the overall performance of the bird's-eye view generation method of the embodiment of the present application.

[0104] The image BEV features and lidar fusion features are processed by the image branch and the laser point cloud branch respectively. The two branches have the same structure, consisting of an encoder, a cross-attention module and a decoder.

[0105] Specifically, the original features (image BEV features and lidar fusion features) are first input into the encoder for dimensionality reduction. The encoder consists of three convolutional modules and three pooling layers. The convolutional module consists of a 3×3 kernel, followed by normalization and activation operations. After processing by the encoder, the resulting downsampled features are 1 / 8 the original size, and the channel dimension is uniformly reduced to 256. It is noteworthy that downsampling is not performed directly using convolutions with a stride of 2. Instead, the original features are first convolved with a stride of 1 and then downsampled using max pooling. This is because downsampling directly using convolutions often results in a rapid loss of feature information. Increasing the stride will result in skipping more feature elements between adjacent convolutions. However, performing convolutions with a stride of 1 on the original features first allows for more robust capture of the correlations between different features while maintaining the dimensionality of the feature space. Then, downsampling is performed through maximum pooling. Maximum pooling selects the maximum value in a small area to represent the characteristics of the area. Relatively speaking, it is more selective in feature screening and can retain key local features to a certain extent while reducing the spatial dimension, reducing the loss of overall information, so that the downsampled features still retain sufficient rich information for subsequent processing and analysis.

[0106] After being processed by the encoder, the intermediate features (including intermediate image features and intermediate radar features) are obtained. The intermediate features are input into the cross-attention module, aiming to establish a deep association between the front view image fusion features and the lidar fusion features, and the front view image fusion features and the image BEV features, and use the front view image fusion features for enhancement and completion.

[0107] For the intermediate features and the front view image fusion features, we first convert the intermediate features into a query vector Q through dimensionality adjustment, and convert the front view image fusion features into a key vector K and a value vector V. We then obtain the relationship matrix between the intermediate features and the front view image fusion features through the inner product operation of the query vector Q and the key vector K. We then normalize the relationship matrix using the softmax activation function, and perform a weighted sum aggregation of the normalized relationship matrix and the value vector V to obtain the aggregated BEV features. The aggregated BEV features include image aggregated BEV features and radar aggregated BEV features. The formula is as follows:

[0108]

[0109] in, Indicates the channel dimension used for scaling.

[0110] Then, the aggregated BEV features are subjected to a convolution operation to reduce the channel dimension, and the aggregated BEV features and the intermediate features are connected in the channel dimension through a Concat operation, and finally subjected to a convolution, normalization and activation operation to obtain preliminary enhanced BEV features.

[0111] The intermediate features retain the basic feature structure and semantic information of the laser radar or image itself in the BEV space, and the aggregated BEV features incorporate supplementary information from the image features, and the preliminary enhanced BEV features obtained by splicing and fusing the two features have the advantages of both and provide a richer and higher quality feature basis for subsequent steps.

[0112] After obtaining the preliminary enhanced BEV features through the cross-attention module, the features need to be sent to the decoder to restore and reconstruct the fused features from the low-dimensional representation compressed by the encoder to the feature form conforming to the size requirements of the bird's eye view. The decoder is composed of 3 convolution modules and 3 reverse max-pooling layers. Among them, the 3 reverse max-pooling layers are responsible for gradually restoring the feature dimension in the decoder, and through the index information saved in the max-pooling process, the feature elements are put back to the original position, ensuring that the key information is not lost and the relative position relationship remains unchanged during the feature dimension restoration process. The convolution module is responsible for further refining and optimizing the features. By alternately using the convolution module and the reverse max-pooling layer, the conversion from the preliminary enhanced BEV features to the enhanced BEV features is completed, and the enhanced BEV features include enhanced image BEV features and enhanced radar BEV features.

[0113] In step 105, an original map is obtained, and the original map is processed to obtain a rasterized map feature.

[0114] Although the existing bird's eye view generation method has achieved many results, however, the perception range of the existing method is mostly limited within 50 meters. This is mainly because the performance limitation of the sensor itself leads to a lack of sufficient semantic information for the generation of long-distance bird's eye view, which further affects the performance of the algorithm. Therefore, the embodiment of the present application proposes a feature fusion method based on map prior, which provides road structure prior information through the rasterized map, and fuses and enhances the enhanced BEV features at different scales to improve the generation accuracy of long-distance bird's eye view.

[0115] For the task of extracting rasterized map features, in order to simplify the map and eliminate the influence of irrelevant map information on the bird's eye view production task, the embodiment of the present application only performs rasterization processing on the road structure information (road boundary, lane line, pedestrian crossing) in the map to obtain a rasterized map.

[0116] Specifically, the embodiment of the present application first obtains the original map, then extracts the road structure information in the original map, rasterizes the road structure information to obtain a rasterized map, uses VGG-13 as a feature extraction network, performs feature extraction on the rasterization, and obtains 5 rasterized map features of different sizes, with corresponding downsampling ratios of {2, 4, 8, 16, 32}, which include the road semantics and spatial structure information provided by the map.

[0117] Step 106 : Downsample the enhanced image BEV features and the enhanced radar BEV features respectively through a BEV encoder, and fuse them with the rasterized map features to obtain fused BEV features.

[0118] For enhanced BEV features, convolutional downsampling is performed to obtain multi-scale BEV features. For rasterized maps, feature extraction is performed to match feature dimensions. After obtaining multi-scale representations of different features, multi-level feature matching is performed, using rasterized map features to guide the enhanced BEV features across different feature dimensions. Finally, feature fusion is used to fuse and restore the enhanced BEV features to obtain fused BEV features.

[0119] First, feature extraction is required, including extracting rasterized map features and downsampling the enhanced BEV features to achieve dimensional matching between the two. For the enhanced BEV features, the embodiment of the present application downsamples them through the BEV encoder. The BEV encoder adopts a 4-level feature pyramid. After each level of convolution module, its height and width will be halved, and the number of channels will be doubled. For enhanced BEV features of different scales, the semantic information they focus on is different. Low-level features retain more details, such as the color of lane lines, texture features, etc.; high-level features focus more on the overall layout of the map, including the approximate distribution of map elements, the positional relationship between elements, etc.

[0120] Specifically, the 4-level feature pyramid corresponds to 4 processing stages. The embodiment of the present application selects the features of the 1st, 2nd and 3rd stages as the first fusion features, and selects the rasterized map features with downsampling ratios of 2, 4 and 8 as the second fusion features. The first fusion features and the second fusion features are fused and dimensionally restored through two convolution blocks and an upsampling operation to obtain a fused BEV feature; the convolution block consists of two serially connected 3×3 convolution layers and an activation function; the upsampling operation adopts bilinear interpolation.

[0121] Since the image branch, laser point cloud branch and map branch all provide high- and low-level features of different sizes. In order to better play the guiding role of map features, the embodiment of the present application will match according to the scale of the features, connect the first fused features and the second fused features, and form a feature association block. The fusion operation of the first fused features and the second fused features is performed in the feature association block. However, map accuracy issues and GPS deviations and other issues also bring difficulties to the prior information of the BEV feature fusion map, resulting in the bird's-eye view generation result being basically the same as the true value position, and it cannot be completely aligned with the grid map, and there is an obvious misalignment. This misalignment makes it impossible to obtain the best effect by directly adding or connecting operations at the feature level. In order to solve this problem, the embodiment of the present application adopts a cross-attention mechanism in the feature association block, and uses cross-attention to selectively use the second fused feature to enhance the related part of the first fused feature.

[0122] Specifically, the embodiment of the present application uses the second fused feature as the query vector Q of the cross attention, and uses the first fused feature as the key vector K and value vector V of the cross attention. The formula is as follows:

[0123]

[0124] in, Represent multi-scale features; Represent the mapping matrices of Q, K, and V of the i-th layer respectively; Indicates the channel dimension of Q and K.

[0125] After obtaining the multi-scale features, bilinear interpolation is used to upsample the features with sizes of 1 / 8 and 1 / 4 in the multi-scale features, and match the features with size of 1 / 2 to obtain the multi-scale aggregated features. The formula is as follows:

[0126]

[0127] in, Represents multi-scale aggregate features; Indicates the feature with a size of 1 / 8 in the multi-scale feature; Indicates a feature with a size of 1 / 4 in the multi-scale feature; Indicates the features with size 1 / 2 in multi-scale features.

[0128] Finally, the jump connection method is used to connect the features of the first stage in the first fusion feature with the multi-scale aggregation features to obtain the enhanced image BEV features. The formula is as follows:

[0129]

[0130] in, Represents enhanced image BEV features; The same method is used to obtain the enhanced radar BEV feature, which will not be repeated here.

[0131] Finally, the enhanced image BEV features and enhanced radar BEV features are fused and dimensionally restored to obtain the fused BEV features.

[0132] Specifically, this is accomplished through two convolutional blocks and an upsampling operation. The convolutional block fuses the features of the two modalities using two serially connected 3×3 convolutional layers and an activation function. The upsampling operation uses bilinear interpolation, performing reasonable interpolation calculations in the spatial dimension based on the existing feature information to restore the feature size. After this fusion process, the resulting fused BEV feature is a combination of image, laser point cloud, and map information.

[0133] Compared to methods that fuse image and lidar modalities, the map-prior-based feature fusion method proposed in this application embodiment additionally incorporates map information and uses rasterized map features to guide the enhancement of BEV features across different feature dimensions. This effectively enhances the remote representation capability of features and improves the accuracy of long-range bird's-eye view generation.

[0134] Step 107 : Upsample the fused BEV features through the decoder to obtain a bird's-eye view image generation result.

[0135] After the previous steps, we obtain fused BEV features, which contain a variety of information from images, radar, and maps. To generate the final bird's-eye view from these features, we need to use a decoder to upsample them.

[0136] The decoder is designed to restore and reconstruct features that were reduced in dimension during the encoder process. It includes components such as convolutional modules and reverse max pooling layers (or transposed convolutional layers). Among these components, reverse max pooling layers (or transposed convolutional layers) are the core, responsible for gradually restoring the spatial resolution of the fused BEV features to or near the original size, thereby generating a high-resolution bird's-eye view.

[0137] Specifically, the decoder first performs a series of convolution operations on the fused BEV features to further extract and integrate feature information. It then upsamples the feature maps through unmax pooling layers (or transposed convolution layers) to gradually restore their spatial resolution. During this process, the decoder may use techniques such as skip connections to incorporate intermediate features from the encoder, thereby helping to recover more detailed information.

[0138] Finally, after processing by the decoder, a high-resolution bird's-eye view image is generated. This result not only contains information from images and radar, but also incorporates prior knowledge of the map, resulting in higher accuracy and robustness.

[0139] It should be noted that the specific structure and parameter settings of the decoder may vary depending on the application scenario and dataset. In practical applications, we need to design and optimize the decoder according to specific needs and resources to obtain the best bird's-eye view generation effect.

[0140] It should be noted that, in the present application, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0141] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for generating a bird's-eye view image based on feature mutual enhancement and map prior, characterized in that: The method comprises: Acquire raw image data and raw point cloud data, and preprocess the raw image data and the raw point cloud data, including processing the raw image data through a Swin-T backbone network to obtain front view image features; and processing the raw point cloud data through a PointPillars backbone network to obtain radar BEV features; The process of processing the original image data by the Swin-T backbone network is divided into four stages, and as the number of stages increases, the size of the front view image features continues to decrease; The process of processing the original point cloud data by the PointPillars backbone network is divided into three stages, including PointNet feature extraction, pseudo-imaging and 2D convolution processing; Performing bidirectional feature enhancement on the front view image features and the radar BEV features to obtain front view image fusion features and lidar fusion features, including performing bidirectional feature projection on the image features obtained in the first stage of processing the original image data by the Swin-T backbone network and the lidar features obtained in the PointNet feature extraction stage of processing the original point cloud data by the PointPillars backbone network to obtain a mapping relationship between the two modal features, and then fusing features in different feature spaces and integrating channel dimensions to obtain front view image fusion features and lidar fusion features, respectively; Performing perspective conversion on the front view image fusion feature to obtain an image BEV feature; The image BEV feature and the lidar fusion feature are respectively enhanced and completed by using the front view image fusion feature to obtain an enhanced image BEV feature and an enhanced radar BEV feature; Obtaining an original map, and processing the original map to obtain rasterized map features; Downsampling the enhanced image BEV feature and the enhanced radar BEV feature respectively through a BEV encoder, and fusing them with the rasterized map feature to obtain a fused BEV feature; The fused BEV features are upsampled by a decoder to obtain a bird's-eye view generation result.

2. The method for generating a bird's-eye view based on feature mutual enhancement and map prior according to claim 1, characterized in that: In the bidirectional feature projection, the mapping of the lidar features to the front view image features includes: The point cloud coordinates in the lidar feature are converted from the lidar coordinate system to the vehicle coordinate system at the lidar timestamp, the global coordinate system, the vehicle coordinate system at the camera timestamp, the camera coordinate system, and the pixel coordinate system in sequence, and the camera intrinsic parameter matrix is ​​reduced to 1 / 4 of its original size to complete the projection of the point cloud coordinates to the image coordinates; After the projection of the point cloud coordinates to the image coordinates is completed, the mapping of the lidar features to the image features is constructed through the Scatter operation. The formula is as follows: in, Represents the features mapped from the lidar feature to the image plane; Represents the image coordinate point set corresponding to the lidar coordinates; Represents a lidar feature.

3. The method for generating a bird's-eye view based on feature mutual enhancement and map prior according to claim 1 or 2, characterized in that: In the bidirectional feature projection, the mapping of the front view image features to the lidar features includes: The features of the 2D position projected by the 3D point are sampled by bilinear interpolation to obtain the mapping of the front view image features to the lidar features. The formula is as follows: in, Indicates the mapping of image features to lidar features; Represents the image coordinate point set corresponding to the lidar coordinates; Represents the front view image features.

4. The method for generating a bird's-eye view based on feature mutual enhancement and map prior according to claim 1, wherein: The method of enhancing and completing the image BEV feature and the lidar fusion feature respectively by using the front view image fusion feature to obtain an enhanced image BEV feature and an enhanced radar BEV feature includes: Performing a dimensionality reduction operation on the image BEV feature and the lidar fusion feature through an encoder to obtain intermediate features, wherein the intermediate features include intermediate image features and intermediate radar features; Resizing the dimensions of the intermediate features and the front view image fusion features by cross attention, converting the intermediate features into a query vector Q, and converting the front view image fusion features into a key vector K and a value vector V; obtaining a relationship matrix between the intermediate features and the front view image fusion features by performing an inner product operation on the query vector Q and the key vector K; Normalizing the relationship matrix through a softmax activation function, and weighting and aggregating the value vector V using the normalized relationship matrix to obtain aggregated BEV features, wherein the aggregated BEV features include image aggregated BEV features and radar aggregated BEV features; Performing a convolution operation on the aggregated BEV features to reduce the channel dimension, connecting the intermediate features and the aggregated BEV features in the channel dimension through a Concat operation, and performing convolution, normalization, and activation operations on the connected features to obtain preliminary enhanced BEV features; The preliminary enhanced BEV features are restored and reconstructed by a decoder to obtain enhanced BEV features, where the enhanced BEV features include enhanced image BEV features and enhanced radar BEV features.

5. The method for generating a bird's-eye view based on feature mutual enhancement and map prior according to claim 4, characterized in that: The encoder consists of three convolutional modules and three pooling layers. The convolutional module consists of a convolutional layer with a convolution kernel size of 3×3, as well as normalization and activation operations. The decoder consists of 3 convolutional modules and 3 inverse max pooling layers.

6. The method for generating a bird's-eye view based on feature mutual enhancement and map prior according to claim 1, characterized in that: The obtaining of the original map and processing the original map to obtain rasterized map features include: The road structure information in the original map is extracted and rasterized to obtain a rasterized map. VGG-13 is used as a feature extraction network to perform feature extraction on the rasterization to obtain rasterized map features of five different sizes, corresponding to downsampling ratios of {2, 4, 8, 16, 32}.

7. The method for generating a bird's-eye view based on feature mutual enhancement and map prior according to claim 6, characterized in that: The downsampling of the enhanced image BEV feature and the enhanced radar BEV feature by the BEV encoder is respectively performed, and the downsampling is performed with the rasterized map feature to obtain a fused BEV feature, including: The BEV encoder downsamples the enhanced image BEV features and the enhanced radar BEV features using a 4-level feature pyramid, selects features from the first, second, and third stages as first fused features, selects rasterized map features with downsampling ratios of 2, 4, and 8 as second fused features, and fuses and restores the dimensions of the first and second fused features through two convolution blocks and an upsampling operation to obtain a fused BEV feature. The convolution block consists of two serially connected 3×3 convolution layers and an activation function; the upsampling operation adopts bilinear interpolation.

8. The method for generating a bird's-eye view based on feature mutual enhancement and map prior according to claim 7, characterized in that: The method comprises: The first fused feature and the second fused feature are connected to form a feature association block. In the feature association block, the first fused feature and the second fused feature are fused. The second fused feature is used as the query vector Q, and the first fused feature is used as the key vector K and the value vector V. The formula is as follows: in, Represent multi-scale features; Represent the mapping matrices of Q, K, and V of the i-th layer respectively; represents the channel dimension of Q and K; The features with sizes of 1 / 8 and 1 / 4 in the multi-scale features are upsampled by bilinear interpolation, and the features with size of 1 / 2 are matched to obtain the multi-scale aggregated features. The formula is as follows: Among them, the Represents multi-scale aggregate features; Indicates the feature with a size of 1 / 8 in the multi-scale feature; Indicates a feature with a size of 1 / 4 in the multi-scale feature; Indicates the feature with size 1 / 2 in the multi-scale feature; The first-stage features in the first fusion features are connected with the multi-scale aggregation features by using a skip connection method to obtain enhanced image BEV features; the same method is used to obtain enhanced radar BEV features; The enhanced image BEV feature and the enhanced radar BEV feature are fused and dimensionally restored to obtain a fused BEV feature.

Images