Method, device, vehicle and storage medium for generating a bird's-eye view of a vehicle

Abstract

The present application relates to the field of autonomous driving technology, and in particular to a method, device, vehicle, and storage medium for generating a bird's-eye view of a vehicle, wherein the method comprises: obtaining multiple perception information around the vehicle and extracting corresponding two-dimensional feature maps; performing geometric transformation on each two-dimensional feature map according to a preset transformation strategy to obtain a first feature map and performing feature fusion and spatial conversion to obtain single-frame feature maps at different moments; performing fusion of spatial dimensions and temporal dimensions in the vehicle coordinate system at the current moment to generate a second feature map; obtaining dynamic and static obstacle features around the vehicle based on the second feature map, and generating a bird's-eye view around the vehicle based on the dynamic and static obstacle features. Thus, the present application can receive raw two-dimensional image inputs and / or laser radar inputs from multiple cameras surrounding the vehicle body 360 degrees, and directly output perception information from the bird's-eye view perspective, reducing the workload of post-processing and improving the robustness of the entire system.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method, device, vehicle, and storage medium for generating a bird's-eye view of a vehicle. Background Art

[0002] The autonomous driving system is one of the core systems of smart cars. The perception module is the upstream of the entire system, and its perception quality directly affects the performance of the entire autonomous driving system.

[0003] In related technologies, the perception module architecture of most OEMs and autonomous driving solution providers is based on post-perception fusion, that is, the images of a single camera are perceived separately, and then the perception results are post-fused using logic-based post-processing.

[0004] However, this framework suffers from numerous issues, the most prominent of which is the need for rule-based post-processing, which is complex and intricate for the vision module. Consequently, the solution suffers from poor robustness and is unable to handle some critical conditions. This is particularly true for complex road structures like intersections and roundabouts, where the post-fusion processing accuracy is low. Furthermore, post-fusion also exhibits limited robustness in scenarios involving cut-ins of dynamic obstacles. Summary of the Invention

[0005] The present application provides a method, device, vehicle and storage medium for generating a bird's-eye view of a vehicle, which can receive raw two-dimensional image inputs from multiple cameras and / or lidar inputs surrounding the vehicle body 360 degrees, and directly output perception information from a bird's-eye view perspective, thereby reducing the workload of post-processing and improving the robustness of the entire system.

[0006] The first aspect of the present application provides a method for generating a bird's-eye view of a vehicle, comprising the following steps: acquiring multiple perception information around the vehicle; extracting a two-dimensional feature map of each perception information, and performing a geometric transformation on each two-dimensional feature map according to a preset transformation strategy to obtain a first feature map; performing feature fusion and spatial transformation on all first feature maps to obtain single-frame feature maps at different moments, and performing spatial and temporal fusion of the single-frame feature maps at different moments in the vehicle coordinate system at the current moment to generate a second feature map; acquiring dynamic obstacle features and static obstacle features around the vehicle according to the second feature map, and generating a bird's-eye view around the vehicle based on the dynamic obstacle features and the static obstacle features.

[0007] According to the above technical means, the embodiment of the present application can receive raw two-dimensional image inputs and / or lidar inputs from multiple cameras surrounding the vehicle body 360 degrees, and directly output perception information from a bird's-eye view perspective, thereby reducing the workload of post-processing and improving the robustness of the entire system.

[0008] Furthermore, the geometric transformation of each two-dimensional feature map according to a preset transformation strategy to obtain a first feature map includes: obtaining corresponding points on the first feature map and each two-dimensional feature map; and applying the corresponding points to each two-dimensional feature map according to a pre-constructed inverse perspective transformation matrix to obtain the first feature map.

[0009] According to the above-mentioned technical means, the embodiment of the present application can use the intrinsic and extrinsic parameters of the camera to construct an inverse perspective transformation matrix, obtain the corresponding point of each point on the bird's-eye view feature map on the two-dimensional feature map, and obtain the first feature map of the bird's-eye view according to the inverse perspective transformation matrix. In this way, the spatial homography transformation can be applied to the two-dimensional feature map, effectively utilizing the original image information, and the feature map generation result is more accurate and stable.

[0010] Furthermore, the feature fusion and spatial transformation of all the first feature maps to obtain single-frame feature maps at different moments include: extracting the feature relationship between each first feature map, performing feature fusion based on the feature relationship to obtain the single-frame feature map at the current moment; converting the single-frame feature map at the current moment and the single-frame feature maps at multiple historical moments to the vehicle coordinate system at the current moment to obtain single-frame feature maps at different moments.

[0011] According to the above-mentioned technical means, the embodiment of the present application can introduce timing information so that all features at different moments are in the same unified coordinate system (i.e., the vehicle body coordinate system at the current moment). In this way, a single-frame feature map after feature fusion can be accurately and effectively obtained through a simpler fusion method, so that the feature information can be effectively overlapped without being misplaced, the operation is simpler and faster, and the fusion efficiency and the accuracy of the fusion results are improved.

[0012] Furthermore, obtaining dynamic obstacle features and static obstacle features around the vehicle based on the second feature map includes: inputting the second feature map into a pre-established first decoder, decoding the dynamic obstacle semantic information on the second feature map, and obtaining the dynamic obstacle features; and inputting the second feature map into a pre-established second decoder, decoding the static obstacle semantic information on the second feature map, and obtaining the static obstacle features.

[0013] According to the above-mentioned technical means, the embodiment of the present application can realize rotating box detection from a bird's-eye view perspective through dynamic decoding and static decoding, and output the tangent direction of each point based on point detection and point instance clustering, and vectorize the static road information through simple post-processing. This can effectively solve the problem of complex post-processing, reduce the workload of perception post-processing, and directly output all necessary information required for planning and control from a bird's-eye view perspective. The decoding process is more complete and accurate, thereby improving the robustness of the entire perception system.

[0014] Furthermore, generating a bird's-eye view of the vehicle's surroundings based on the dynamic obstacle features and the static obstacle features includes: acquiring point cloud data from the vehicle's lidar; inputting the point cloud data into a pre-established point cloud feature extraction model, and outputting point cloud features of the point cloud data; and fusing the point cloud features, the dynamic obstacle features, and the static obstacle features to obtain a bird's-eye view of the vehicle's surroundings.

[0015] According to the above-mentioned technical means, the embodiment of the present application can use the lidar input as a selection branch. In addition to realizing multi-perspective fusion, it can also simultaneously fuse lidar and vision, process multimodal information, improve the comprehensive information processing capability, and consider the features of bird's-eye view generation more comprehensively, thereby improving the accuracy and comprehensiveness of the generated results.

[0016] The second aspect of the present application provides a device for generating a bird's-eye view of a vehicle, including: an acquisition module for acquiring multiple perception information around the vehicle; a first transformation module for extracting a two-dimensional feature map of each perception information, and performing a geometric transformation on each two-dimensional feature map according to a preset transformation strategy to obtain a first feature map; a second transformation module for performing feature fusion and spatial transformation on all first feature maps to obtain single-frame feature maps at different times, and performing spatial and temporal fusion of the single-frame feature maps at different times in the vehicle coordinate system at the current time to generate a second feature map; a generation module for acquiring dynamic obstacle features and static obstacle features around the vehicle according to the second feature map, and generating a bird's-eye view around the vehicle based on the dynamic obstacle features and the static obstacle features.

[0017] Furthermore, the first transformation module is used to: obtain corresponding points on the first feature map and each two-dimensional feature map; apply the corresponding points to each two-dimensional feature map according to a pre-constructed inverse perspective transformation matrix to obtain the first feature map.

[0018] Furthermore, the second transformation module is used to: extract the feature relationship between each first feature map, perform feature fusion based on the feature relationship, and obtain a single-frame feature map at the current moment; convert the single-frame feature map at the current moment and the single-frame feature maps at multiple historical moments into the vehicle coordinate system at the current moment to obtain single-frame feature maps at different moments.

[0019] Furthermore, the generation module is used to: input the second feature map into a pre-established first decoder, decode the dynamic obstacle semantic information on the second feature map, and obtain the dynamic obstacle feature; input the second feature map into a pre-established second decoder, decode the static obstacle semantic information on the second feature map, and obtain the static obstacle feature.

[0020] Furthermore, the generation module is further used to: obtain point cloud data of the vehicle's lidar; input the point cloud data into a pre-established point cloud feature extraction model, and output point cloud features of the point cloud data; and fuse the point cloud features, the dynamic obstacle features, and the static obstacle features to obtain a bird's-eye view of the area around the vehicle.

[0021] The third aspect of the present application provides a vehicle, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for generating a bird's-eye view of the vehicle as described in the above embodiment.

[0022] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the method for generating a bird's-eye view of a vehicle as described in the above embodiment.

[0023] Therefore, this application has at least the following beneficial effects:

[0024] (1) The embodiments of the present application can receive raw two-dimensional image inputs from multiple cameras and / or lidar inputs that surround the vehicle body 360 degrees, and directly output perception information from a bird's-eye view perspective, thereby reducing the workload of post-processing and improving the robustness of the entire system.

[0025] (2) The embodiment of the present application can use the intrinsic and extrinsic parameters of the camera to construct an inverse perspective transformation matrix, obtain the corresponding point of each point on the bird's-eye view feature map on the two-dimensional feature map, and obtain the first feature map of the bird's-eye view according to the inverse perspective transformation matrix. In this way, the spatial homography transformation can be applied to the two-dimensional feature map, effectively utilizing the original image information, and the feature map generation result is more accurate and stable.

[0026] (3) The embodiment of the present application can introduce time series information so that all features at different times are in the same unified coordinate system (i.e., the vehicle body coordinate system at the current moment). As a result, a single-frame feature map after feature fusion can be accurately and effectively obtained through a simpler fusion method, so that the feature information can be effectively overlapped without being misplaced, the operation is simpler and faster, and the fusion efficiency and the accuracy of the fusion results are improved.

[0027] (4) The embodiments of the present application can realize rotating frame detection from a bird's-eye view perspective through dynamic decoding and static decoding, and output the tangent direction of each point based on point detection and point instance clustering. The static road information is vectorized through simple post-processing, thereby effectively solving the problem of complex post-processing, reducing the workload of perception post-processing, and directly outputting all necessary information required for planning and control from a bird's-eye view perspective. The decoding process is more complete and accurate, thereby improving the robustness of the entire perception system.

[0028] (5) The embodiment of the present application can use the lidar input as a selection branch. In addition to realizing multi-perspective fusion, it can also simultaneously fuse lidar and vision, process multimodal information, improve the comprehensive information processing capability, consider the features of bird's-eye view generation more comprehensively, and improve the accuracy and comprehensiveness of the generated results.

[0029] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0031] Figure 1 This is a diagram of a multi-view multi-modal conventional post-fusion processing architecture provided according to an embodiment of the present application;

[0032] Figure 2 This is a diagram of a multi-view multi-modal pre-fusion processing architecture provided according to an embodiment of the present application;

[0033] Figure 3 This is a flowchart of a method for generating a bird's-eye view of a vehicle according to an embodiment of the present application;

[0034] Figure 4 Schematic diagram of a geometric transformation module under the condition of 2D=>BEV according to an embodiment of the present application;

[0035] Figure 5 Schematic diagram of a transformer encoder module provided according to an embodiment of the present application;

[0036] Figure 6 Schematic diagram of a geometric transformation module under the condition that frame t-1 => frame t according to an embodiment of the present application;

[0037] Figure 7 A schematic diagram of a three-dimensional convolution module provided according to an embodiment of the present application;

[0038] Figure 8 A schematic diagram of a dynamic obstacle branch provided according to an embodiment of the present application;

[0039] Figure 9 This is a flow chart of dynamic obstacle post-processing according to an embodiment of the present application;

[0040] Figure 10 This is a prediction effect diagram provided according to an embodiment of the present application;

[0041] Figure 11 This is a flowchart of post-processing of static road surface information detection provided according to an embodiment of the present application;

[0042] Figure 12 A schematic diagram of a static road surface information detection branch provided according to an embodiment of the present application;

[0043] Figure 13 A schematic diagram of an additional input module for a laser radar according to an embodiment of the present application;

[0044] Figure 14 A schematic diagram of the overall model architecture provided according to an embodiment of the present application;

[0045] Figure 15 This is a diagram of the software architecture of a multi-view multi-modal front-fusion bird's-eye view according to an embodiment of the present application;

[0046] Figure 16 This is an example diagram of a device for generating a bird's-eye view of a vehicle according to an embodiment of the present application;

[0047] Figure 17 Schematic diagram of the structure of a vehicle according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0049] In related technologies, the perception module architecture of most OEMs and autonomous driving solution providers is based on post-perception fusion, such as Figure 1 As shown in the figure, the multi-view multi-modal front fusion solution is a new trend that can effectively solve many problems of the above-mentioned related technologies, such as poor perception of complex road structures, field of view occlusion, poor post-processing robustness, and lack of complete perception information downstream. Among them, the main approaches of multi-view multi-modal front fusion BEV are as follows:

[0050] (1) Based on spatial transformer: This paper proposes to use spatial transformer to geometrically transform the features on 2D (2-dimensional) images, with the target space being the BEV (Bird's Eye View) space. Then, multiple transformed BEV feature maps are spliced by channel to achieve multi-view fusion. However, the output of this method is only the original semantic segmentation map, which requires more complex post-processing.

[0051] (2) Based on visual point cloud: This fusion method first generates a depth map on each 2D feature map, and uses the depth map to convert the feature points on each feature map into 3D (3-dimensional) space, so that each map generates a corresponding visual pseudo point cloud. Then, each cluster of pseudo point clouds is converted to a unified vehicle coordinate system using camera extrinsic parameters to achieve multi-view fusion. Finally, the point cloud processing operator is used to convert the point cloud features into bird's-eye view features. This method has natural fusion, but because it depends a priori on the accuracy of depth estimation, errors are easily accumulated and propagated, and the pseudo point cloud will jump, making the perception results unstable.

[0052] (3) Based on MLP (Multilayer Perceptron): This method directly uses MLP to replace the spatial transformer in (1). It uses MLP to convert perspective features into BEV features. Then, it uses camera extrinsics to aggregate all camera BEV features into a unified vehicle coordinate system to obtain BEV features. This fusion method also does not require depth estimation. Multi-view fusion is more direct and simple. However, since the camera intrinsic parameters are not input as a priori, the detection effect is easily affected once the camera intrinsic parameters change.

[0053] (4) BEV decoder: Most current solutions output semantic segmentation maps from the BEV perspective, but the semantic segmentation results are prone to noise, and the post-processing is based on logic and is not very stable.

[0054] The multi-view multi-modal front-fusion BEV model in the above-mentioned related technologies has a fusion strategy based on spatial transformation, visual point cloud or MLP; however, the fusion method based on spatial transformation simply concatenates multiple features, and its decoding strategy is mostly semantic segmentation, which still has the problem of complicated post-processing.

[0055] To this end, an embodiment of the present application proposes a method, device, vehicle and storage medium for generating a bird's-eye view of a vehicle. The method, device, vehicle and storage medium for generating a bird's-eye view of a vehicle in an embodiment of the present application will be described below with reference to the accompanying drawings.

[0056] Specifically, Figure 3 A flowchart of a method for generating a bird's-eye view of a vehicle provided in an embodiment of the present application.

[0057] like Figure 3 As shown, the method for generating a bird's-eye view of a vehicle includes the following steps:

[0058] In step S101, a plurality of perception information around the vehicle is acquired.

[0059] Among them, the embodiment of the present application can obtain the perception information around the vehicle through at least one method, which is not specifically limited.

[0060] It is understandable that the embodiments of the present application can obtain multiple perception information around the vehicle for use in subsequent steps to generate a bird's-eye view of the vehicle.

[0061] In step S102, a two-dimensional feature map of each perception information is extracted, and a geometric transformation is performed on each two-dimensional feature map according to a preset transformation strategy to obtain a first feature map.

[0062] Among them, the preset transformation strategy can be specifically set according to actual conditions, and there is no specific limitation on this; for example, the embodiment of the present application can use geometric transformation to first generate samplegrid (perspective transformation matrix), and then apply the gridsample (inverse perspective transformation matrix) operation to the 2D feature map, thereby obtaining the feature map on the BEV, etc.

[0063] It is understandable that the embodiment of the present application can refer to some solutions based on spatial transformer to transform the 2D image to obtain the first feature map; for example, the embodiment of the present application can design an STN (SpatialTransform Network, geometric transformation) module to achieve the above-mentioned generation of the first feature map. Among them, the geometric transformation of the embodiment of the present application can include two steps: first, the intrinsic and extrinsic parameters of the camera are used to construct an inverse perspective transformation matrix to obtain the corresponding point of each point on the BEV feature map on the 2D feature map, that is, to generate a samplegrid; then, the obtained samplegrid is used to apply the gridsample operation to the 2D feature map, thereby obtaining the feature map on the BEV.

[0064] It should be noted that for a monocular camera with a certain viewing angle, its imaging surface is a 2D perspective plane. Each pixel on the imaging surface is formed by projecting a point in the world coordinate system. However, the front-fusion BEV model in the embodiment of this application does not perceive the final result in the 2D perspective plane as described above, but in the BEV. Therefore, the embodiment of this application uses the STN module to geometrically transform the feature points of the 2D perspective plane coordinate system to the BEV. The geometric transformation process of the embodiment of this application is described in detail below.

[0065] In an embodiment of the present application, each two-dimensional feature map is geometrically transformed according to a preset transformation strategy to obtain a first feature map, including: obtaining corresponding points on the first feature map and each two-dimensional feature map; applying the corresponding points to each two-dimensional feature map according to a pre-constructed inverse perspective transformation matrix to obtain the first feature map.

[0066] The inverse perspective transformation matrix can be obtained by specific settings according to actual conditions, and is not specifically limited thereto.

[0067] It should be noted that the input of the STN module is not the original 2D image, but the 2D feature map after the feature extraction module extracts the features. The STN module of this application mainly has two submodules, such as Figure 4 As shown in the figure, the first module is the affinegrid module, and the second module is the gridsample module; the Affinegrid module obtains the corresponding coordinate point of each point in the original perspective image under the BEV perspective through affine transformation.

[0068] 1. Affine grid (image affine) module: For a point (u feat_bev ,v feat_bev )∈W feat_bev ×H feat_bev , which is in the 2D feature map W feat_img ×H feat_img The corresponding point (u feat_img ,v feat_img )The calculation process is as follows:

[0069]

[0070]

[0071]

[0072] For formula (1), Z c is the depth of a certain point, ratio downsample is the downsampling multiple of the feature extraction module. K, R, and T can all be given by the camera’s internal and external parameter calibration, which are the intrinsic parameter matrix, extrinsic rotation matrix, and extrinsic translation matrix of the monocular camera, respectively. The three are combined and recorded as P matrix, which is the camera projection matrix.

[0073] For formula (2) and (3), And W bev and H bev The width and height of the final output BEV canvas, ppx bev and ppy bev is the resolution of the BEV canvas, and the M matrix connects the BEV pixel coordinates and the BEV world coordinates. Formula (4) is derived from formulas (1), (2), and (3):

[0074]

[0075] Therefore, from formula (4), we can see that the point (u feat_bev ,v feat_bev)∈W feat_bev ×H feat_bev , the corresponding points (u feat_img ,v feat_img ). And, for each feature point u on the BEV feature map feat_bev =0,1,2,...,W feat_bev , v feat_bev =0,1,2,...,H feat_bev , the corresponding feature points can be obtained, so all such correspondences generate a samplegrid matrix with a dimension of [H feat_bev ,W feat_bev ,2], denoted as TENSOR_GRID.

[0076] 2. Gridsample module: This module uses the TENSOR_GRID calculated above to perform gridsample on the 2D feature map. Specifically, for a point (u feat_bev ,v feat_bev )∈W feat_bev ×H feat_bev , which can correspond to the four coordinate values x0, y0, x1, y1 of the input 2D feature map, and is calculated as follows:

[0077] x base ,y base =TENSOR_GRID[u feat_bev ,v feat_bev ,:] (5)

[0078] x0,y0=(int(pt base [0]),int(pt base [1])) (6)

[0079] x1,y1=(int(pt base [0])+1,int(pt base [1])+1) (7)

[0080] Then, based on these four points, bilinear interpolation is performed on each eigenvector. The interpolation formula is as follows:

[0081]

[0082] Among them, the FeatOnImg dimension is (C feat ,H feat_img ,W feat_img ), FeatOnBEV is the output BEV feature map, the dimension is (C feat ,H feat_bev,W feat_bev ).

[0083] According to the calculation process of the above two modules, the STN module proposed in the embodiment of the present application first uses formula (4) to obtain the corresponding point of each point on the BEV feature map on the 2D feature map, that is, generates a samplegrid; then uses formula (8) to apply the gridsample operation to the 2D feature map, thereby obtaining the feature map on the BEV.

[0084] In step S103, feature fusion and spatial transformation are performed on all first feature maps to obtain single-frame feature maps at different times, and the single-frame feature maps at different times are fused in spatial and temporal dimensions in the vehicle coordinate system at the current time to generate a second feature map.

[0085] It is understandable that the embodiment of the present application can use a multi-perspective fusion transformer encoding method to fuse the intangible cultural heritage feature map, and use a geometric transformation method to perform spatial conversion and obtain single-needle feature maps at different times; the embodiment of the present application can also use a three-dimensional convolution method of temporal fusion to achieve the fusion of spatial and temporal dimensions of single-frame feature maps at different times in the current vehicle coordinate system. The multi-perspective fusion method and temporal fusion method of the embodiment of the present application will be specifically explained below in conjunction with specific embodiments.

[0086] 1. Transformer Encoder module for multi-view fusion:

[0087] In an embodiment of the present application, feature fusion and spatial transformation are performed on all first feature maps to obtain single-frame feature maps at different moments, including: extracting the feature relationship between each first feature map, performing feature fusion based on the feature relationship, and obtaining a single-frame feature map at the current moment; converting the single-frame feature map at the current moment and the single-frame feature maps at multiple historical moments to the vehicle coordinate system at the current moment to obtain single-frame feature maps at different moments.

[0088] Among them, the embodiment of the present application can use at least one method to extract the feature relationship between each first feature graph. For example, the embodiment of the present application can use a self-attention module to effectively extract the correlation relationship between each BEV graph.

[0089] It can be understood that STN transforms a single 2D feature map, and each view will generate a BEV feature map after STN transformation. Therefore, the embodiment of the present application can set up a multi-view fusion Transformer Encoder module. The function of this module is to perform a feature-level deep fusion of all single BEV maps, and finally output the memory as the final single-frame feature map of deep fusion, which is sent to the subsequent timing module.

[0090] Specifically, after the STN transformation in step S101 above, a bev feature map is obtained under a single perspective. It can be seen that after N perspective transformations, N FeatOnBEVi (i=1,2,…,N) in the bev coordinate system will be generated. The function of this multi-perspective fusion Transformer Encoder module is to perform a feature-level deep fusion of all FeatOnBEVi, and the Transformer can be used as a deep fusion structure. Since this method only focuses on feature fusion and does not use the Transformer for information decoding, the feature fusion module of this application only takes the standard Transformer Encoder part and obtains the fused memory as the final single-frame BEV feature.

[0091] The TransformerEncoder of this application includes a PositionEmbedding module and multiple TransformerLayer (transformer layer), where each TransformerLayer performs actual feature fusion, including a Multi-head Self-Attention (multi-head mechanism, self-attention mechanism) module and a Feedforward (forward feed transmission) module. The input structure of the TransformerEncoder model is as follows:

[0092] (1) Concat N FeatOnBEVi according to Channel (concat() function, connect two or more arrays) to obtain the concat feature map FeatBEVConcat, whose dimension is (C feat ×N,H feat_bev ,W feat_bev );

[0093] (2) Expand FeatBEVConcat in width and height dimensions to obtain FeatBEVExpand, whose dimensions are (C feat ×N,H feat_bev ×W feat_bev );

[0094] (3) Transform the dimension to obtain the standard form FeatBEVin that can be input into the Transformer Encoder, whose dimension is (H feat_bev ×W feat_bev ,1,C feat ×N);

[0095] (4) Then use the BEV feature map dimension (C feat ,H feat_bev ,Wfeat_bev ), calculate the EmbedVector through the Embedding module, whose dimension is (H feat_bev ×W feat_bev ,1,C feat ×N).

[0096] Through the above steps, the required input vector of TransformerEncoder is obtained. The entire TransformerEncode module diagram can be shown as follows: Figure 5 After the input tensor passes through multiple layers of TransformerLayer, a fused bev feature is finally obtained, denoted as BEVMemory, whose spatial dimension is the same as the input spatial dimension, and the number of channels is amplified N times, which is (C feat ×N,H feat_bev ,W feat_bev ).

[0097] 2. STN module of time series fusion:

[0098] In order to improve the model perception performance, this application can further introduce time series information. Specifically, Figure 6 As shown, for each frame Frame1, Frame2, ..., FrameM (Note: each frame contains multiple perspectives, where FrameM represents the current moment frame) before the M moment, the aggregated features, namely BEVMemory1, BEVMemory2, ..., BEVMemoryM, can be extracted through the above steps. However, since BEVMemoryi is Coordi relative to the vehicle coordinate system Coord at moment i, in order to convert all BEVMemory to the vehicle coordinate system CoordM at the current moment (that is, moment M), STN transformation is required for BEVMemoryi. Compared with the STN in the first step, this STN transformation converts the space from Coordi to CoordM (instead of 2D to BEV).

[0099] Specifically, let the position of the vehicle at time i be (R i ,T i ), then for a point p in the world coordinate system, its coordinate p in the vehicle coordinate system Coordi at the i-th moment is i and the coordinate p of the i+1th vehicle body coordinate system Coord(i+1) i+1 The relationship is:

[0100] R i p i +T i =R i+1 p i+1 +T i+1 (9)

[0101] After sorting, we get:

[0102]

[0103] Formula (10) describes the transformation relationship of the same point in the Coordi and Coord(i+1) coordinate systems. Simplifying Formula (10) by writing it in the form of a homogeneous coordinate system, we have:

[0104]

[0105] For BEVMemoryi, STN is used to transform it into BEV features in the coordinate system CoordM, and its transformation matrix T M,i The following forms of continuous multiplication are available:

[0106] T M,i =T M,M-1 ...T i+2,i+1 T i+1,i (12)

[0107]

[0108] Using (11), for each point (u feat_bevM ,v feat_bevM ), its coordinates under Coordi (u feat_bevi ,v feat_bevi ), that is, TENSOR_GRID is calculated, and the subsequent calculation is the same as the STN module in the first step to perform bilineargridsample. Finally, the BEVMemory at the 1st, 2nd..., M-1th moment is converted to the coordinate system at the Mth moment. In order to include the displayed pose information as part of the feature, Equivalently converted to a 6-dimensional vector VectorT M,i =[yaw, pitch, roll, transX, transY, transZ], then concat to the original C feat ×N-dimensional vector, that is, each dimension after the final transformation is C feat ×N+6 dimensions.

[0109] Concat all BEVMemory after STN and pose concat in the time dimension to obtain a 4-dimensional feature tensor TENSOR_FRAMES (C feat ×N+6,M,H feat_bev ,W feat_bev ), where the first dimension is the feature channel, the second dimension is the time series, and the last two dimensions are the spatial width and height dimensions.

[0110] 3. 3D convolution module for temporal fusion:

[0111] In order to aggregate the features in the temporal direction, the embodiment of the present application can introduce a 3D convolution module to perform temporal feature fusion on the TENSOR_FRAMES generated above. Specifically, the entire 3D convolution temporal fusion module is similar to a residual network. There are multiple 3D 1x1x1 convolutions in the network, but only one aggregated 4D feature vector is output in the end. The embodiment of the present application can extract the feature layer corresponding to the current moment as the final fused feature map TENSOR_SPACETIME_FUSION. The schematic diagram of the entire 3D module can be as follows: Figure 7 As shown in the figure, the right sub-figure represents the first 3D residual module with an input dimension of C, while the input dimensions of subsequent residual modules are all changed to C-6; among them, all convolution operations are 1x1x1 convolutions to reduce the computational complexity of 3D operations.

[0112] In step S104 , dynamic obstacle features and static obstacle features around the vehicle are acquired according to the second feature map, and a bird's-eye view of the vehicle's surroundings is generated based on the dynamic obstacle features and the static obstacle features.

[0113] It can be understood that after the above steps S101 to S103, the model of the embodiment of the present application can extract fusion information TENSOR_SPACETIME_FUSION of multiple perspectives and time sequences. The dimension of this fusion information is (C fusion ,H feat_bev ,W feat_bev ), among which C fusion =C feat ×N+6, therefore, the embodiment of the present application can acquire obstacle features from both dynamic and static aspects to generate a bird's-eye view around the vehicle.

[0114] In an embodiment of the present application, dynamic obstacle features and static obstacle features around the vehicle are obtained based on the second feature map, including: inputting the second feature map into a pre-established first decoder, decoding the dynamic obstacle semantic information of the second feature map, and obtaining the dynamic obstacle features; inputting the second feature map into a pre-established second decoder, decoding the static obstacle semantic information of the second feature map, and obtaining the static obstacle features.

[0115] 1. Dynamic obstacle detection branch of the first decoder:

[0116] For dynamic obstacle detection, this application can use an anchor-free detection method similar to CenterNet. Since target detection is done in BEV, in order to regress the rotation angle of the detection frame, this application improves CenterNet into 4 output branches, such as Figure 8 As shown, they are heatmap branch, offset branch, width and height branch, and rotation angle branch.

[0117] Specifically, if Figure 9 As shown in Figure 1, TENSOR_SPACETIME_FUSION inputs 4 branches, performs several convolution operations, and finally outputs 4 final TENSORs. Among them, TENSOR_heatmap is used to predict the center point of each dynamic obstacle, and its shape is (1, H feat_bev ,W feat_bev ); TENSOR_offset is used to predict the deviation between the actual obstacle point and the pixel coordinate point, including the u direction deviation and the v direction deviation, and the shape is (2,H feat_bev ,W feat_bev ); TENSOR_wh is used to predict the relative width w and height h of each dynamic obstacle, and its shape is (2,H feat_bev ,W feat_bev ); TENSOR_rotation is used to predict the yaw angle theta of each dynamic obstacle, and its shape is (1,H feat_bev ,W feat_bev The four TENSORs predicted by the model are post-processed similar to centernet to finally obtain the position, yaw angle, width and height of the center point of each dynamic obstacle from the BEV perspective, such as Figure 10 shown.

[0118] 2. Second decoder static road information detection branch:

[0119] This module can decode the TENSOR_SPACETIME_FUSION generated in the above dynamic branch. Different from conventional semantic segmentation, the embodiment of the present application can use key point detection and key point instance segmentation ideas to perceive static road information, and use point tangent direction classification to give the tangent direction of the key point line; thus, the embodiment of the present application can adopt a method such as Figure 11 The decoder shown performs real-time object detection for static curbs, lane markings, and crosswalks.

[0120] Specifically, if Figure 12As shown in the figure, static road surface detection has five branches, namely the key point heatmap branch, the key point offset branch, the key point cluster feature branch, the key point category branch, and the key point direction classification branch. TENSOR_SPACETIME_FUSION inputs each of the five branches, performs several convolution operations, and finally outputs five final TENSORs. Among them, TENSOR_heatmap is used to predict the key point position; TENSOR_offset is used to predict the decimal offset of the key point; TENSOR_cluster is used to regress the cluster feature of each key point; TENSOR_class is used to determine the type of key point (curb, crosswalk, lane line, etc.); TENSOR_direct is used to determine a tangent direction of the key point.

[0121] TENSOR_heatmap and TENSOR_offset, after post-processing similar to centernet, can obtain the precise position of each key point; TENSOR_cluster can instantiate each key point through the cluster algorithm, that is, divide all key points into multiple instances (PointSet1, PointSet2, PointSet3...); and for each PointSet, the category of each point can be obtained through the TENSOR_class branch.

[0122] Since a PointSet corresponds to an instance line, ideally, each point in the PointSet is predicted to have the same type (for example, the category prediction result of each point that makes up the roadside line should be roadside). However, in actual prediction, there may be inconsistencies in the categories of each point in the PointSet. In this case, the embodiment of the present application can count the categories of all points in the PointSet, and the category with the largest number of counts, maxClass, is selected as the category of this PointSet. Finally, the present application uses TENSOR_direct to sort and sparsely sample the points in the PointSet, and obtains the final structured static road surface line, that is, PointSet = {p start ,p1,p2,...,p N ,p end}, the corresponding line is the structured line Line = p start -p1-p2-...-p N -p end , the category is maxClass; among them, the prediction effect diagram can be as follows Figure 10 shown.

[0123] In an embodiment of the present application, a bird's-eye view of the vehicle's surroundings is generated based on dynamic obstacle features and static obstacle features, including: acquiring point cloud data from the vehicle's lidar; inputting the point cloud data into a pre-established point cloud feature extraction model, and outputting point cloud features of the point cloud data; and fusing the point cloud features, dynamic obstacle features, and static obstacle features to obtain a bird's-eye view of the vehicle's surroundings.

[0124] It is understandable that since the decoding process described above in this embodiment of the application obtains perception information by decoding TENSOR_SPACETIME_FUSION from the BEV's perspective, this embodiment of the application is a natural multimodal fusion framework. Therefore, this embodiment of the application can use a lidar as an optional input modality and then combine all the features obtained in the above steps to generate a bird's-eye view of the vehicle's surroundings. The following will explain the lidar input and overall feature fusion in conjunction with specific examples.

[0125] 1. LiDAR input branch:

[0126] It is understandable that LiDAR naturally contains 3D information, so it is easy to project it to the BEV perspective. Therefore, the embodiment of the present application can use LiDAR as an optional input modality.

[0127] Specifically, if Figure 13 As shown, the embodiment of the present application can first obtain the features of the point cloud under the BEV perspective through the point cloud feature extraction model and point cloud pillarpooling; secondly, the BEV features obtained from the point cloud are recorded as TENSOR_LIDAR, where its width and height dimensions can be made consistent with the visual BEV features by adjusting the convolution parameters, and its dimensions can be expressed as (C lidar ,H feat_bev ,W feat_bev ); Then, TENSOR_SPACETIME_FUSION and TENSOR_LIDAR are concat- ed on the feature dimension to obtain the final multimodal feature, whose dimension is (C lidar +C fusion ,H feat_bev ,W feat_bev ); In the subsequent decoding module, except for the corresponding change of the input channel dimension of the convolution operation, the rest can remain unchanged.

[0128] 2. Overall model architecture:

[0129] It is understandable that the embodiment of the present application can fuse the features of each obstacle obtained after all the above steps to generate the following Figure 14The multi-view multi-modal fusion architecture shown; wherein, since the embodiment of the present application is mainly based on visual information, the above-mentioned laser radar can be used as an optional supplementary modality.

[0130] Specifically, the entire model is trained using supervised training, with labels representing the true values of dynamic obstacles and static road surfaces from the BEV's perspective. The loss function is the sum of the loss functions of the two decoding branches. Table 1 below shows the sequence length and evaluation metrics. The evaluation metrics for the entire model at different sequence lengths in this embodiment of the application can be found in Table 1. The forward inference time of the entire model at different sequence lengths can also be found in Table 1. As can be seen, longer time series increase the amount of available information and lead to higher overall metrics.

[0131] Table 1

[0132]

[0133] The actual visualization of the entire framework of the embodiment of the present application can be as follows Figure 10 As shown, Figure 10 The result is only the visual fusion result, and does not involve the laser radar; the multi-view multi-modal fusion framework proposed in the embodiment of the present application needs to be coordinated with a specific software architecture, among which the alternative software architecture can be as follows Figure 15 shown.

[0134] In summary, the embodiments of the present application can provide a method for generating a bird's-eye view of a vehicle, while innovating the multi-view front fusion method and the perception information encoder, and in order to reduce the jitter of the perception results, the specific details are as follows:

[0135] 1. The embodiment of the present application can apply spatial homography transformation to the 2D feature map, convert it into a feature map on the bird's-eye view, and splice different feature maps by channel, and then send them to a Transformer encoder fusion module, so that features from different perspectives can be deeply fused.

[0136] 2. This embodiment of the application designs two decoders, one for decoding dynamic obstacles and the other for decoding static road semantic information. The dynamic obstacle decoder uses the CenterNet heatmap method to detect rotating boxes from the BEV perspective. The static obstacle decoder is based on point detection and point instance clustering, and outputs the tangent direction of each point. Finally, simple post-processing is used to vectorize the static road information.

[0137] 3. The embodiment of the present application introduces time series information, performs feature extraction on the previous N historical frames simultaneously, generates BEV features for each frame, and transforms all time series frames into the vehicle coordinate system of the current frame using spatial geometric transformation. Then, 3D convolution is used to fuse the time series frames into the final feature map for decoding.

[0138] According to the bird's-eye view generation method for a vehicle proposed in an embodiment of the present application, it is possible to receive original two-dimensional image inputs from multiple cameras surrounding the vehicle body 360 degrees and / or laser radar inputs, and directly output perception information from the bird's-eye view perspective, thereby reducing the workload of post-processing and improving the robustness of the entire system; the camera's internal and external parameters can be used to construct an inverse perspective transformation matrix to obtain the corresponding point of each point on the bird's-eye view feature map on the two-dimensional feature map, and the first feature map of the bird's-eye view can be obtained according to the inverse perspective transformation matrix, thereby applying the spatial homography transformation to the two-dimensional feature map, effectively utilizing the original image information, and making the feature map generation result more accurate and stable; time series information can be introduced so that all features at different times are in the same unified coordinate system (i.e., the vehicle body coordinate system at the current moment), thereby accurately and effectively obtaining the single-frame features after feature fusion through a simpler fusion method. The feature map allows feature information to overlap effectively without being misplaced, making the operation simpler and faster, and improving the fusion efficiency and accuracy of the fusion results. Dynamic decoding and static decoding can be used to detect rotating boxes from a bird's-eye view perspective, and based on point detection and point instance clustering, the tangent direction of each point is output. The static road information is vectorized through simple post-processing, which can effectively solve the problem of complex post-processing, reduce the workload of perception post-processing, and directly output all the necessary information required for planning and control from a bird's-eye view perspective. The decoding process is more complete and accurate, and the robustness of the entire perception system is improved. The lidar input can be used as a selection branch. In addition to realizing multi-perspective fusion, it can also fuse lidar and vision at the same time, process multi-modal information, improve the comprehensive information processing capability, and consider the features generated by the bird's-eye view more comprehensively, thereby improving the accuracy and comprehensiveness of the generated results.

[0139] Next, a bird's-eye view generating device for a vehicle proposed in an embodiment of the present application will be described with reference to the accompanying drawings.

[0140] Figure 16 It is a block diagram of a bird's-eye view generating device for a vehicle according to an embodiment of the present application.

[0141] like Figure 16 As shown, the vehicle bird's-eye view generating device 10 includes: an acquisition module 100 , a first transformation module 200 , a second transformation module 300 and a generation module 400 .

[0142] Among them, the acquisition module 100 is used to obtain multiple perception information around the vehicle; the first transformation module 200 is used to extract the two-dimensional feature map of each perception information, and perform geometric transformation on each two-dimensional feature map according to the preset transformation strategy to obtain the first feature map; the second transformation module 300 is used to perform feature fusion and spatial transformation on all first feature maps to obtain single-frame feature maps at different times, and fuse the single-frame feature maps at different times in the spatial dimension and time dimension in the vehicle coordinate system at the current time to generate the second feature map; the generation module 400 is used to obtain the dynamic obstacle features and static obstacle features around the vehicle according to the second feature map, and generate a bird's-eye view around the vehicle based on the dynamic obstacle features and static obstacle features.

[0143] In an embodiment of the present application, the first transformation module 200 is used to: obtain corresponding points on the first feature map and each two-dimensional feature map; apply the corresponding points to each two-dimensional feature map according to the pre-constructed inverse perspective transformation matrix to obtain the first feature map.

[0144] In an embodiment of the present application, the second transformation module 300 is used to: extract the feature relationship between each first feature map, perform feature fusion based on the feature relationship, and obtain a single-frame feature map at the current moment; convert the single-frame feature map at the current moment and the single-frame feature maps at multiple historical moments into the vehicle coordinate system at the current moment to obtain single-frame feature maps at different moments.

[0145] In an embodiment of the present application, the generation module 400 is used to: input the second feature map into a pre-established first decoder, decode the dynamic obstacle semantic information of the second feature map, and obtain dynamic obstacle features; input the second feature map into a pre-established second decoder, decode the static obstacle semantic information of the second feature map, and obtain static obstacle features.

[0146] In an embodiment of the present application, the generation module 400 is further used to: obtain point cloud data of the vehicle's lidar; input the point cloud data into a pre-established point cloud feature extraction model, and output the point cloud features of the point cloud data; and fuse the point cloud features, dynamic obstacle features, and static obstacle features to obtain a bird's-eye view of the area around the vehicle.

[0147] It should be noted that the above explanation of the embodiment of the method for generating a bird's-eye view of a vehicle is also applicable to the device for generating a bird's-eye view of a vehicle in this embodiment, and will not be repeated here.

[0148] According to the bird's-eye view generation device for a vehicle proposed in an embodiment of the present application, it can receive original two-dimensional image inputs from multiple cameras surrounding the vehicle body 360 degrees and / or laser radar inputs, and directly output perception information from the bird's-eye view perspective, thereby reducing the workload of post-processing and improving the robustness of the entire system; the camera's internal and external parameters can be used to construct an inverse perspective transformation matrix to obtain the corresponding point of each point on the bird's-eye view feature map on the two-dimensional feature map, and the first feature map of the bird's-eye view can be obtained according to the inverse perspective transformation matrix, thereby allowing the spatial homography transformation to be applied to the two-dimensional feature map, effectively utilizing the original image information, and making the feature map generation result more accurate and stable; time series information can be introduced so that all features at different times are in the same unified coordinate system (i.e., the vehicle body coordinate system at the current moment), thereby allowing the single-frame feature after feature fusion to be accurately and effectively obtained through a simpler fusion method. The feature map allows feature information to overlap effectively without being misplaced, making the operation simpler and faster, and improving the fusion efficiency and accuracy of the fusion results. Dynamic decoding and static decoding can be used to detect rotating boxes from a bird's-eye view perspective, and based on point detection and point instance clustering, the tangent direction of each point is output. The static road information is vectorized through simple post-processing, which can effectively solve the problem of complex post-processing, reduce the workload of perception post-processing, and directly output all the necessary information required for planning and control from a bird's-eye view perspective. The decoding process is more complete and accurate, and the robustness of the entire perception system is improved. The lidar input can be used as a selection branch. In addition to realizing multi-perspective fusion, it can also fuse lidar and vision at the same time, process multi-modal information, improve the comprehensive information processing capability, and consider the features generated by the bird's-eye view more comprehensively, thereby improving the accuracy and comprehensiveness of the generated results.

[0149] Figure 17 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application. The vehicle may include:

[0150] A memory 1701 , a processor 1702 , and a computer program stored in the memory 1701 and executable on the processor 1702 .

[0151] When the processor 1702 executes the program, the method for generating a bird's-eye view of a vehicle provided in the above embodiment is implemented.

[0152] Furthermore, the vehicle further comprises:

[0153] The communication interface 1703 is used for communication between the memory 1701 and the processor 1702 .

[0154] The memory 1701 is used to store computer programs that can be run on the processor 1702 .

[0155] The memory 1701 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0156] If the memory 1701, the processor 1702, and the communication interface 1703 are implemented independently, the communication interface 1703, the memory 1701, and the processor 1702 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 17 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0157] Optionally, in a specific implementation, if the memory 1701, the processor 1702 and the communication interface 1703 are integrated on a chip, the memory 1701, the processor 1702 and the communication interface 1703 can communicate with each other through an internal interface.

[0158] The processor 1702 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0159] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for generating a bird's-eye view of a vehicle.

[0160] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0161] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0162] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0163] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0164] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0165] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for generating a bird's-eye view of a vehicle, characterized in that: The following steps are involved: Acquire multiple perception information around the vehicle; Extracting a two-dimensional feature map of each perception information, and performing a geometric transformation on each two-dimensional feature map according to a preset transformation strategy to obtain a first feature map; Performing feature fusion and spatial transformation on all first feature maps to obtain single-frame feature maps at different times, and fusing the single-frame feature maps at different times in the current vehicle coordinate system based on the 3D convolution module in the spatial and temporal dimensions to generate a second feature map; Acquire dynamic obstacle features and static obstacle features around the vehicle according to the second feature map, and generate a bird's-eye view around the vehicle based on the dynamic obstacle features and the static obstacle features; The step of performing feature fusion and spatial transformation on all first feature maps to obtain single-frame feature maps at different moments includes: Extracting the feature relationship between each first feature map, performing feature fusion based on the feature relationship, and obtaining a single-frame feature map at the current moment; The single-frame feature map at the current moment and the single-frame feature maps at multiple historical moments are converted to the vehicle coordinate system at the current moment to obtain single-frame feature maps at different moments.

2. The method according to claim 1, characterized in that The step of geometrically transforming each two-dimensional feature map according to a preset transformation strategy to obtain a first feature map includes: Obtaining corresponding points on the first feature map and each two-dimensional feature map; The corresponding points are applied to each of the two-dimensional feature maps according to a pre-constructed inverse perspective transformation matrix to obtain the first feature map.

3. The method according to claim 1, characterized in that The acquiring, according to the second feature map, dynamic obstacle features and static obstacle features around the vehicle includes: Inputting the second feature map into a pre-established first decoder, decoding the dynamic obstacle semantic information on the second feature map, and obtaining the dynamic obstacle feature; The second feature map is input into a pre-established second decoder, and the static obstacle semantic information is decoded on the second feature map to obtain the static obstacle feature.

4. The method according to any one of claims 1 to 3, characterized in that Generating a bird's-eye view of the vehicle surroundings based on the dynamic obstacle features and the static obstacle features includes: Obtaining point cloud data of the vehicle's laser radar; Inputting the point cloud data into a pre-established point cloud feature extraction model, and outputting the point cloud features of the point cloud data; The point cloud features, the dynamic obstacle features, and the static obstacle features are fused to obtain a bird's-eye view of the area around the vehicle.

5. A device for generating a bird's-eye view of a vehicle, characterized in that: include: An acquisition module, used to acquire multiple perception information around the vehicle; A first transformation module is used to extract a two-dimensional feature map of each perception information, and perform a geometric transformation on each two-dimensional feature map according to a preset transformation strategy to obtain a first feature map; A second transformation module is configured to perform feature fusion and spatial transformation on all first feature maps to obtain single-frame feature maps at different times, and to fuse the single-frame feature maps at different times in the current vehicle coordinate system based on the 3D convolution module in the spatial and temporal dimensions to generate a second feature map; a generating module, configured to obtain dynamic obstacle features and static obstacle features around the vehicle according to the second feature map, and generate a bird's-eye view around the vehicle based on the dynamic obstacle features and the static obstacle features; The first transformation module is used to: obtain corresponding points on the first feature map and each two-dimensional feature map; apply the corresponding points to each two-dimensional feature map according to a pre-constructed inverse perspective transformation matrix to obtain the first feature map; The second transformation module is used to extract the feature relationship between each first feature map, perform feature fusion based on the feature relationship, and obtain a single-frame feature map at the current moment; The single-frame feature map at the current moment and the single-frame feature maps at multiple historical moments are converted to the vehicle coordinate system at the current moment to obtain single-frame feature maps at different moments.

6. The device according to claim 5, characterized in that The generating module is configured to: input the second feature map into a pre-established first decoder, decode the dynamic obstacle semantic information on the second feature map, and obtain the dynamic obstacle feature; The second feature map is input into a pre-established second decoder, and the static obstacle semantic information is decoded on the second feature map to obtain the static obstacle feature.

7. The device according to any one of claims 5-6, characterized in that The generating module is further configured to: Obtaining point cloud data of the vehicle's laser radar; Inputting the point cloud data into a pre-established point cloud feature extraction model, and outputting the point cloud features of the point cloud data; The point cloud features, the dynamic obstacle features, and the static obstacle features are fused to obtain a bird's-eye view of the area around the vehicle.

8. A vehicle, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for generating a bird's-eye view of a vehicle according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method for generating a bird's-eye view of a vehicle as described in any one of claims 1 to 4.

Images