A bird's-eye view perception method based on unified spatiotemporal fusion

Abstract

The present invention discloses a unified spatiotemporal fusion surround bird's-eye view perception method for visual perception in an autonomous driving system. Specifically, the method includes the following steps: obtaining an image data set for training a neural network and defining an algorithm target; establishing a virtual perspective model; extracting surround image features through a basic backbone network; establishing a temporal feature queue; unified spatiotemporal fusion modeling fusion features; and a head network outputting prediction results. Compared with other perception models in the prior art, the present invention can effectively fuse the spatial relationship of surround images at the same time, and can also fuse the temporal relationship of surround images at different times, and achieves better perception effects and faster perception speeds by better fusing different temporal steps.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a unified spatiotemporal fusion surround bird's-eye view perception method. Background Art

[0002] In recent years, the surround perception system based on bird's-eye view (BEV) for autonomous driving has gradually become the mainstream perception paradigm. The core meaning of the perception expression based on the bird's-eye view is to map the collected surround view images into the bird's-eye view BEV space based on the current position of the vehicle. This spatial fusion method can form a unified bird's-eye view BEV space and greatly reduce the difficulty of surround view multi-camera fusion. In addition, this bird's-eye view BEV space fusion naturally has a consistent 3D spatial position with other perception modalities, such as lidar, which simplifies the fusion of the visual system and the lidar system. Therefore, the surround perception system based on the bird's-eye view of autonomous driving has a unified expression capability that is convenient for other modal processing. However, in the face of surround view images in long time and long time sequence during autonomous driving, how to realize surround view bird's-eye view perception is one of the technical problems that need to be solved urgently. Summary of the invention

[0003] The purpose of the present invention is to solve the problem in the prior art that it is difficult to efficiently fuse surround view images over a long period of time and in a long time sequence during autonomous driving, resulting in poor perception effect, and to provide a surround view bird's-eye view perception method with unified time and space fusion.

[0004] To achieve the above purpose, the specific technical solutions adopted by the present invention are as follows:

[0005] A unified spatiotemporal fusion surround bird's-eye view perception method includes the following steps:

[0006] S1, establishing a virtual perspective model for transforming the camera space into the BEV space;

[0007] S2, constructing a basic backbone network for extracting multi-level features from a single surround view image in the surround view image group;

[0008] S3, based on the backbone network features constructed in S2, extracts frame-by-frame features of the input surround view images at different times, sorts the extracted features according to time, and establishes a time series feature queue;

[0009] S4, based on the virtual view model constructed in S1 and the temporal feature queue established in S3, the fusion feature is obtained by modeling a unified spatiotemporal fusion feature fuser;

[0010] S5. Based on the fusion features obtained in S4, the head network is used to output the prediction results to obtain the surrounding environment perception for autonomous driving.

[0011] Preferably, the surround bird's-eye view perception network framework composed of the basic backbone network, the unified spatiotemporal fusion feature fuser and the head network is pre-trained with an image dataset for surround bird's-eye view perception, and the surround image samples in the image dataset are composed of surround images collected by multi-directional cameras, camera internal and external parameter information corresponding to each camera, and annotation information corresponding to each image.

[0012] Preferably, the image data set for surround bird's-eye view perception includes an image set Among them I i is the i-th group of surround view images, each group of surround view images contains N images taken by cameras in different directions, and N is the total number of cameras on the vehicle facing different directions; Info i is the camera internal and external parameter information corresponding to the i-th group of surround images, including the rotation matrix and translation matrix; T i is the real 3D world annotation information corresponding to the entire group of the i-th group of surround view images, including map information centered on the group of surround view images, and / or annotation information of 3D vehicles in the surrounding world centered on the group of surround view images; M is the number of surround view image groups in the image dataset.

[0013] Preferably, in step S1, the method for establishing the virtual perspective model is as follows:

[0014] S11, for each group of surround images I i The corresponding camera internal and external parameter information Info i , the rotation matrix it contains is defined as The translation matrix is For the rotation matrix R, define R c is the rotation matrix at the current moment, R p is the rotation matrix at the past moment, R i,j is the rotation matrix used by the camera corresponding to the jth image in the i-th group of surround images, i∈{1,…,M}, j∈{1,…,N}; for the translation matrix t, define t c is the translation matrix at the current moment, t p is the translation matrix of the past time, t i,j is the translation matrix used by the camera corresponding to the jth image in the i-th group of surround images;

[0015] S12. Define the following virtual view model transformation:

[0016]

[0017]

[0018] in is the virtual rotation matrix of the camera corresponding to the jth image in the i-th group of surround images, is the virtual translation matrix of the camera corresponding to the jth image in the i-th group of surround images;

[0019] S13. For the virtual viewing angle model transformation obtained in S12, a virtual viewing angle model for transforming the camera space into the BEV space is established according to the following formula:

[0020]

[0021] Where P bev is the coordinate point in BEV space, P img is the coordinate point in the image space, K i,j is the camera intrinsic parameter of the camera corresponding to the jth image in the i-th group of surround images.

[0022] Preferably, in step S2, the basic backbone network constructed is as follows:

[0023] Select one of ResNet, Swin-Tiny, and VoVNet as the backbone network. Use the same selected backbone network to extract features for all input surround images, so that the backbone network can be shared between images taken by different cameras. For a backbone network with L levels, when extracting multi-level features, it is necessary to retain the features extracted from each layer (stage) of the backbone network, and finally obtain L levels of multi-level features.

[0024] Preferably, in step S3, the method for establishing a timing feature queue is as follows:

[0025] The surround image groups at different times are input into the backbone network, and the backbone network extracts single-frame multi-level features frame by frame, and stores the multi-level features in a temporal feature queue according to the time sequence of image acquisition.

[0026] Preferably, in step S4, the method for obtaining the fusion feature is as follows:

[0027] S41, based on the time series feature queue obtained in S3, obtaining the latest time series feature group with a length of P moments;

[0028] S42. Establish BEV spatial expression. expresses the entire BEV space; where Q x,yis the query (Query) of BEV, which represents the feature information at the BEV space position (x, y); C is the feature dimension, X and Y are the length and width of the BEV space; for each BEV space position (x, y), uniform sampling is performed from -3m to 5m at height z to obtain Z height points, thereby obtaining a set of 3D coordinates {(x, y, z)|z∈[-3, 5]} corresponding to the BEV space position (x, y); for each BEV space position (x, y), the sampled 3D coordinates {(x, y, z)|z∈[-3, 5]} are further transformed through the virtual perspective model established in S23, and the 3D coordinates {(x, y, z)|z∈[-3, 5]} in the BEV space are converted to the camera space to obtain its position P in the image space. img , thereby establishing a mapping relationship between position points in the BEV space and position points in the image space; according to the mapping relationship, the multi-level features in the image space in the temporal feature group are mapped to the BEV space;

[0029] S43, establish a unified spatiotemporal fusion feature fuser, which includes a self-attention layer, a first normalization layer, a mutual attention layer, a second normalization layer, a feedforward network layer and a third normalization layer cascaded from shallow to deep; wherein the self-attention layer uses a deformable attention method, the first normalization layer, the second normalization layer and the third normalization layer all use a layer normalization method (LayerNorm), the feedforward network consists of a fully connected network, and the mutual attention layer is defined by the following formula:

[0030]

[0031]

[0032] in The feature corresponding to the l-th level feature at the p-th moment in the temporal feature group at the BEV space position (x, y, z) after being mapped to the BEV space by the virtual view model; for The result after superimposing positional embedding (PE); is the mutual attention weight, and Q in its calculation formula x,y is Q in the original BEV space expression x,y Output after the self-attention layer and the first normalization layer;

[0033] S44, input the temporal feature group in S41 into the unified spatiotemporal fusion feature fuser established in S43, and obtain the output features of the unified spatiotemporal fusion feature fuser.

[0034] Preferably, in step S5, the process is as follows:

[0035] S51, using ERFNet as the head network, the output features obtained in S44 are passed through the ERFNet head network to obtain the autonomous driving surround map perception output;

[0036] S52. Use CenterPoint as the head network to pass the output features obtained in S44 through the CenterPoint head network to obtain the autonomous driving 3D target detection perception output.

[0037] S53: Integrate the perception output results obtained from S51 and S52 to finally obtain the autonomous driving surrounding environment perception.

[0038] The present invention provides a unified time-space fusion surround bird's-eye view perception method, which has the following beneficial effects compared with the existing surround bird's-eye view perception method:

[0039] Firstly, the spatiotemporal fusion scheme of the present invention can effectively fuse the spatial relationship of surround view images at the same time, and can also fuse the temporal relationship of surround view images at different times.

[0040] Secondly, the fusion method of the present invention can achieve fusion over a long period of time and in a long time sequence, and the fusion performance increases monotonically with the increase of the viewing angle.

[0041] Finally, the present invention can dynamically fuse different timing steps and has the characteristics of better effect and faster speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of the steps of a unified spatiotemporal fusion surround bird's-eye view perception method. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0044] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0045] In a preferred embodiment of the present invention, a unified spatiotemporal fusion surround view bird's-eye view perception method is provided. The method is used to perceive the real 3D world corresponding to the group of surround view images, such as map perception, 3D object perception, etc., based on surround view images collected by cameras in different directions on an autonomous driving vehicle. The method comprises the following steps:

[0046] S1. Establish a virtual perspective model for transforming the camera space into the bird's-eye view space (BE space).

[0047] In this embodiment, the specific method of establishing the virtual viewing angle model in the above step S1 is as follows:

[0048] S11, for each group of surround images I i The corresponding camera internal and external parameter information Info i , the rotation matrix it contains is defined as The translation matrix is For the rotation matrix R, define R c is the rotation matrix at the current moment, R p is the rotation matrix at the past moment, R i,j is the rotation matrix used by the camera corresponding to the jth image in the i-th group of surround images, i∈{1,…,M}, j∈{1,…,N}; for the translation matrix t, define t c is the translation matrix at the current moment, t p is the translation matrix of the past time, t i,j is the translation matrix used by the camera corresponding to the jth image in the i-th group of surround images.

[0049] In this embodiment, N=6, that is, each set of surround view images includes 6 pictures in different directions, which are respectively collected by cameras in 6 different directions.

[0050] S12. Define the following virtual view model transformation:

[0051]

[0052]

[0053] in is the virtual rotation matrix of the camera corresponding to the jth image in the i-th group of surround images, is the virtual translation matrix of the camera corresponding to the jth image in the i-th group of surround images.

[0054] S13. For the virtual perspective model transformation obtained in S12, a virtual perspective model for transforming the camera space into the bird's-eye view space is established according to the following formula:

[0055]

[0056] Where P bev is the coordinate point in the bird's-eye view space, P img is the coordinate point in the image space, K i,j is the camera intrinsic parameter of the camera corresponding to the jth image in the i-th group of surround images.

[0057] S2. Construct a basic backbone network for extracting multi-level features from a single surround view image in the surround view image group.

[0058] In this embodiment, in the above step S2, the basic backbone network constructed is as follows:

[0059] Select one of ResNet, Swin-Tiny, and VoVNet as the backbone network. Use the same selected backbone network to extract features for all input surround images, so that the backbone network can be shared between images taken by different cameras. For a backbone network with L levels, when extracting multi-level features, it is necessary to retain the features extracted from each layer (stage) of the backbone network, that is, retain all intermediate features and final features, and finally obtain L levels of multi-level features.

[0060] S3, based on the backbone network features constructed in S2, extracts frame-by-frame features of the input surround view images at different times, sorts the extracted features according to time, and establishes a time series feature queue.

[0061] In this embodiment, in the above step S3, the method for establishing the timing feature queue is as follows:

[0062] The surround image groups at different times are input into the backbone network, and the backbone network extracts single-frame multi-level features frame by frame, and stores the multi-level features in a temporal feature queue according to the time sequence of image acquisition.

[0063] S4. Based on the virtual perspective model constructed in S1 and the temporal feature queue established in S3, the fusion feature is obtained by modeling a unified spatiotemporal fusion feature fuser.

[0064] In this embodiment, in the above step S4, the method for obtaining the fusion feature is as follows:

[0065] S41. Based on the time series feature queue obtained in S3, obtain the latest time series feature group with a length of P moments.

[0066] It should be noted that P is a hyperparameter that needs to be optimized and adjusted. When obtaining fusion features, it is necessary to use the time series features extracted from the latest P surround image groups stored in the time series feature queue to form a time series feature group, so as to achieve real-time perception of the external world. Therefore, the time series feature queue is equivalent to a first-in, first-out sequence of length P, through which the latest time series feature group can be maintained.

[0067] S42. Create a bird's-eye view spatial expression. Express the entire bird's-eye view space; among them, Q x,y is the query (Query) of the bird's-eye view, which represents the feature information at the spatial position (x, y) of the bird's-eye view; C is the feature dimension, X and Y are the length and width of the bird's-eye view space; for each spatial position (x, y) of the bird's-eye view, uniform sampling is performed from -3m to 5m at height z to obtain Z height points, thereby obtaining a set of 3D coordinates {(x, y, z)|z∈[-3, 5]} corresponding to the spatial position (x, y) of the bird's-eye view; for each spatial position (x, y) of the bird's-eye view, the sampled 3D coordinates {(x, y, z)|z∈[-3, 5]} are further transformed through the virtual perspective model established in S23, and the 3D coordinates {(x, y, z)|z∈[-3, 5]} in the bird's-eye view space are converted to the camera space to obtain its position P in the image space. img , thereby establishing a mapping relationship between position points in the bird's-eye view space and position points in the image space; according to the mapping relationship, the multi-level features in the image space in the temporal feature group are mapped to the bird's-eye view space.

[0068] S43, establish a unified spatiotemporal fusion feature fuser, which includes a self-attention layer, a first normalization layer, a mutual attention layer, a second normalization layer, a feedforward network layer and a third normalization layer cascaded from shallow to deep; wherein the self-attention layer uses a deformable attention method, the first normalization layer, the second normalization layer and the third normalization layer all use a layer normalization method (LayerNorm), the feedforward network consists of a fully connected network, and the mutual attention layer is defined by the following formula:

[0069]

[0070]

[0071] in is the feature corresponding to the l-th level feature at the p-th moment in the temporal feature group at the bird's-eye view space position (x, y, z) after being mapped to the bird's-eye view space by the virtual perspective model, p = 1, 2, ..., P; l = 1, 2, ..., L; there are a total of Z z, which is specifically determined according to the Z height points obtained by uniform sampling; for The result after superimposing positional embedding (PE); is the mutual attention weight, and Q in its calculation formula x,y It is Q in the original bird's-eye view spatial expression x,y The output after the self-attention layer and the first normalization layer.

[0072] It should be noted that the self-attention layer, the first normalization layer, the mutual attention layer, the second normalization layer, the feedforward network layer and the third normalization layer are cascaded layer by layer. The output of the previous layer is used as the input of the next layer, and the output of the third normalization layer is used as the output of the entire aggregator. x,y It can be regarded as a parameter that is continuously updated iteratively. The input of the self-attention layer is Q in the original bird's-eye view spatial expression. x,y , the output is the updated Q x,y , the updated Q x,y After the layer normalization operation in the first normalization layer, the mutual attention layer is input, and so on.

[0073] It should be noted that the self-attention layer uses the deformable attention method, namely Deformable DETR, which belongs to the prior art. The specific principle can be found in the prior art literature: Zhu, Xizhou, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. "Deformable detr: Deformable transformers for end-to-end object detection." arXiv preprint arXiv:2010.04159(2020).

[0074] S44, input the temporal feature group in S41 into the unified spatiotemporal fusion feature fuser established in S43, and obtain the output features of the unified spatiotemporal fusion feature fuser.

[0075] S5. Based on the fusion features obtained in S4, the head network is used to output the prediction results to obtain the surrounding environment perception for autonomous driving.

[0076] In this embodiment, in the above step S5, the process is as follows:

[0077] S51, using ERFNet as the head network, the output features obtained in S44 are passed through the ERFNet head network to obtain the autonomous driving surround map perception output, that is, to obtain the map information centered on the group of surround images;

[0078] S52. Use CenterPoint as the head network to obtain the output features obtained in S44 through the CenterPoint head network to obtain the autonomous driving 3D target detection perception output, that is, to obtain the information of the surrounding world 3D targets (such as vehicles) centered on the group of surround view images.

[0079] S53: Integrate the perception output results obtained from S51 and S52 to finally obtain the autonomous driving surrounding environment perception.

[0080] It should be noted that in the above-mentioned step framework of S1 to S5, the basic backbone network, the unified spatiotemporal fusion feature fuser and the head network constitute the surround bird's-eye view perception network framework. However, before the surround bird's-eye view perception network framework is used for actual prediction perception, it needs to be trained in advance with an image dataset for surround bird's-eye view perception. The surround image samples in the image dataset are composed of surround images collected by multi-directional cameras, camera internal and external parameter information corresponding to each camera, and annotation information corresponding to each image.

[0081] In this embodiment, the image data set used for surround bird's-eye view perception includes an image set Among them I i is the i-th group of surround view images, each group of surround view images contains N images taken by cameras in different directions, and N is the total number of cameras on the vehicle facing different directions; Info i is the camera internal and external parameter information corresponding to the i-th group of surround images, including the rotation matrix and translation matrix; T i is the real 3D world annotation information corresponding to the entire group of surround view images of the i-th group, including two types of annotations. The first type is the map information centered on the group of surround view images, and the second type is the annotation information of the 3D vehicles in the surrounding world centered on the group of surround view images. M is the number of surround view image groups in the image dataset.

[0082] The specific method of using image datasets to train the surround bird's-eye view perception network framework belongs to the existing technology and will not be elaborated on here.

[0083] The unified spatiotemporal fusion surround bird's-eye view perception method described in S1 to S5 above is applied to a specific example to demonstrate its technical effect.

[0084] Example

[0085] The implementation method of this embodiment is as described above, and the specific steps will not be elaborated in detail. The following only shows its effect based on case data. The present invention is implemented on a dataset with true value annotations, and the detailed information is as follows:

[0086] NuScenes dataset [1]: This dataset contains 28,130 sets of surround view training images and 6,019 sets of surround view test images. Each set of images contains BEV spatial annotations and camera internal and external parameters.

[0087] This example mainly evaluates the BEV map segmentation task on the NuScenes dataset.

[0088] Table 1 Comparison of evaluation indicators of this embodiment on the NuScenes dataset (100m x 100m range)

[0089] Method Road mIoU Lane mIoU LSS[2] 72.9 20.0 VPN[3] 76.9 19.4 M2BEV[4] 77.2 - BEVFormer[5] 80.1 25.7 Method of the present invention 85.4 31.0

[0090] Table 1 Comparison of evaluation indicators of this embodiment on NuScenes dataset (60m x 30m range)

[0091] Method Divider Ped crossing Boundary All LSS[2] 38.3 14.9 39.3 30.8 VPN[3] 36.5 15.8 35.6 29.3 BEVSegFormer[6] 51.1 32.6 50.0 44.6 BEVerse[7] 56.1 44.9 58.7 53.2 Method of the present invention 60.6 49.0 62.5 57.4

[0092] The sources of the above NuScenes datasets and various comparison methods can be found in the following prior art documents:

[0093] [1]Caesar, Holger, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice ErinLiong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. "nuscenes: A multimodal dataset for autonomous driving." In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition,pp.11621-11631.2020.

[0094] [2]Philion, Jonah, and Sanja Fidler. "Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d." In European Conference on Computer Vision, pp. 194-210. Springer, Cham, 2020.

[0095] [3]Pan,Bowen,Jiankai Sun,Ho Yin Tiga Leung,Alex Andonian,and BoleiZhou."Cross-view semantic segmentation for sensing surroundings."IEEERobotics and Automation Letters 5,no.3(2020):4867-4873.

[0096] [4]Xie,Enze,Zhiding Yu,Daquan Zhou,Jonah Philion,Anima Anandkumar,Sanja Fidler,Ping Luo,and Jose M.Alvarez."M^2BEV:Multi-Camera Joint 3DDetection and Segmentation with Unified Birds-Eye View Representation."arXivpreprint arXiv:2204.05088(2022).

[0097] [5]Li,Zhiqi,Wenhai Wang,Hongyang Li,Enze Xie,Chonghao Sima,Tong Lu,Qiao Yu,and Jifeng Dai."BEVFormer:Learning Bird's-Eye-View Representationfrom Multi-Camera Images via Spatiotemporal Transformers."arXiv preprintarXiv:2203.17270(2022).

[0098] [6]Peng,Lang,Zhirong Chen,Zhangjie Fu,Pengpeng Liang,and ErkangCheng."BEVSegFormer:Bird's Eye View Semantic Segmentation From ArbitraryCamera Rigs."arXiv preprint arXiv:2203.04050(2022).

[0099] [7] Zhang, Yunpeng, Zheng Zhu, Wenzhao Zheng, Junjie Huang, Guan Huang, Jie Zhou, and Jiwen Lu. "BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving." arXiv preprint arXiv:2205.09743(2022).

[0100] The above results show that compared with other perception models in the prior art, the present invention can effectively fuse the spatial relationship of surround images at the same time, and can also fuse the temporal relationship of surround images at different times, and achieve better perception effect and faster perception speed by better fusing different temporal steps.

[0101] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A unified spatiotemporal fusion surround bird's-eye view perception method, characterized in that: The following steps are involved: S1, establishing a virtual perspective model for transforming the camera space into the bird's-eye view space; S2, constructing a basic backbone network for extracting multi-level features from a single surround view image in the surround view image group; S3, based on the backbone network features constructed in S2, extracts frame-by-frame features of the input surround view images at different times, sorts the extracted features according to time, and establishes a time series feature queue; S4, based on the virtual view model constructed in S1 and the temporal feature queue established in S3, the fusion feature is obtained by modeling a unified spatiotemporal fusion feature fuser; S5, based on the fusion features obtained in S4, the head network is used to output the prediction results to obtain the surrounding environment perception for autonomous driving; In step S4, the method for obtaining the fusion feature is as follows: S41, based on the time series feature queue obtained in S3, obtaining the latest time series feature group with a length of P moments; S42. Create a bird's-eye view spatial expression. Express the entire bird's-eye view space; among them, Q x，y is the query (Query) of the bird's-eye view, which represents the feature information at the bird's-eye view spatial position (x, y); C is the feature dimension, X and Y are the length and width of the bird's-eye view space; for each bird's-eye view spatial position (x, y), uniform sampling is performed from -3m to 5m at height z to obtain Z height points, thereby obtaining a set of 3D coordinates {(x, y, z)|z∈[-3,5]} corresponding to the bird's-eye view spatial position (x, y); for each bird's-eye view spatial position (x, y), the sampled 3D coordinates {(x, y, z)|z∈[-3,5]} are further transformed through the virtual perspective model established in S23, and the 3D coordinates {(x, y, z)|z∈[-3,5]} in the bird's-eye view space are converted to the camera space to obtain its position P in the image space. img , thereby establishing a mapping relationship between position points in the bird's-eye view space and position points in the image space; according to the mapping relationship, the multi-level features in the image space in the temporal feature group are mapped to the bird's-eye view space; S43, establish a unified spatiotemporal fusion feature fuser, which includes a self-attention layer, a first normalization layer, a mutual attention layer, a second normalization layer, a feedforward network layer and a third normalization layer cascaded from shallow to deep; wherein the self-attention layer uses a deformable attention method, the first normalization layer, the second normalization layer and the third normalization layer all use a layer normalization method (LayerNorm), the feedforward network consists of a fully connected network, and the mutual attention layer is defined by the following formula: in is the feature corresponding to the position (x, y, z) in the bird's-eye view space after the feature of the th level at the p-th moment in the temporal feature group is mapped to the bird's-eye view space by the virtual perspective model; for The result after superimposing positional embedding (PE); is the mutual attention weight, and Q in its calculation formula x，y It is Q in the original bird's-eye view spatial expression x，y Output after the self-attention layer and the first normalization layer; S44, input the temporal feature group in S41 into the unified spatiotemporal fusion feature fuser established in S43, and obtain the output features of the unified spatiotemporal fusion feature fuser.

2. The unified spatiotemporal fusion surround bird's-eye view perception method according to claim 1, characterized in that: The surround view bird's eye view perception network framework composed of the basic backbone network, the unified spatiotemporal fusion feature fuser and the head network is pre-trained with an image dataset for surround view bird's eye view perception. The surround view image samples in the image dataset are composed of surround view images collected by multi-directional cameras, camera internal and external parameter information corresponding to each camera, and annotation information corresponding to each image.

3. The unified spatiotemporal fusion surround bird's-eye view perception method according to claim 2, characterized in that: The image dataset for surround bird's-eye view perception includes an image set Among them I i is the i-th group of surround view images, each group of surround view images contains N images taken by cameras in different directions, and N is the total number of cameras on the vehicle facing different directions; Info i is the camera internal and external parameter information corresponding to the i-th group of surround images, including the rotation matrix and translation matrix; T i The real 3D world annotation information corresponding to the entire group of the i-th group of surround view images includes map information centered on the group of surround view images and / or annotation information of 3D vehicles in the surrounding world centered on the group of surround view images; M is the number of surround image groups in the image dataset.

4. The unified spatiotemporal fusion surround bird's-eye view perception method according to claim 1, characterized in that: In step S1, the method of establishing a virtual perspective model is as follows: S11, for each group of surround images I i The corresponding camera internal and external parameter information Info i , the rotation matrix it contains is defined as The translation matrix is For the rotation matrix R, define R c is the rotation matrix at the current moment, R p is the rotation matrix at the past moment, R i，j is the rotation matrix used by the camera corresponding to the jth image in the i-th group of surround images, i∈{1,...,M}, j∈{1,...,N}; for the translation matrix t, define t c is the translation matrix at the current moment, t p is the translation matrix of the past time, t i，j is the translation matrix used by the camera corresponding to the jth image in the i-th group of surround images; S12. Define the following virtual view model transformation: in is the virtual rotation matrix of the camera corresponding to the jth image in the i-th group of surround images, is the virtual translation matrix of the camera corresponding to the jth image in the i-th group of surround images; S13. For the virtual perspective model transformation obtained in S12, a virtual perspective model for transforming the camera space into the bird's-eye view space is established according to the following formula: Where P bev is the coordinate point in the bird's-eye view space, P img is the coordinate point in the image space, K i，j is the camera intrinsic parameter of the camera corresponding to the jth image in the i-th group of surround images.

5. The unified spatiotemporal fusion surround bird's-eye view perception method according to claim 4, characterized in that: In step S2, the basic backbone network constructed is as follows: Select one of ResNet, Swin-Tiny, and VoVNet as the backbone network. Use the same selected backbone network to extract features from all input surround view images, so that the backbone network can be shared between images taken by different cameras. For a backbone network with L levels, when extracting multi-level features, the features extracted from each stage of the backbone network need to be retained, and finally L levels of multi-level features are obtained.

6. The unified spatiotemporal fusion surround bird's-eye view perception method according to claim 5, characterized in that: In step S3, the method for establishing a timing feature queue is as follows: The surround image groups at different times are input into the backbone network, and the backbone network extracts single-frame multi-level features frame by frame, and saves the multi-level features in a temporal feature queue according to the time sequence of image acquisition.

7. The unified spatiotemporal fusion surround bird's-eye view perception method according to claim 6, characterized in that: In step S5, the process is as follows: S51, using ERFNet as the head network, the output features obtained in S44 are passed through the ERFNet head network to obtain the autonomous driving surround map perception output; S52, using CenterPoint as the head network, the output features obtained in S44 are passed through the CenterPoint head network to obtain the autonomous driving 3D target detection perception output; S53: Integrate the perception output results obtained from S51 and S52 to finally obtain the surrounding environment perception of autonomous driving.

Images