Bird's eye view feature generation method based on vehicle-mounted surround view images

Abstract

The present disclosure provides a bird's eye view feature generation method based on a vehicle-mounted surround view image, comprising: extracting all vehicle-mounted camera collected images of the ego vehicle at the current time; obtaining image features of each collected image at the current time; obtaining the bird's eye view feature after initialization at the current time based on the bird's eye view feature at the historical time; performing spatial feature sampling in the image features of each collected image based on the bird's eye view feature after initialization at the current time to obtain the sampled features; performing spatial cross-attention between the bird's eye view feature after initialization at the current time and the sampled features to generate the bird's eye view feature at the current time with the ego vehicle as the center; and establishing a connection between different channels of the bird's eye view feature at the current time based on a separable convolutional neural network to obtain an enhanced bird's eye view feature at the current time. The present disclosure also provides an electronic device, a readable storage medium and a program product.

Description

Technical Field

[0001] This disclosure relates to the fields of autonomous driving technology and computer vision technology, and in particular to a method, apparatus, electronic device, storage medium and program product for generating bird's-eye view features based on vehicle surround view images. Background Technology

[0002] Bird's-eye view (BEV) is a top-down view representation that helps downstream tasks such as 3D object detection and lane segmentation build on a unified feature representation, enhancing model reusability. It also facilitates the transition from post-sensor fusion to pre-sensor fusion, making the fusion process entirely data-driven. Furthermore, it effectively represents the spatial relationships between objects, aiding in prediction and planning. It is also highly compatible with previous 2D object detection methods, facilitating deployment.

[0003] Using bird's-eye view features as a unified feature representation for downstream tasks is an important emerging technology in the perception layer of autonomous driving, playing a crucial role in spatial perception. Previous bird's-eye view feature generation techniques were mostly based on LiDAR (Light Detection and Ranging) technology. While coordinate transformation facilitates feature generation, the high cost of LiDAR makes it difficult to equip autonomous vehicles with large numbers of high-precision LiDAR systems. In contrast, methods based on surround-view images are inexpensive, and their perception principle is the same as that of human drivers—using pure vision—making them suitable for real-world autonomous driving scenarios and possessing significant research value. The quality of generated bird's-eye view features directly affects the perception accuracy of downstream tasks, thus impacting subsequent prediction and planning. Image-based bird's-eye view feature generation methods also face challenges such as difficulty in effectively extracting and utilizing temporal features, difficulty in reasonably sampling high-quality spatial features, and difficulty in efficiently using spatial features to generate high-quality bird's-eye view features.

[0004] The following are some existing technical solutions.

[0005] Technical Solution 1, "DETR: End-to-End Object Detection With Transformers," first introduces the Transformer model into object detection. It departs from the traditional paradigm of first proposing predicted bounding boxes and then determining whether and what content those boxes contain, instead proposing a query paradigm. It uses a learnable vector as the ObjectQuery to query whether an object exists at different locations in the image and what kind of object it is. While this solution proposes a query mechanism for identifying the target, convergence is very slow due to bipartite graph matching, and the use of an Encoder for global attention significantly increases computational overhead.

[0006] Technical Solution 2, "DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries," is the first to introduce the DETR series into 3D object detection. It uses a CNN backbone to extract high-dimensional feature representations of images, then uses an Object Query as the query vector. A Transformer Decoder iteratively predicts the 3D center point of the object and regresses the object bounding box. The predicted center point is then geometrically projected onto the high-dimensional feature representation extracted by the CNN backbone to extract corresponding features for the next iteration. This technical solution provides a highly effective way to generate 3D feature representations from 2D feature representations—that is, a query mechanism that predicts the center point of a 3D object and projects it into a 2D feature space using an Object Query. However, this method is only suitable for 3D object detection and struggles to support dense feature representation predictions such as lane line detection, thus failing to unify downstream perception tasks in autonomous driving.

[0007] Technical Solution 3, "BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers," further improves upon DETR3D. First, it replaces the Object Query with a BEV Query. This involves initializing a fixed-size bird's-eye-view feature map (BEV Map) and projecting each point on the BEV Map as a BEV Query onto a 2D feature map extracted by the CNN backbone to extract features, thus providing a dense feature representation through the introduction of the bird's-eye-view. Second, it introduces a temporal attention module. Through the vehicle's speed and pose changes, it uses cross-attention mixing of the previous and current BEV Maps to incorporate temporal information. However, the layered and mixed use of temporal and spatial information makes it difficult for the attention module to clearly and effectively distinguish and learn temporal and spatial biases, hindering the construction of high-quality bird's-eye-view features.

[0008] Technical Solution 4, "UniFormer: Unified Multi-view Fusion Transformer for Spatial-Temporal Representation in Bird's-Eye-View," uses a similar model architecture to Technical Solution 3, but differs in its utilization of temporal information. Instead of using cross-attention between the previous and current BEV maps, it only uses the current BEV map for self-attention calculation of the BEV query, while retaining the convolutional feature maps extracted from the previous time step. During BEV query projection, features are extracted from convolutional feature maps across multiple time steps. This method of directly extracting image features gathers more raw temporal information, facilitating model analysis, but it also significantly increases memory overhead, making the model difficult to deploy in practical applications.

[0009] Technical Solution 5: "Translating Images into Maps" utilizes the physical relationship between image viewpoint and bird's-eye viewpoint. The pixel features of each vertical line in the image viewpoint correspond to the geometrically identical polar ray features centered on the vehicle in the bird's-eye view. First, an attention mechanism maps the vertical line pixel features of the image viewpoint to a memory cell feature of the same size. Then, the same attention mechanism maps the memory cell features to the polar ray features of the bird's-eye view. This effectively utilizes real-world physical relationships to model the connection between image viewpoint and bird's-eye viewpoint. However, this correspondence is a mapping from image features to bird's-eye view features, not a sampling from bird's-eye view features to image features, and it can only be applied to the generation of bird's-eye view features from a single image.

[0010] Technical Solution 1 constructs a target detection paradigm based on the Query mechanism, laying the foundation for the solution, but it only performs 2D target detection. Technical Solution 2 introduces the Query mechanism into 3D target detection, building a bridge between 2D and 3D feature representations, but it only performs 3D target detection. Technical Solution 3 uses the Query mechanism to generate a BEV Map, but the utilization of temporal information is still relatively rudimentary, and no enhancement modules are designed for the generated BEV Map. Technical Solution 4 directly uses historical image features to extract temporal information, but this significantly increases the use of video memory. Technical Solution 5 uses the geometric relationship between the image viewpoint and the bird's-eye view viewpoint for joint feature generation, improving the robustness of bird's-eye view feature construction. However, this is the generation from image features to bird's-eye view features, not the sampling from bird's-eye view features to image features, and it can only be applied to the generation of bird's-eye view features for a single image. Summary of the Invention

[0011] To address at least one of the aforementioned technical problems, this disclosure provides a method, apparatus, electronic device, storage medium, and program product for generating bird's-eye view features based on vehicle surround view images.

[0012] According to one aspect of this disclosure, a method for generating bird's-eye view features based on vehicle surround view images is provided, comprising:

[0013] Extract images captured by all onboard cameras of this vehicle at the current moment;

[0014] Obtain the image features of each captured image at the current moment;

[0015] Obtain the initialized bird's-eye view features at the current moment based on the bird's-eye view features at historical moments;

[0016] Based on the bird's-eye view features after initialization at the current moment, spatial feature sampling is performed on the image features of each acquired image to obtain the sampled features;

[0017] Spatial cross-attention is performed between the initialized bird's-eye view features at the current moment and the sampled features to generate bird's-eye view features centered on the vehicle at the current moment.

[0018] Based on a separable convolutional neural network, connections are established between different channels of the bird's-eye view features at the current moment to obtain enhanced bird's-eye view features at the current moment.

[0019] According to at least one embodiment of the bird's-eye view feature generation method based on vehicle surround view images of this disclosure, image features of each acquired image at the current time are obtained, including:

[0020] Feature extraction is performed on each captured image at the current moment using a convolutional neural network to obtain the image features of each captured image from the vehicle-mounted camera at the current moment.

[0021] According to at least one embodiment of the method for generating bird's-eye view features based on vehicle surround view images, the convolutional neural network includes a standard ResNet-101 backbone network, an FPN network, and an additional convolutional layer.

[0022] The ResNet-101 backbone network is a four-stage ResNet-101 backbone network pre-trained using ImageNet.

[0023] A method for generating bird's-eye view features based on vehicle surround view images according to at least one embodiment of the present disclosure, which obtains the initialized bird's-eye view features at the current moment based on the bird's-eye view features at historical moments, includes:

[0024] Randomly initialize a learnable bird's-eye view feature centered on the vehicle as the bird's-eye view feature for the current moment;

[0025] Add positional encoding to the learnable bird's-eye view features to obtain a position-encoded bird's-eye view of the current frame;

[0026] Extract a bird's-eye view (BEV Map) of historical frames within a preset time period prior to the current moment;

[0027] Obtain the relative pose of the bird's-eye view after the current frame's position encoding and the bird's-eye view of the historical frames;

[0028] For each location point in the bird's-eye view query vector of the current frame's location-encoded bird's-eye view, based on the relative pose, the corresponding historical frame bird's-eye view query vector is found in the historical frame bird's-eye view as temporal prior information.

[0029] By appending the temporal prior information to the position-encoded bird's-eye view of the current frame, the initialized bird's-eye view features at the current moment are obtained.

[0030] According to at least one embodiment of the bird's-eye view feature generation method based on vehicle surround view images of this disclosure, the method extracts historical frame bird's-eye view images (BEV Map) within a preset time period prior to the current time, including:

[0031] Extract one or more historical frames from the bird's-eye view.

[0032] According to at least one embodiment of the method for generating bird's-eye view features based on vehicle surround view images, if two or more historical frame bird's-eye view images are extracted, the temporal prior information obtained from each historical frame bird's-eye view image is weighted and summed to obtain the final temporal prior information.

[0033] According to at least one embodiment of the bird's-eye view feature generation method based on vehicle surround view images, spatial feature sampling is performed on the image features of each acquired image based on the initialized bird's-eye view features at the current moment to obtain the sampled features, including:

[0034] The spatial features are sampled based on the geometric correspondence between the bird's-eye view and the image view.

[0035] According to at least one embodiment of the present disclosure, a method for generating bird's-eye view features based on vehicle surround view images, spatial feature sampling is performed based on the geometric correspondence between the bird's-eye view perspective and the image perspective, including:

[0036] Each location point in the current initial bird's-eye view is used as the bird's-eye view query vector (BEV Query) to perform spatial feature sampling of the feature maps of the images acquired by each vehicle-mounted camera based on geometric projection.

[0037] According to at least one embodiment of the present disclosure, a method for generating bird's-eye view features based on vehicle surround view images, spatial feature sampling is performed based on the geometric correspondence between the bird's-eye view perspective and the image perspective, including:

[0038] Using each vertical line of the feature map representing the image features of each acquired image as a reference line, the polar ray features centered on the vehicle in the bird's-eye view are obtained, and then the bird's-eye view query vector (BEV Query) is obtained for spatial feature sampling based on the reference line.

[0039] According to at least one embodiment of the present disclosure, a method for generating bird's-eye view features based on vehicle surround view images involves performing spatial cross-attention on the initialized bird's-eye view features at the current moment and the sampled features to generate bird's-eye view features centered on the vehicle at the current moment, including:

[0040] For each BEV query vector in the BEV Map, the BEV viewpoint is projected to the image viewpoint and the sampled features are then subjected to cross-attention of the sampling points.

[0041] Reference lines are sampled for each polar ray centered on the vehicle on the bird's-eye view (BEV Map), and reference line intersection attention is performed;

[0042] Output the current moment's bird's-eye view features centered on this vehicle.

[0043] According to at least one embodiment of the method for generating bird's-eye view features based on vehicle surround view images, the features sampled by projecting from the BEV viewpoint to the image viewpoint based on each bird's-eye view query vector (BEV Query) in the bird's-eye view (BEV Map) undergo sampling point cross-attention, including:

[0044] Obtain m projection points for each bird's-eye view query vector, with each projection point serving as a feature sampling point;

[0045] Based on the features of each projection point, k position offsets are predicted. Each position offset is added to the coordinates corresponding to the projection point to obtain an additional k feature sampling points. Thus, (k+1) feature sampling points are obtained for each projection point, and m×(k+1) feature sampling points are obtained for each bird's-eye view query vector.

[0046] The bird's-eye view query vector (BEV Query) is used as the query vector for cross-attention. The sampling points and sampling point features are used as the key and value of cross-attention, respectively. Cross-attention is calculated to update the bird's-eye view query vector (BEV Query).

[0047] According to at least one embodiment of the bird's-eye view feature generation method based on vehicle surround view image of this disclosure, reference line sampling is performed on each polar ray centered on the vehicle on the bird's-eye view (BEV Map), and reference line cross attention is performed, including:

[0048] Each polar ray and its adjacent bird's-eye view query vector (BEV Query) are used as the query vector for cross-attention, and cross-attention is calculated using the vertical lines from left to right in the image features and the vertical line features as the key and value to update the bird's-eye view query vector (BEV Query).

[0049] A method for generating bird's-eye view features based on vehicle surround-view images according to at least one embodiment of the present disclosure establishes connections between different channels of the bird's-eye view features at the current moment based on a separable convolutional neural network to obtain enhanced bird's-eye view features at the current moment, including:

[0050] Keep the channel dimension of the bird's-eye view features unchanged, so that the number of convolutional kernels in the separable convolutional neural network is the same as the channel dimension of the bird's-eye view features.

[0051] According to another aspect of this disclosure, a bird's-eye view feature generation apparatus based on vehicle surround view images is provided, comprising:

[0052] The image acquisition module extracts images captured by all the vehicle's onboard cameras at the current moment;

[0053] An image feature extraction module acquires the image features of each acquired image at the current moment;

[0054] A temporal prior module, which obtains the initialized bird's-eye view features at the current moment based on the bird's-eye view features at historical moments;

[0055] The sampling module performs spatial feature sampling on the image features of each acquired image based on the initialized bird's-eye view features at the current moment, so as to obtain the sampled features;

[0056] A spatial cross-attention module performs spatial cross-attention between the initialized bird's-eye view features at the current moment and the sampled features to generate bird's-eye view features centered on the vehicle at the current moment.

[0057] A separable convolutional module establishes connections between different channels of the bird's-eye view features at the current moment to obtain enhanced bird's-eye view features at the current moment.

[0058] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0059] The memory stores execution instructions;

[0060] A processor that executes execution instructions stored in the memory, causing the processor to execute a bird's-eye view feature generation method based on vehicle surround view images according to any embodiment of the present disclosure.

[0061] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement the bird's-eye view feature generation method based on vehicle surround view images according to any embodiment of this disclosure.

[0062] According to another aspect of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the bird's-eye view feature generation method based on vehicle surround view images according to any embodiment of this disclosure. Attached Figure Description

[0063] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0064] Figure 1 This is a flowchart illustrating a method for generating bird's-eye view features based on vehicle surround view images, according to one embodiment of this disclosure.

[0065] Figure 2 This is a schematic diagram of the overall model architecture for implementing the bird's-eye view feature generation method based on vehicle surround view images of this disclosure, which is one embodiment of the present disclosure.

[0066] Figure 3 This is an architecture diagram of an image feature extraction module according to one embodiment of the present disclosure.

[0067] Figure 4 This is a schematic flowchart illustrating one embodiment of the present disclosure of obtaining the initialized bird's-eye view features at the current moment.

[0068] Figure 5 This is a schematic diagram illustrating the correspondence between the current frame BEV Query and the same real-world location in historical frames, according to one embodiment of this disclosure.

[0069] Figure 6 This is a projection diagram of the BEV viewpoint and image viewpoint position according to one embodiment of the present disclosure.

[0070] Figure 7 This is a schematic diagram of a reference line sampling method according to one embodiment of the present disclosure.

[0071] Figure 8 This is a schematic diagram of the spatial attention module according to one embodiment of the present disclosure.

[0072] Figure 9 This is a schematic block diagram of a bird's-eye view feature generation device based on vehicle surround view image, which employs a hardware implementation of a processing system according to one embodiment of the present disclosure.

[0073] Explanation of reference numerals in the attached figures

[0074] 1000 Bird's-eye View Feature Generation Device

[0075] 1002 Image Acquisition Module

[0076] 1004 Image Feature Extraction Module

[0077] 1006 Timing Prior Module

[0078] 1008 Sampling Module

[0079] 1010 Spatial Cross-Attention Module

[0080] 1012 Separable Convolutional Modules

[0081] 1100 bus

[0082] 1200 processor

[0083] 1300 memory

[0084] 1400 Other circuits. Detailed Implementation

[0085] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0086] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0087] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0088] The use of crosshairs and / or shading in the accompanying drawings is generally used to clarify the boundaries between adjacent components. Thus, unless otherwise stated, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for the specific material, material properties, dimensions, proportions, commonalities between the illustrated components, or any other characteristics, properties, etc., of the components. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of components may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously or in the reverse order of their description. Furthermore, the same reference numerals denote the same components.

[0089] When a component is referred to as being "on" or "above" another component, "connected to," or "joined to" another component, the component may be directly on, directly connected to, or directly joined to the other component, or there may be intermediate components. However, when a component is referred to as being "directly on" another component, "directly connected to," or "directly joined to" another component, there are no intermediate components. Therefore, the term "connection" can refer to a physical connection, an electrical connection, etc., and may or may not have intermediate components.

[0090] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0091] The following text combines Figures 1 to 9 The present invention provides a detailed description of the method and apparatus for generating bird's-eye view features based on vehicle surround view images.

[0092] Figure 1 This is a flowchart illustrating a method for generating bird's-eye view features based on vehicle surround view images, according to one embodiment of this disclosure.

[0093] refer to Figure 1 The bird's-eye view feature generation method S100 based on vehicle surround view image disclosed herein includes:

[0094] S102. Extract the images captured by all the vehicle's onboard cameras at the current moment (the images captured by all the onboard cameras constitute the vehicle surround view image).

[0095] S104. Obtain the image features of each acquired image at the current moment (the image features are represented by 3 feature maps (exemplarily));

[0096] S106. Obtain the initialized bird's-eye view features at the current moment based on the bird's-eye view features at historical moments (temporal prior module);

[0097] S108. Based on the bird's-eye view features after initialization at the current moment, spatial feature sampling (projection sampling from BEV viewpoint to image viewpoint + reference line sampling) is performed on the image features (high-dimensional image features) of each acquired image to obtain the sampled features (BEV Map (bird's-eye view) extracts the corresponding features from the image space).

[0098] S110. Perform spatial cross-attention on the initialized bird's-eye view features at the current moment and the sampled features to generate bird's-eye view features centered on the vehicle at the current moment.

[0099] S112. Based on the separable convolutional neural network, establish connections between different channels of the bird's-eye view features at the current time to obtain enhanced bird's-eye view features at the current time.

[0100] The bird's-eye view feature generation method disclosed herein is a high-quality method for generating bird's-eye view features. It generates bird's-eye view features based on a top-down perspective in real time from the vehicle's surrounding surround view images, which facilitates subsequent downstream tasks to be carried out on the bird's-eye view features.

[0101] To address the technical challenge of effectively extracting and utilizing temporal features, this disclosure proposes a temporal prior module. This module achieves effective temporal information fusion by fusing features of the same real-world location on a bird's-eye view. Temporal information is used only to initialize the bird's-eye view features at the current moment and does not participate in the iterative sampling and generation of these features. This separate spatiotemporal feature fusion method enhances the learning ability of both temporal and spatial feature information, whereas existing techniques rely on overlapping temporal and spatial cross-attention.

[0102] To address the technical problem of difficulty in reasonably sampling high-quality spatial features, this disclosure proposes a spatial feature sampling method based on reference line sampling, which transforms bird's-eye view features into image features. It combines a BEV (Browser-Earth View) to image view projection sampling method with a reference line sampling method to achieve high-quality spatial feature sampling. Most existing feature sampling methods are based on point-to-point projection sampling, which loses the corresponding geometric relationship between the image view and the bird's-eye view. Therefore, using a reference line sampling method in conjunction with this method will help to sample high-quality spatial features.

[0103] To address the technical challenge of efficiently using spatial features to generate high-quality bird's-eye view features, this disclosure proposes a spatial attention module. This module efficiently fuses sampled spatial features using cross-attention and then generates a high-quality bird's-eye view feature using separable convolution. By employing cross-attention and automatically learning the fusion of different spatial features based on data, and then passing a separable convolutional neural network layer after the spatial cross-attention, connections are established between different channels of the bird's-eye view feature, which helps generate high-quality bird's-eye view features.

[0104] Figure 2 This is a schematic diagram of the overall model architecture (i.e., bird's-eye view feature device) for implementing the bird's-eye view feature generation method based on vehicle surround view images disclosed herein, with reference to... Figure 2 For generating bird's-eye view features at a given moment, the system first extracts images captured by all vehicle-mounted surround-view cameras at that moment. These images are then processed through a convolutional neural network to extract high-dimensional features for spatial feature sampling. Next, a learnable bird's-eye view feature (BEV Map) centered on the vehicle is randomly initialized and location encoding is added, before being fed into the temporal prior module. In the temporal prior module, the initialized BEV Map is fused with historically generated BEV Maps based on their real-world locations, and the historically generated BEV Maps are used to re-initialize the current BEV Map. After temporal initialization, the BEV Map undergoes spatial feature sampling based on the geometric correspondence between the bird's-eye view and the image view. A sampling method combining BEV view-to-image view projection sampling and reference line sampling is used, and spatial cross-attention is calculated to generate the current bird's-eye view feature centered on the vehicle. A separable convolution is then used to improve the quality of the generated bird's-eye view features. The spatial attention module and the separable convolution module operate iteratively; the BEV Map generated after each iteration is used to calculate the loss for specific downstream tasks.

[0105] The acquisition of image features of each acquired image at the current moment, as described above (in some embodiments of this disclosure, image features are represented by three feature maps), includes:

[0106] Feature extraction is performed on each captured image at the current moment using a convolutional neural network to obtain the image features of each captured image from the vehicle-mounted camera at the current moment.

[0107] In the bird's-eye view feature generation method S100 based on vehicle surround view image disclosed herein, the convolutional neural network preferably includes a standard ResNet-101 backbone network, an FPN network and an additional convolutional layer.

[0108] The ResNet-101 backbone network is a four-stage ResNet-101 backbone network pre-trained using ImageNet.

[0109] In some embodiments of this disclosure, for example, it is assumed that the vehicle has N on-board cameras working, and each time N images are captured simultaneously, then each time N images will be input.

[0110] In some embodiments of this disclosure, the convolutional neural network of this disclosure includes a standard ResNet-101 backbone network, an FPN network, and an additional convolutional layer.

[0111] Assuming the input at time T (the current time) consists of N images, the input is fed into a standard ResNet-101 backbone network, and then through an FPN network and an additional convolutional layer to obtain the image features at time T. in It is the image feature of the i-th image captured by the vehicle-mounted camera at time T. For example, this disclosure assumes that all captured images have a pixel resolution of 800×1600.

[0112] This disclosure preferably pre-trains a four-stage ResNet-101 backbone network based on the following method: using an ImageNet-pretrained four-stage ResNet-101 as the backbone network, during model training, the weights of the first stage of ResNet-101 are frozen, and only the weights of the last three stages are updated. Simultaneously, the outputs of the last block of the last two stages are subjected to a single-group deformable convolution, and then the feature maps output from the last block of the last three stages are fed into an FPN network. All feature maps output by the FPN network undergo an additional convolution operation (i.e., using an additional convolutional layer). The feature maps after the FPN and the additional convolution are used as image features for spatial feature sampling.

[0113] Figure 3 This is an architecture diagram of an image feature extraction module according to one embodiment of the present disclosure.

[0114] refer to Figure 3For each image input to the image feature extraction module at time T (the current time) from the vehicle camera, three feature maps will be obtained and used as the original features sampled by the spatial attention module (described in detail below).

[0115] Figure 4 This is a schematic flowchart illustrating an embodiment of the present disclosure of obtaining the initialized bird's-eye view features at the current moment. (Refer to...) Figure 4 In some embodiments of this disclosure, S106 described above, obtaining the initialized bird's-eye view features at the current moment based on the bird's-eye view features of historical moments (temporal prior module), includes:

[0116] S1061. Randomly initialize a learnable bird's-eye view feature centered on the vehicle as the bird's-eye view feature at the current moment;

[0117] S1062. Add position encoding to the learnable bird's-eye view features to obtain the position-encoded bird's-eye view of the current frame;

[0118] S1063. Extract a bird's-eye view (BEV Map) of historical frames within a preset time period prior to the current moment;

[0119] S1064. Obtain the relative pose (i.e. pose change) between the bird's-eye view after the current frame is encoded and the bird's-eye view of the historical frames.

[0120] S1065. For each location point of the bird's-eye view query vector in the bird's-eye view after the current frame is encoded, based on the relative pose, find the corresponding historical frame bird's-eye view query vector (at least one) in the historical frame bird's-eye view as temporal prior information.

[0121] S1066. Append the temporal prior information to the position-encoded bird's-eye view of the current frame to obtain the initialized bird's-eye view features at the current moment (temporal prior module).

[0122] In the bird's-eye view feature generation method S100 based on vehicle surround view image in some embodiments of this disclosure, extracting historical frame bird's-eye view (BEV Map) within a preset time length before the current time includes: extracting one historical frame bird's-eye view or extracting two or more historical frame bird's-eye view.

[0123] In some embodiments of this disclosure, if two or more historical frame bird's-eye view images are extracted, the temporal prior information obtained from each historical frame bird's-eye view image is weighted and summed to obtain the final temporal prior information.

[0124] More specifically, for the generation of bird's-eye view features at the current moment, this disclosure first initializes a learnable BEV Map∈R centered on the vehicle. H×W×CWhere H is the height of the generated bird's-eye view, W is the width of the generated bird's-eye view, and C is the dimension of the generated bird's-eye view feature. For example, this disclosure sets H = W = 200 and C = 256.

[0125] This disclosure uses a temporal prior module to initialize the current frame's BEV Map (bird's-eye view) using the BEV Map (bird's-eye view) of historical frames. Specifically, this disclosure first randomly initializes a learnable BEV Map centered on the vehicle for the current frame and adds two-dimensional relative position encoding. Then, within the past 0.5 seconds (exemplarily), four different frames (i.e., multiple frames) are randomly selected as historical frames, and the relative pose (including position offset and / or motion direction offset) of the vehicle's current frame bird's-eye view and other historical frame bird's-eye views is obtained through IMU information (IMU (Inertial Measurement Unit)).

[0126] For each location point on the BEV Map of the current frame, it is treated as a BEV Query (bird's-eye view query vector), resulting in H*W BEV Queries, each with a dimension of R. 1×C Then, for each BEV Query in the current frame, the corresponding BEV Query is searched in the historical frames using pose information (i.e., the relative pose described above). That is, based on the position offset and / or motion direction offset, the BEV Query corresponding to the real-world position coordinates of the current frame's BEV Query is searched in the historical frames for the BEV Query corresponding to those real-world position coordinates.

[0127] It should be noted that the same real-world location point in the historical frame corresponding to the BEV Query of the current frame may be included in multiple BEV Query of the historical frame. Therefore, this disclosure preferably performs a weighted summation of the Query in the historical frame within an adjacent 3×3 (exemplary) Query range based on the ratio of the distances from the (same real-world location point) to different BEV Query. If the corresponding location point in the historical frame is on the boundary of the BEV Map (bird's-eye view), then the weighted summation of the Query in the historical frame is performed only within an adjacent 2×3 Query range. If the corresponding location point in the historical frame is at one of the four corner points of the BEV Map, then the weighted summation of the Query in the historical frame is performed only within an adjacent 2×2 Query range.

[0128] For each historical frame, the BEV Query of the current frame extracts the corresponding temporal prior information based on the relative pose described above. Then, the temporal prior information from multiple frames (multiple historical frames) is weighted and summed to obtain the final temporal prior information. In this disclosure, it is preferable to use the same weight for different historical frames. Finally, the obtained temporal prior information is added to the BEV Query of the corresponding current frame, thus obtaining the BEV Map∈R after the temporal prior module. H×W×C .

[0129] Figure 5 This is a schematic diagram illustrating the correspondence between the current frame BEV Query and the same real-world location in historical frames, according to one embodiment of this disclosure.

[0130] In some embodiments of this disclosure, it is worth noting that for queries without corresponding temporal information in the historical BEV Map, no prior information from the temporal sequence will be obtained. If the current frame is the first frame, the learnable BEV Map will only have two-dimensional relative position encoding added, and the temporal prior module described above in this disclosure will be skipped. If the current frame has fewer than four historical frames, only the existing historical frames will be used for temporal prior fusion.

[0131] In some embodiments of this disclosure, S108 described above, which involves performing spatial feature sampling (projection sampling from BEV viewpoint to image viewpoint + reference line sampling) on ​​the image features (high-dimensional image features) of each acquired image based on the bird's-eye view features after initialization at the current time, to obtain the sampled features (i.e., the corresponding features extracted from the image space by the bird's-eye view (BEV Map), includes: performing spatial feature sampling based on the geometric correspondence between the bird's-eye view and the image viewpoint.

[0132] Preferably, the spatial feature sampling based on the geometric correspondence between the bird's-eye view and the image view, as described above in this disclosure, includes:

[0133] Each location point in the current initial bird's-eye view is used as the bird's-eye view query vector (BEV Query) to perform spatial feature sampling of the feature maps of the images acquired by each vehicle-mounted camera based on geometric projection.

[0134] In some embodiments of this disclosure, the BEV Map∈R initialized by the time attention module (i.e., the temporal prior module described above) H×W×C Similarly, if each location point on the BEV Map is used as a BEV Query, we will obtain H*W BEV Queries, each with a dimension of R. 1×C .

[0135] For each BEV Query, its position on the BEV Map is geometrically projected onto a feature map (used to represent image features) extracted by a convolutional neural network based on the camera's intrinsic and extrinsic parameters for spatial feature sampling. For example, this disclosure assumes that each grid in the BEV Map corresponds to a real-world dimension of S meters. By default, the feature center of the BEV Map corresponds to the vehicle's position. Therefore, for any BEV Query, if its coordinates on the BEV Map are (x, y), then its coordinates projected onto the real-world top view are (x′, y′):

[0136]

[0137] Where (x, y) is the origin of the coordinate system on the BEV Map with the lower left corner as the origin, and (x′, y′) is the origin of the coordinate system on the real-world top view with the vehicle as the origin. In this disclosure, S is exemplarily set to 0.5.

[0138] For example, for each coordinate point (x′, y′), R sampling points are averaged from (-5m, 5m) along the z-axis. That is, for each BEV Query, there will be R real-world 3D coordinate points. Then, each 3D coordinate point can be projected onto one or more images captured by the vehicle-mounted camera using the camera's intrinsic and extrinsic parameters, resulting in multiple image coordinate points (p). x p y That is, the image projection coordinates. For example, this disclosure sets R = 8.

[0139] For each BEV Query, multiple corresponding image projection coordinate points can be obtained, as shown in the reference. Figure 6 . Figure 6 This is a projection diagram of the BEV viewpoint and image viewpoint position according to one embodiment of the present disclosure.

[0140] It should be noted that for each BEV Query, the R real-world 3D coordinate points generated will be projected based on the intrinsic and extrinsic parameters of each vehicle camera. However, since the viewpoints of at most two cameras overlap, each BEV Query will be projected onto at most two vehicle images. The projection points of other images will be projected outside the image range. Also, since the value range in the z-axis direction is fixed, not all R projection points on an image may be projected onto the image. Projection points projected outside the image will be discarded.

[0141] According to a preferred embodiment of this disclosure, the spatial feature sampling based on the geometric correspondence between the bird's-eye view and the image view described above further includes:

[0142] Using each vertical line of the feature map representing the image features of each acquired image as a reference line, the polar ray features centered on the vehicle in the bird's-eye view are obtained, and then the bird's-eye view query vector (BEV Query) is obtained for spatial feature sampling based on the reference line.

[0143] This disclosure takes into account that simply using the corresponding projection point projection method cannot make good use of the geometric relationships in the real world. Therefore, in some preferred embodiments of this disclosure, a reference line sampling projection method is also introduced.

[0144] In the geometric relationship between the image view and the BEV view, each vertical line feature in the image view corresponds to the same geometric relationship as each polar ray centered on the vehicle in the BEV view. Therefore, by using the vertical lines in the image as reference lines and sampling the reference line features, we can introduce real-world geometric prior information to help generate a more robust BEV Map.

[0145] Generally speaking, the generated bird's-eye view features are centered on the vehicle itself. Therefore, the polar ray centered on the vehicle in the bird's-eye view features has a 360° field of view. However, the field of view of the vehicle camera is narrower. Within the same field of view of the vehicle camera and the polar ray centered on the vehicle, the geometric relationship of each vertical line feature of the image view is the same as that of each polar ray centered on the vehicle in the BEV view.

[0146] In other words, the vertical line features from left to right in the image correspond to the polarimetric rays from right to left on the BEVMap, centered on the vehicle, within the corresponding camera's field of view. Figure 7 , Figure 7 This is a schematic diagram of a reference line sampling method according to one embodiment of the present disclosure.

[0147] It is worth noting that since the corresponding features extracted on the BEV Map are polar ray features, the corresponding feature extraction range may fall between multiple BEV Queries. Therefore, the corresponding feature sampling method is: each vertical line feature of the image features remains unchanged, and its corresponding bird's-eye view features are the BEV Queries of the nearest landing point of each horizontal line of the BEV Map and its left and right adjacent BEV Queries are all taken as the corresponding features.

[0148] Therefore, the feature sampling method based on reference lines will, for each vertical line feature on the feature map, obtain the following on the H×W bird's-eye view feature: or The BEV Query. The specific quantity, whether it's H or W, depends on whether the direction corresponding to the polarimetric rays is horizontal or vertical. Dividing by 2 is because the vehicle is at the center point of the bird's-eye view.

[0149] The above section introduced the BEV viewpoint to image viewpoint projection sampling and reference line sampling methods. Both of these sampling methods will help the BEV Map extract the corresponding features from the image space.

[0150] The method described above, which involves performing spatial cross-attention on the initialized bird's-eye view features at the current moment and the sampled features to generate bird's-eye view features centered on the vehicle at the current moment, preferably includes:

[0151] For each BEV query vector in the BEV Map, the BEV viewpoint is projected to the image viewpoint and the sampled features are then subjected to cross-attention of the sampling points.

[0152] Reference lines are sampled for each polar ray centered on the vehicle on the bird's-eye view (BEV Map), and reference line intersection attention is performed;

[0153] Output the current moment's bird's-eye view features centered on this vehicle.

[0154] In some embodiments of this disclosure, preferably, the sampled features are cross-attentionally fused with the BEVMap using a spatial attention module to generate a reliable current frame BEVMap.

[0155] In some preferred embodiments of this disclosure, the spatial attention module includes two parts: sampling point cross-attention and reference line cross-attention. For a bird's-eye view feature input (the bird's-eye view feature after initialization at the current moment), firstly, projection sampling from the BEV viewpoint to the image viewpoint is performed on each BEV Query in the bird's-eye view (BEV Map), and sampling point cross-attention is performed on the sampled features. Then, reference line sampling is performed on each polar ray centered on the vehicle on the BEV Map, and reference line cross-attention is performed to obtain the final output of the spatial attention module. Figure 8 , Figure 8 A schematic diagram of the spatial attention module according to one embodiment of this disclosure is shown.

[0156] For the features sampled by projecting from the BEV viewpoint to the image viewpoint using the BEV query vector in the BEV map as described above, preferably, the following steps are taken:

[0157] Obtain m projection points for each bird's-eye view query vector, with each projection point serving as a feature sampling point;

[0158] Based on the features of each projection point, k position offsets are predicted. Each position offset is added to the coordinates corresponding to the projection point to obtain an additional k feature sampling points. Thus, (k+1) feature sampling points are obtained for each projection point, and m×(k+1) feature sampling points are obtained for each bird's-eye view query vector.

[0159] The bird's-eye view query vector (BEV Query) is used as the query vector for cross-attention. The sampling points and sampling point features are used as the key and value of cross-attention, respectively. Cross-attention is calculated to update the bird's-eye view query vector (BEV Query).

[0160] In a preferred embodiment of this disclosure, the sampling point cross-attention updates each BEV Query. First, it is assumed that a BEV Query will be projected onto m projection points on the image features. For each projection point, k offset displacements are predicted from the corresponding features. Each offset position, plus the coordinates of the projection point, yields k feature sampling points. Thus, each projection point receives an additional k feature sampling points, and each BEV Query receives m×(k+1) feature sampling points. Then, using the BEV Query as the Query for cross-attention and the sampling point features as the Key and Value, cross-attention is calculated to obtain the updated BEV Query.

[0161]

[0162] Where Q = Query * W q K = Key * W k V = Value * W v W q W k W v All are learnable matrices. It is worth noting that the number of projections of each BEV Query onto the image features is not fixed, i.e., m is not fixed, while the number of bias displacements predicted by each projection point is fixed. In this disclosure, k = 4 (adjustable).

[0163] For the above-described sampling of reference lines along each polar ray centered on the vehicle on the BEV Map, and the application of reference line cross-attention, preferably, it includes:

[0164] Each polar ray and its adjacent bird's-eye view query vector (BEV Query) are used as the query vector for cross-attention, and cross-attention is calculated using the vertical lines from left to right on the image features (feature map) and the vertical line features as the key and value, in order to update the bird's-eye view query vector (BEV Query).

[0165] In a preferred embodiment of this disclosure, after sampling point cross attention is reference line cross attention. Instead of using a single BEV Query as the cross attention Query, it uses the epipolar ray and its adjacent BEV Query as the cross attention Query. At the same time, the cross attention is calculated using the vertical line features from left to right on the image feature map as the Key and Value. The calculation method is the same as that of sampling point cross attention described above.

[0166] The method described above for establishing connections between different channels of the bird's-eye view features at the current time based on a separable convolutional neural network to obtain enhanced bird's-eye view features at the current time preferably includes:

[0167] Keep the channel dimension of the bird's-eye view features unchanged, so that the number of convolutional kernels in the separable convolutional neural network is the same as the channel dimension of the bird's-eye view features.

[0168] In downstream perception tasks of bird's-eye view features, it is generally believed that different channels store different information. Therefore, passing a separable convolutional neural network after spatial cross-attention will help generate high-quality bird's-eye view features.

[0169] While generating high-quality bird's-eye view features, it is necessary to keep the channel dimension of the features unchanged. Therefore, the number of convolutional kernels is the same as the channel dimension and they are placed after the spatial attention module.

[0170] This disclosure also provides a bird's-eye view feature generation apparatus (i.e., model architecture) based on vehicle surround view images.

[0171] refer to Figure 9 The bird's-eye view feature generation device 1000 based on vehicle surround view image disclosed herein includes:

[0172] Image acquisition module 1002 extracts images captured by all on-board cameras of the vehicle at the current moment;

[0173] Image feature extraction module 1004 acquires the image features of each acquired image at the current moment;

[0174] The temporal prior module 1006 obtains the initialized bird's-eye view features at the current moment based on the bird's-eye view features at historical moments.

[0175] The sampling module 1008 performs spatial feature sampling on the image features of each acquired image based on the bird's-eye view features after initialization at the current time, so as to obtain the sampled features.

[0176] The spatial cross-attention module 1010 performs spatial cross-attention on the initial bird's-eye view features at the current moment and the sampled features to generate bird's-eye view features centered on the vehicle at the current moment.

[0177] Separable convolutional module 1012 establishes connections between different channels of the bird's-eye view features at the current time to obtain enhanced bird's-eye view features at the current time.

[0178] In the model architecture of the bird's-eye view feature generation device 1000 based on vehicle surround view image disclosed herein, the method of generating bird's-eye view features by iterative sampling is used. For bird's-eye view features initialized by time sequence (based on time sequence prior module), after passing through spatial attention module and separable convolution module, the output of bird's-eye view features will be obtained for the first time.

[0179] To enhance the learning ability of bird's-eye view features on spatial features, this disclosure re-feeds the generated bird's-eye view features into the spatial cross-attention module and the separable convolution module, and re-samples and calculates spatial attention for spatial features. In this disclosure, spatial feature sampling is iterated 6 times (adjustable).

[0180] Each time spatial attention sampling is performed to generate bird's-eye view features, multi-task loss can be calculated on the bird's-eye view features, and existing 2D / 3D detection, segmentation and other task heads can be well inherited.

[0181] In some embodiments of this disclosure, the model training uses only the DETR3D 3D object detection head, employing a 6-layer decoder method as the 3D object detection head. Each Object Query predicts the object's center point on the BEV Map to achieve 3D object detection. The task loss is calculated using the 3D object detection head on the bird's-eye view features generated in each iteration.

[0182] The bird's-eye view feature generation device 1000 based on vehicle surround view images disclosed herein can be implemented based on a computer software architecture.

[0183] Figure 9This is a schematic block diagram of a bird's-eye view feature generation device 1000 based on vehicle surround view image, which is a hardware implementation of a processing system according to one embodiment of the present disclosure.

[0184] The bird's-eye view feature generation device 1000 based on vehicle surround view images may include corresponding modules that perform one or more steps in the flowchart described above. Therefore, each or more steps in the flowchart can be performed by a corresponding module, and the device may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0185] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0186] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0187] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0188] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0189] This disclosure also provides an electronic device, including: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform a bird's-eye view feature generation method based on vehicle surround view images according to any embodiment of this disclosure.

[0190] This disclosure also provides a readable storage medium storing execution instructions, which, when executed by a processor, are used to implement the bird's-eye view feature generation method based on vehicle surround view images according to any embodiment of this disclosure.

[0191] This disclosure also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements a bird's-eye view feature generation method based on vehicle surround view images according to any embodiment of this disclosure.

[0192] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0193] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0194] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware. The program can be stored in a readable storage medium, and when executed, the program includes one or a combination of the steps of the method implementation.

[0195] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0196] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0197] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0198] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method for generating bird's-eye view features based on vehicle surround view images, characterized in that, include: Extract images captured by all onboard cameras of this vehicle at the current moment; Obtain the image features of each captured image at the current moment; Obtain the initialized bird's-eye view features at the current moment based on the bird's-eye view features at historical moments; Based on the bird's-eye view features after initialization at the current moment, spatial feature sampling is performed on the image features of each acquired image to obtain the sampled features; Spatial cross-attention is performed between the initialized bird's-eye view features at the current moment and the sampled features to generate bird's-eye view features centered on the vehicle at the current moment. as well as Based on the separable convolutional neural network, connections are established between different channels of the bird's-eye view features at the current time to obtain enhanced bird's-eye view features at the current time. Based on the aerial view features from historical moments, the initial aerial view features for the current moment are obtained, including: Randomly initialize a learnable bird's-eye view feature centered on the vehicle as the bird's-eye view feature for the current moment; Add positional encoding to the learnable bird's-eye view features to obtain a position-encoded bird's-eye view of the current frame; Extract bird's-eye view images of historical frames within a preset time period prior to the current moment; Obtain the relative pose of the bird's-eye view after the current frame's position encoding and the bird's-eye view of the historical frames; For each location point in the current frame's position-encoded bird's-eye view query vector, based on the relative pose, the corresponding historical frame bird's-eye view query vector is searched in the historical frame bird's-eye view as temporal prior information; and By appending the temporal prior information to the position-encoded bird's-eye view of the current frame, the initialized bird's-eye view features at the current moment are obtained.

2. The method for generating bird's-eye view features based on vehicle surround view images according to claim 1, characterized in that, Obtain the image features of each acquired image at the current moment, including: Feature extraction is performed on each captured image at the current moment using a convolutional neural network to obtain the image features of each captured image from the vehicle-mounted camera at the current moment.

3. The method for generating bird's-eye view features based on vehicle surround view images according to claim 2, characterized in that, The convolutional neural network includes a standard ResNet-101 backbone network, an FPN network, and an additional convolutional layer; The ResNet-101 backbone network is a four-stage ResNet-101 backbone network pre-trained using ImageNet.

4. The method for generating bird's-eye view features based on vehicle surround view images according to claim 1, characterized in that, Extract a bird's-eye view of historical frames within a preset time period prior to the current moment, including: Extract one or more historical frames from the bird's-eye view.

5. The method for generating bird's-eye view features based on vehicle surround view images according to claim 4, characterized in that, If two or more historical frame bird's-eye view images are extracted, the temporal prior information obtained from each historical frame bird's-eye view image is weighted and summed to obtain the final temporal prior information.

6. The method for generating bird's-eye view features based on vehicle surround view images according to claim 1, characterized in that, Based on the initialized bird's-eye view features at the current moment, spatial feature sampling is performed on the image features of each acquired image to obtain the sampled features, including: The spatial features are sampled based on the geometric correspondence between the bird's-eye view and the image view.

7. The method for generating bird's-eye view features based on vehicle surround view images according to claim 6, characterized in that, The spatial feature sampling based on the geometric correspondence between the bird's-eye view and the image view includes: Each location point in the current initialized bird's-eye view is used as a bird's-eye view query vector to perform spatial feature sampling of the feature maps of the images acquired by each vehicle-mounted camera based on geometric projection.

8. The method for generating bird's-eye view features based on vehicle surround view images according to claim 6, characterized in that, The spatial feature sampling is performed based on the geometric correspondence between the bird's-eye view and the image view, including: Using each vertical line of the feature map representing the image features of each acquired image as a reference line, the polar ray features centered on the vehicle in the bird's-eye view are obtained, thereby obtaining the bird's-eye view query vector for spatial feature sampling based on the reference line.

9. The method for generating bird's-eye view features based on vehicle surround view images according to claim 1, characterized in that, The initialized bird's-eye view features at the current moment are spatially cross-attentioned with the sampled features to generate a bird's-eye view feature centered on the vehicle at the current moment, including: For each bird's-eye view query vector in the bird's-eye view, the BEV view is projected to the image view and the sampled features are then subjected to cross-attention at the sampling points. Reference lines are sampled for each polarimeter centered on the vehicle on the bird's-eye view, and reference line intersection attention is performed; and Output the current moment's bird's-eye view features centered on this vehicle.

10. The method for generating bird's-eye view features based on vehicle surround view images according to claim 9, characterized in that, For each bird's-eye view query vector in the bird's-eye view, the features sampled by projection sampling from the BEV viewpoint to the image viewpoint are subjected to sampling point cross-attention, including: Obtain m projection points for each bird's-eye view query vector, with each projection point serving as a feature sampling point; Based on the features of each projection point, k position offsets are predicted. Each position offset is added to the coordinates corresponding to the projection point to obtain k additional feature sampling points. Therefore, k+1 feature sampling points are obtained for each projection point, and m×(k+1) feature sampling points are obtained for each bird's-eye view query vector; and The bird's-eye view query vector is used as the query vector for cross-attention, and the sampling points and sampling point features are used as the keys and values ​​for cross-attention, respectively. Cross-attention is calculated to update the bird's-eye view query vector.

11. The method for generating bird's-eye view features based on vehicle surround view images according to claim 9, characterized in that, For each polar ray centered on the vehicle in the bird's-eye view, reference lines are sampled, and reference line intersection attention is performed, including: Each polar ray and its adjacent bird's-eye view query vector are used as the query vector for cross-attention, and cross-attention is calculated using the vertical lines from left to right in the image features and the vertical line features as keys and values ​​to update the bird's-eye view query vector.

12. The method for generating bird's-eye view features based on vehicle surround view images according to claim 1, characterized in that, Based on a separable convolutional neural network, connections are established between different channels of the bird's-eye view features at the current time to obtain enhanced bird's-eye view features at the current time, including: Keep the channel dimension of the bird's-eye view features unchanged, so that the number of convolutional kernels in the separable convolutional neural network is the same as the channel dimension of the bird's-eye view features.

13. A bird's-eye view feature generation device based on vehicle surround view images, characterized in that, include: The image acquisition module extracts images captured by all the vehicle's onboard cameras at the current moment; An image feature extraction module acquires the image features of each acquired image at the current moment; A temporal prior module, which obtains the initialized bird's-eye view features at the current moment based on the bird's-eye view features at historical moments; The sampling module performs spatial feature sampling on the image features of each acquired image based on the initialized bird's-eye view features at the current moment, so as to obtain the sampled features; A spatial cross-attention module performs spatial cross-attention between the initialized bird's-eye view features at the current moment and the sampled features to generate bird's-eye view features centered on the vehicle at the current moment. as well as A separable convolutional module establishes connections between different channels of the bird's-eye view features at the current moment to obtain enhanced bird's-eye view features at the current moment; Based on the aerial view features from historical moments, the initial aerial view features for the current moment are obtained, including: Randomly initialize a learnable bird's-eye view feature centered on the vehicle as the bird's-eye view feature for the current moment; Add positional encoding to the learnable bird's-eye view features to obtain a position-encoded bird's-eye view of the current frame; Extract bird's-eye view images of historical frames within a preset time period prior to the current moment; Obtain the relative pose of the bird's-eye view after the current frame's position encoding and the bird's-eye view of the historical frames; For each location point in the current frame's position-encoded bird's-eye view query vector, based on the relative pose, the corresponding historical frame bird's-eye view query vector is searched in the historical frame bird's-eye view as temporal prior information; and By appending the temporal prior information to the position-encoded bird's-eye view of the current frame, the initialized bird's-eye view features at the current moment are obtained.

14. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes the execution instructions stored in the memory, causing the processor to perform the bird's-eye view feature generation method based on vehicle surround view images as described in any one of claims 1 to 12.

15. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the bird's-eye view feature generation method based on vehicle surround view images as described in any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the bird's-eye view feature generation method based on vehicle surround view image as described in any one of claims 1 to 12.

Images