Methods, devices, electronic devices, and storage media for converting bird's-eye view perspectives

Abstract

The present application relates to a method, device, electronic device and storage medium for converting a bird's-eye view perspective. The method comprises: extracting image features and depth features from a surround multi-camera perspective image; performing an outer product operation on the image features and the depth features to obtain a view cone point cloud feature; performing a dimensional transformation on the view cone point cloud feature to obtain a preliminary bird's-eye view feature at each depth; performing depth aggregation on each of the preliminary bird's-eye view features to obtain a depth-aggregated bird's-eye view feature; performing multi-perspective aggregation on the depth-aggregated bird's-eye view features to obtain a multi-perspective aggregated bird's-eye view feature; upsampling the multi-perspective aggregated bird's-eye view feature to obtain a bird's-eye view feature of the multi-camera perspective image. The use of this method can improve the conversion efficiency of the bird's-eye view perspective.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, electronic device and storage medium for converting a bird's-eye view. Background Art

[0002] With the continuous development of technologies such as intelligent driving and target detection, the algorithm research on bird's eye view (BEV) perspective conversion has become more and more extensive. BEV perspective conversion mainly projects the image features of the camera perspective to the BEV perspective, thereby applying it to downstream tasks.

[0003] The traditional lift-splat perspective conversion method mainly includes: inputting the surround multi-camera image into the backbone network for image feature extraction, and the depth estimation network uses the extracted multi-camera image features to perform depth estimation to obtain the depth estimation value of each pixel; then, the depth estimation value and the image feature are outer-producted to obtain the feature of the view cone point cloud, and the view cone point cloud is hashed into the BEV grid according to the geometric position relationship. If the number of view cone point clouds in a BEV grid is greater than 1, the features of all view cone point clouds are added and summed to obtain the BEV feature. However, the lift-splat perspective conversion requires the Neural Network Processing Unit (NPU) to support operators such as cumsum, gather / gatherND, and scather / scatherND, while most NPUs cannot fully support these operators, and even if they support these operators, the execution speed is very slow, which affects the conversion efficiency of the BEV perspective. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, electronic device and storage medium for converting a bird's-eye view perspective in order to address the above technical problems, so as to improve the conversion efficiency of the bird's-eye view perspective.

[0005] In a first aspect, the present application provides a method for converting a bird's-eye view perspective, the method comprising:

[0006] Extract image features and depth features from surround multi-camera view images;

[0007] Performing an outer product operation on the image feature and the depth feature to obtain a view frustum point cloud feature;

[0008] Performing dimension transformation on the view cone point cloud features to obtain preliminary bird's-eye view features at each depth;

[0009] Performing deep aggregation on each of the preliminary bird's-eye view features to obtain deeply aggregated bird's-eye view features;

[0010] Performing multi-view aggregation on the deeply aggregated bird's-eye view features to obtain multi-view aggregated bird's-eye view features;

[0011] The multi-view aggregated bird's-eye view features are upsampled to obtain the bird's-eye view features of the multi-camera view image.

[0012] In one embodiment, extracting image features and depth features from the surround multi-camera view images includes:

[0013] Image features are extracted from surround multi-camera view images through a feature extraction network;

[0014] Depth features are extracted from image features of the multi-camera view images through a depth estimation network.

[0015] In one embodiment, the method further comprises:

[0016] Flattening the view cone point cloud feature in the horizontal direction and the vertical direction to obtain a flattened view cone point cloud feature;

[0017] The dimension transformation of the view cone point cloud features to obtain preliminary bird's-eye view features at each depth includes:

[0018] The flattened view cone point cloud features are input into a transformation model, so that the transformation model performs a dimensional transformation on the flattened view cone point cloud features to obtain a preliminary bird's-eye view feature at each depth.

[0019] In one embodiment, the performing deep aggregation on each of the preliminary bird's-eye view features to obtain the deeply aggregated bird's-eye view features comprises:

[0020] Each of the preliminary bird's-eye view features is input into a deep aggregation model, so that the deep aggregation model performs deep aggregation on each of the preliminary bird's-eye view features to obtain deeply aggregated bird's-eye view features.

[0021] In one embodiment, the method further comprises:

[0022] Rearranging the dimensions of the deeply aggregated bird's-eye view features to obtain rearranged deeply aggregated bird's-eye view features;

[0023] The step of performing multi-view aggregation on the bird's-eye view features of the depth aggregation to obtain the multi-view aggregated bird's-eye view features includes:

[0024] The rearranged depth-aggregated bird's-eye view features are input into a camera aggregation model, so that the camera aggregation model performs multi-view aggregation on the rearranged depth-aggregated bird's-eye view features to obtain multi-view aggregated bird's-eye view features.

[0025] In one of the embodiments, the multi-camera perspective image includes a multi-camera perspective driving image or a multi-camera perspective environment image;

[0026] After upsampling the bird's-eye view features of the multi-view aggregation to obtain the bird's-eye view features of the multi-camera view image, the method further includes:

[0027] Perform lane line detection on the multi-camera perspective driving image based on the bird's-eye view features of the multi-camera perspective image to obtain a lane line detection result of the vehicle's environment; or

[0028] Target detection is performed on the multi-camera perspective environment image based on the bird's-eye view features of the multi-camera perspective image to obtain a detection result of the target object in the environment.

[0029] In a second aspect, the present application further provides a device for converting a bird's-eye view perspective, the device comprising:

[0030] An extraction module, used to extract image features and depth features from surround multi-camera view images;

[0031] A calculation module, used for performing an outer product operation on the image feature and the depth feature to obtain a view frustum point cloud feature;

[0032] A transformation module, used for performing dimension transformation on the view cone point cloud features to obtain preliminary bird's-eye view features at each depth;

[0033] A deep aggregation module, used for performing deep aggregation on each of the preliminary bird's-eye view features to obtain deeply aggregated bird's-eye view features;

[0034] A perspective aggregation module, used for performing multi-perspective aggregation on the bird's-eye view features of the depth aggregation to obtain multi-perspective aggregated bird's-eye view features;

[0035] The upsampling module is used to upsample the bird's-eye view features of the multi-view aggregation to obtain the bird's-eye view features of the multi-camera view image.

[0036] In one of the embodiments, the extraction module is further used to extract image features from the surround multi-camera view images through a feature extraction network; and to extract depth features from the image features of the multi-camera view images through a depth estimation network.

[0037] In one embodiment, the device further comprises:

[0038] A flattening module, used for flattening the view cone point cloud features in the horizontal direction and the vertical direction to obtain the flattened view cone point cloud features;

[0039] The transformation module is further used to input the flattened view cone point cloud features into the transformation model, so that the transformation model performs dimensional transformation on the flattened view cone point cloud features to obtain preliminary bird's-eye view features at each depth.

[0040] In one of the embodiments, the deep aggregation module is further used to input each of the preliminary bird's-eye view features into a deep aggregation model, so that the deep aggregation model performs deep aggregation on each of the preliminary bird's-eye view features to obtain deeply aggregated bird's-eye view features.

[0041] In one embodiment, the device further comprises:

[0042] A rearrangement module, used for rearranging the dimensions of the deeply aggregated bird's-eye view features to obtain rearranged deeply aggregated bird's-eye view features;

[0043] The perspective aggregation module is also used to input the rearranged depth-aggregated bird's-eye view features into a camera aggregation model, so that the camera aggregation model performs multi-perspective aggregation on the rearranged depth-aggregated bird's-eye view features to obtain multi-perspective aggregated bird's-eye view features.

[0044] In one embodiment, the multi-camera perspective image includes a multi-camera perspective driving image or a multi-camera perspective environment image; the device further includes:

[0045] A detection module is used to perform lane line detection on the multi-camera perspective driving image based on the bird's-eye view features of the multi-camera perspective image to obtain lane line detection results of the vehicle's environment; and perform target detection on the multi-camera perspective environment image based on the bird's-eye view features of the multi-camera perspective image to obtain detection results of target objects in the environment.

[0046] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method for converting the bird's-eye view perspective when executing the computer program.

[0047] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for converting the bird's-eye view perspective are implemented.

[0048] In a fifth aspect, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the method for converting the bird's-eye view perspective are implemented.

[0049] The above-mentioned bird's-eye view perspective conversion method, device, computer equipment, storage medium and computer program product first extract image features and depth features from the surround multi-camera perspective images; perform outer product operation on the image features and depth features to obtain the view cone point cloud features; then, perform dimensionality transformation on the view cone point cloud features to obtain preliminary bird's-eye view features at each depth; perform depth aggregation on each preliminary bird's-eye view feature to obtain depth-aggregated bird's-eye view features; perform multi-perspective aggregation on the depth-aggregated bird's-eye view features to obtain multi-perspective aggregated bird's-eye view features; upsample the multi-perspective aggregated bird's-eye view features to obtain bird's-eye view features of multi-camera perspective images, and the above-mentioned transformation processing, depth aggregation, multi-perspective aggregation and upsampling use linear operators, which are also supported by the neural network processing unit and have fast execution speed, thereby improving the conversion efficiency of the bird's-eye view; moreover, depth and perspective decoupling calculation is used to reduce the computational complexity, while also ensuring the expressiveness of the bird's-eye view features, which is conducive to improving the accuracy of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A diagram showing an application environment of a method for converting a bird's-eye view perspective in one embodiment;

[0051] Figure 2 A schematic diagram of a process of converting a bird's-eye view perspective in one embodiment;

[0052] Figure 3 A schematic diagram of a flow chart of a method for converting a bird's-eye view perspective in another embodiment;

[0053] Figure 4 is a structural block diagram of a device for converting a bird's-eye view perspective in one embodiment;

[0054] Figure 5 It is a structural block diagram of a device for converting a bird's-eye view perspective in another embodiment;

[0055] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] The method for converting the bird's-eye view perspective provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, the application environment includes a terminal 102 and a server 104, and the terminal 102 communicates with the server 104 through a network. It should be noted that the above-mentioned bird's-eye view perspective conversion method can also be applied to an application environment that only includes a terminal 102 or a server 104.

[0058] It should be noted that the data storage system can store data that needs to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0059] The terminal 102 may be an onboard device in a motor vehicle (including a fuel vehicle and a new energy vehicle), a smart phone, a tablet computer, a laptop computer, a desktop computer, an Internet of Things device, and a portable wearable device.

[0060] Server 104 can be an independent physical server or a service node in a blockchain system. Each service node in the blockchain system forms a peer-to-peer network. The peer-to-peer protocol is an application layer protocol running on the Transmission Control Protocol (TCP). In addition, server 104 can also be a server cluster composed of multiple physical servers, which can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0061] In one embodiment, Figure 2 As shown, a method for converting the perspective of a bird's-eye view is provided. The method can be Figure 1 The method is performed by a server or a terminal in the embodiment, or by a server and a terminal in collaboration. Figure 1 The following steps are used as an example to illustrate the terminal execution in:

[0062] S202, extracting image features and depth features from the surround multi-camera view images.

[0063] Among them, the multi-camera perspective image can refer to the image captured by multiple cameras with different perspectives, which can be called the surround multi-camera image. The above multiple cameras can be used to capture images of all directions under the position of the camera, such as collecting images in the four directions of the front, back, left, and right of the position, or even more directions to ensure that the environment around the position is captured; or, the multi-camera perspective image can be an image captured by a camera with different perspectives during the rotation along the target axis. In some actual application scenarios, the multi-camera perspective image can include at least one of a multi-camera perspective driving image or a multi-camera perspective environment image.

[0064] The depth feature may include a depth estimation value of each pixel in the image feature of the multi-camera view image.

[0065] In one embodiment, the terminal may extract image features from the surround multi-camera view images through a feature extraction network; and extract depth features from the image features of the multi-camera view images through a depth estimation network.

[0066] The feature extraction network may be a neural network for extracting image features including a backbone network and a pyramid network. The depth estimation network may be a deep prediction network based on a fully convolutional residual network (FCRN) or a deep ordinal regression network for monocular depth estimation (Deep Ordinal Regression Network for Monocular Depth Estimation).

[0067] Before extracting image features, the terminal may also perform image enhancement processing on the multi-camera view images, such as denoising, increasing brightness and contrast, etc.; then, the enhanced multi-camera view images are input into a feature extraction network, which can extract image features from the enhanced multi-camera view images, and then input the image features into a depth estimation network for depth estimation to obtain depth features.

[0068] For example, Figure 3 As shown, the terminal will scale to The multi-camera view images are input into the feature extraction network. The backbone network of the feature extraction network can extract the corresponding features from the multi-camera view images, and then downsample them through the pyramid network to obtain the scale of Then, the image feature is input into the depth estimation network, and the depth estimation network uses the image feature to perform depth estimation, and obtains a scale of It should be noted that the above , , and Represents a dimension respectively, where It can be the number of cameras or the number of camera angles. and are the height and width of the multi-camera view images, 3 represents the number of image channels, represents the number of feature channels, Indicates the depth quantity (that is, the number of depth values).

[0069] S204, performing outer product operation on the image features and the depth features to obtain the view frustum point cloud features.

[0070] The view cone point cloud feature may be a point cloud feature of the view cone space.

[0071] For example, Figure 3 As shown, the terminal performs outer product operations on image features and depth features to obtain a scale of The view frustum point cloud features.

[0072] S206, performing dimension transformation on the view cone point cloud features to obtain preliminary bird's-eye view features at each depth.

[0073] The preliminary bird's-eye view features may refer to the bird's-eye view features obtained through preliminary processing (ie, the bird's-eye view spatial features).

[0074] In one embodiment, the terminal may flatten the view cone point cloud features in the horizontal and vertical directions to obtain flattened view cone point cloud features; then, the flattened view cone point cloud features are input into the transformation model so that the transformation model performs dimensional transformation on the flattened view cone point cloud features to obtain preliminary bird's-eye view features at each depth.

[0075] Herein, the transformation model may refer to a dimensional transformation model including a fully connected layer.

[0076] For example, Figure 3 As shown, the terminal can flatten the view cone point cloud features in the horizontal and vertical directions to obtain the flattened view cone point cloud features, and then input the flattened view cone point cloud features into the transformation model, so that the transformation model processes the flattened view cone point cloud features through its internal fully connected layer to obtain a scale of It should be noted that , and Represents a dimension respectively, where represents the height of the bird's-eye view feature, represents the width of the bird's eye view feature, The number of channels representing the bird's-eye view features, is a positive integer greater than 1.

[0077] S208 , performing deep aggregation on each preliminary bird's-eye view feature to obtain a deeply aggregated bird's-eye view feature.

[0078] The depth aggregation of the preliminary bird's-eye view features may be: aggregating the preliminary bird's-eye view features of different depths, thereby obtaining the bird's-eye view features aggregated with different depths.

[0079] In one embodiment, the terminal may input each preliminary bird's-eye view feature into a deep aggregation model, so that the deep aggregation model performs deep aggregation on each preliminary bird's-eye view feature, thereby completing the deep decoupling calculation and obtaining the deeply aggregated bird's-eye view features.

[0080] Among them, the deep aggregation model can be a network model including a fully connected layer for deep aggregation.

[0081] For example, Figure 3 As shown, the terminal can input the preliminary bird's-eye view features at each depth into the deep aggregation model, and the deep aggregation model processes each preliminary bird's-eye view feature through a fully connected layer to obtain a scale of deep aggregation: Bird's eye view feature.

[0082] S210 , performing multi-view aggregation on the bird's-eye view features obtained by depth aggregation to obtain multi-view aggregated bird's-eye view features.

[0083] The multi-view aggregation of the deeply aggregated bird's-eye view features may be: aggregating the deeply aggregated bird's-eye view features of different view angles, so as to obtain the bird's-eye view features aggregated from different view angles.

[0084] In one embodiment, the terminal may input the depth-aggregated bird's-eye view features into a camera aggregation model so that the camera aggregation model performs multi-perspective aggregation on the depth-aggregated bird's-eye view features, thereby completing the decoupling calculation of the perspectives and obtaining multi-perspective aggregated bird's-eye view features.

[0085] In another embodiment, the terminal may first re-arrange the dimensions of the deeply aggregated bird's-eye view features to obtain rearranged deeply aggregated bird's-eye view features; then, the rearranged deeply aggregated bird's-eye view features are input into the camera aggregation model, so that the camera aggregation model performs multi-perspective aggregation on the rearranged deeply aggregated bird's-eye view features to obtain multi-perspective aggregated bird's-eye view features.

[0086] The dimension rearrangement may be to rearrange corresponding dimension features in the deeply aggregated bird's-eye view features to other positions.

[0087] For example, the terminal can first rearrange the dimensions of the bird's-eye view features of the deep aggregation, such as The bird's-eye view features are rearranged into The bird's eye view feature will then The bird's-eye view features are input into the camera aggregation model for multi-view aggregation, and the scale is ( The bird's-eye view features of multi-view aggregation, such as Figure 3 shown.

[0088] S212, up-sampling the bird's-eye view features aggregated from multiple perspectives to obtain bird's-eye view features of multi-camera perspective images.

[0089] In one embodiment, the terminal may upsample the bird's-eye view features of multi-view aggregation through an upsampling network to obtain bird's-eye view features of multi-camera view images, such as upsampling the scale ( The bird's-eye view features of the multi-view aggregation are upsampled to obtain the scale of the multi-camera view image. Bird's eye view feature.

[0090] In one embodiment, when the multi-camera perspective image is a multi-camera perspective driving image, after obtaining the bird's-eye view features of the multi-camera perspective image, the terminal can perform lane line detection on the multi-camera perspective driving image based on the bird's-eye view features of the multi-camera perspective image to obtain the lane line detection result of the vehicle's environment, so as to determine whether the vehicle can change lanes based on the lane line detection result and the vehicle driving conditions of the lanes near the vehicle.

[0091] In the case where the multi-camera perspective image is a multi-camera perspective environment image, the terminal can perform target detection on the multi-camera perspective environment image based on the bird's-eye view features of the multi-camera perspective image to obtain detection results of target objects in the environment. Among them, the multi-camera perspective environment image can be a surround view image collected by a traffic monitoring device in an urban traffic scene, a surround view image collected by a monitoring device in a supermarket scene, or a surround view image collected by a camera of a drone. If the multi-camera perspective environment image is a surround view image collected by a camera of a drone, after obtaining the detection results of the target objects in the environment, the detection results can be used to control the drone to perform obstacle avoidance flight, such as controlling the flight of the drone during landing in a complex environment.

[0092] In addition, after obtaining the bird's-eye view features of the multi-camera perspective images, the terminal can also perform online mapping or occupancy grid prediction based on the bird's-eye view features of the multi-camera perspective images.

[0093] In order to more clearly understand the technical solution of this application, an explanation is given here in combination with the intelligent driving scenario, as follows:

[0094] (1) Collect driving images from multiple camera perspectives during vehicle driving;

[0095] (2) Input the multi-camera driving images into the feature extraction network to extract image features and obtain multi-camera driving image features;

[0096] (3) Input the multi-camera driving image features into the depth estimation network for depth estimation, and obtain the depth estimation value (i.e., depth feature) of each pixel of the multi-camera driving image features;

[0097] (4) Perform the outer product of the depth estimation value and the multi-camera driving image features to obtain the view frustum point cloud features, and then flatten the view frustum point cloud features in both width and height dimensions;

[0098] (5) Input the flattened view cone point cloud features into the fully connected layer transformation model to obtain the preliminary BEV spatial features at all depths;

[0099] (6) Input the preliminary BEV spatial features at all depths into the fully connected layer deep aggregation model to aggregate the depth dimension and obtain the deep aggregated BEV features;

[0100] (7) The deeply aggregated BEV features are dimensionally rearranged, and then the rearranged BEV features are input into the fully connected layer camera aggregation model to obtain the multi-view aggregated BEV spatial features;

[0101] (8) Input the multi-view aggregated BEV spatial features into the upsampling network for upsampling to obtain the final BEV features;

[0102] (9) Lane line detection is performed on the multi-camera perspective driving image based on the bird's-eye view features of the multi-camera perspective image to obtain the lane line detection result of the vehicle's environment. It is then possible to determine whether the vehicle can change lanes based on the lane line detection result and the driving conditions of vehicles in lanes near the vehicle.

[0103] In the above embodiment, image features and depth features are first extracted from the surround multi-camera view images; the image features and depth features are subjected to outer product operations to obtain the view cone point cloud features; then, the view cone point cloud features are dimensionally transformed to obtain preliminary bird's-eye view features at each depth; each preliminary bird's-eye view feature is deeply aggregated to obtain a deeply aggregated bird's-eye view feature; the deeply aggregated bird's-eye view features are multi-view aggregated to obtain a multi-view aggregated bird's-eye view feature; the multi-view aggregated bird's-eye view features are upsampled to obtain a bird's-eye view feature of the multi-camera view image, and the above-mentioned transformation processing, deep aggregation, multi-view aggregation and upsampling use linear operators (such as fully connected layers and convolutions and other linear transformation operators), which are also supported by the neural network processing unit and have fast execution speed, so that efficient BEV view transformation can be achieved on a general neural network processing unit, thereby improving the conversion efficiency of the bird's-eye view; moreover, depth and view decoupling calculations are used to reduce the computational complexity, while also ensuring the expressiveness of the bird's-eye view features, which is conducive to improving the accuracy of downstream tasks.

[0104] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0105] Based on the same inventive concept, the embodiment of the present application also provides a bird's-eye view perspective conversion device for implementing the above-mentioned bird's-eye view perspective conversion method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more bird's-eye view perspective conversion devices provided below can refer to the limitations of the bird's-eye view perspective conversion method above, and will not be repeated here.

[0106] In one embodiment, Figure 4 As shown, a bird's-eye view perspective conversion device is provided, comprising: an extraction module 402, a calculation module 404, a transformation module 406, a depth aggregation module 408, a perspective aggregation module 410 and an upsampling module 412, wherein:

[0107] An extraction module 402 is used to extract image features and depth features from the surround multi-camera view images;

[0108] A calculation module 404 is used to perform an outer product operation on the image feature and the depth feature to obtain a view frustum point cloud feature;

[0109] A transformation module 406 is used to perform dimensional transformation on the view cone point cloud features to obtain preliminary bird's-eye view features at each depth;

[0110] A depth aggregation module 408 is used to perform depth aggregation on each preliminary bird's-eye view feature to obtain a depth-aggregated bird's-eye view feature;

[0111] A perspective aggregation module 410 is used to perform multi-perspective aggregation on the bird's-eye view features of the depth aggregation to obtain multi-perspective aggregated bird's-eye view features;

[0112] The up-sampling module 412 is used to up-sample the bird's-eye view features aggregated from multiple perspectives to obtain bird's-eye view features of multi-camera perspective images.

[0113] In one embodiment, the extraction module 402 is further configured to extract image features from the surround multi-camera view images through a feature extraction network; and extract depth features from the image features of the multi-camera view images through a depth estimation network.

[0114] In one embodiment, if Figure 5 As shown, the device also includes:

[0115] A flattening module 414 is used to flatten the view cone point cloud features in the horizontal direction and the vertical direction to obtain the flattened view cone point cloud features;

[0116] The transformation module 406 is further used to input the flattened view cone point cloud features into the transformation model, so that the transformation model performs dimensional transformation on the flattened view cone point cloud features to obtain preliminary bird's-eye view features at each depth.

[0117] In one embodiment, the deep aggregation module 408 is further configured to input each preliminary bird's-eye view feature into a deep aggregation model, so that the deep aggregation model performs deep aggregation on each preliminary bird's-eye view feature to obtain deeply aggregated bird's-eye view features.

[0118] In one embodiment, if Figure 5 As shown, the device also includes:

[0119] A rearrangement module 416 is used to rearrange the dimensions of the deeply aggregated bird's-eye view features to obtain rearranged deeply aggregated bird's-eye view features;

[0120] The perspective aggregation module 410 is further used to input the rearranged depth-aggregated bird's-eye view features into the camera aggregation model, so that the camera aggregation model performs multi-perspective aggregation on the rearranged depth-aggregated bird's-eye view features to obtain multi-perspective aggregated bird's-eye view features.

[0121] In one embodiment, the multi-camera perspective image includes a multi-camera perspective driving image or a multi-camera perspective environment image; Figure 5 As shown, the device also includes:

[0122] The detection module 418 is used to perform lane line detection on the multi-camera perspective driving images based on the bird's-eye view features of the multi-camera perspective images to obtain the lane line detection results of the vehicle's environment; and perform target detection on the multi-camera perspective environment images based on the bird's-eye view features of the multi-camera perspective images to obtain the detection results of the target objects in the environment.

[0123] In the above embodiment, image features and depth features are first extracted from the surround multi-camera perspective images; the image features and depth features are subjected to outer product operations to obtain the view cone point cloud features; then, the view cone point cloud features are dimensionally transformed to obtain preliminary bird's-eye view features at each depth; each preliminary bird's-eye view feature is depth aggregated to obtain a depth-aggregated bird's-eye view feature; the depth-aggregated bird's-eye view features are multi-perspective aggregated to obtain a multi-perspective aggregated bird's-eye view feature; the multi-perspective aggregated bird's-eye view features are upsampled to obtain a bird's-eye view feature of a multi-camera perspective image, and the above-mentioned transformation processing, depth aggregation, multi-perspective aggregation and upsampling use linear operators, which are also supported by the neural network processing unit and have fast execution speed, thereby improving the conversion efficiency of the bird's-eye view; moreover, depth and perspective decoupling calculations are used to reduce the computational complexity while ensuring the expressiveness of the bird's-eye view features, which is conducive to improving the accuracy of downstream tasks.

[0124] Each module in the above-mentioned bird's-eye view perspective conversion device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0125] In one embodiment, a computer device is provided, which may be a server, and includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data and driving data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for converting a bird's-eye view perspective is implemented.

[0126] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for converting a bird's-eye view perspective is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0127] It should be pointed out that the above-mentioned processor may refer to a neural network processing unit (NPU).

[0128] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned method for converting the perspective of a bird's-eye view when executing the computer program.

[0130] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for converting the perspective of the bird's-eye view are implemented.

[0131] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the above-mentioned method for converting the perspective of a bird's-eye view when executed by a processor.

[0132] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0134] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for converting a bird's-eye view perspective, characterized in that: The method comprises: Extract image features and depth features from surround multi-camera view images; Performing an outer product operation on the image feature and the depth feature to obtain a view frustum point cloud feature; Flattening the view cone point cloud features to obtain flattened view cone point cloud features, and inputting the flattened view cone point cloud features into a transformation model, so that the transformation model processes the flattened view cone point cloud features through a fully connected layer inside the transformation model to obtain preliminary bird's-eye view features at each depth; the transformation model is a fully connected layer transformation model; Inputting the preliminary bird's-eye view features at each depth into a deep aggregation model, so that the deep aggregation model processes each preliminary bird's-eye view feature through a fully connected layer to obtain a deeply aggregated bird's-eye view feature; the deep aggregation model is a fully connected layer deep aggregation model; Rearranging the dimensions of the deeply aggregated bird's-eye view features to obtain rearranged deeply aggregated bird's-eye view features; the dimensional rearrangement is to rearrange corresponding dimensional features in the deeply aggregated bird's-eye view features to other positions; Inputting the rearranged depth-aggregated bird's-eye view features into a camera aggregation model, so that the camera aggregation model performs multi-view aggregation on the rearranged depth-aggregated bird's-eye view features to obtain multi-view aggregated bird's-eye view features; The multi-view aggregated bird's-eye view features are upsampled to obtain the bird's-eye view features of the multi-camera view image.

2. The method according to claim 1, characterized in that The extracting of image features and depth features from the surround multi-camera view images includes: Image features are extracted from surround multi-camera view images through a feature extraction network; Depth features are extracted from image features of the multi-camera view images through a depth estimation network.

3. The method according to claim 1, characterized in that The flattening of the view cone point cloud features to obtain the flattened view cone point cloud features comprises: The view cone point cloud feature is flattened in the horizontal direction and the vertical direction to obtain a flattened view cone point cloud feature.

4. The method according to any one of claims 1 to 3, characterized in that: The multi-camera perspective image includes a multi-camera perspective driving image or a multi-camera perspective environment image; After upsampling the bird's-eye view features of the multi-view aggregation to obtain the bird's-eye view features of the multi-camera view images, the method further includes: Performing lane line detection on the multi-camera perspective driving image based on the bird's-eye view features of the multi-camera perspective image to obtain a lane line detection result of the vehicle environment; or, Target detection is performed on the multi-camera perspective environment image based on the bird's-eye view features of the multi-camera perspective image to obtain a detection result of the target object in the environment.

5. The method according to claim 4, characterized in that The method further comprises: If the multi-camera perspective environment image is a surround view image captured by a camera of a drone, after obtaining the detection result of the target object in the environment, the detection result is used to control the drone to perform obstacle avoidance flight.

6. A device for converting a bird's-eye view, characterized in that: The device comprises: An extraction module, used to extract image features and depth features from surround multi-camera view images; A calculation module, used for performing an outer product operation on the image feature and the depth feature to obtain a view frustum point cloud feature; A transformation module, used for flattening the view cone point cloud features to obtain flattened view cone point cloud features, and inputting the flattened view cone point cloud features into a transformation model, so that the transformation model processes the flattened view cone point cloud features through a fully connected layer inside the transformation model to obtain preliminary bird's-eye view features at each depth; the transformation model is a fully connected layer transformation model; A deep aggregation module, used for inputting the preliminary bird's-eye view features at each depth into a deep aggregation model, so that the deep aggregation model processes each preliminary bird's-eye view feature through a fully connected layer to obtain a deeply aggregated bird's-eye view feature; the deep aggregation model is a fully connected layer deep aggregation model; A rearrangement module, used for dimensional rearrangement of the deeply aggregated bird's-eye view features to obtain rearranged deeply aggregated bird's-eye view features; the dimensional rearrangement is to rearrange corresponding dimensional features in the deeply aggregated bird's-eye view features to other positions; A perspective aggregation module, used for inputting the rearranged depth-aggregated bird's-eye view features into a camera aggregation model, so that the camera aggregation model performs multi-perspective aggregation on the rearranged depth-aggregated bird's-eye view features to obtain multi-perspective aggregated bird's-eye view features; The upsampling module is used to upsample the bird's-eye view features of the multi-view aggregation to obtain the bird's-eye view features of the multi-camera view image.

7. The device according to claim 6, characterized in that The extraction module is further used to extract image features from the surround multi-camera view images through a feature extraction network; and to extract depth features from the image features of the multi-camera view images through a depth estimation network.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images