Bird's-eye view perspective conversion method, device, equipment, storage medium and program product

Abstract

The present application relates to a method, device, equipment, storage medium and program product for converting a bird's-eye view. The method comprises: extracting image features and depth features from a multi-camera perspective image of a surround view; extracting key information in the horizontal direction and the vertical direction for the image features and the depth features of the multi-camera perspective image, respectively, to obtain key image features and key depth features in the horizontal direction and key image features and key depth features in the vertical direction; performing matrix operations on the key image features and key depth features in the horizontal direction to obtain a first bird's-eye view feature; performing matrix operations on the key image features and key depth features in the vertical direction to obtain a second bird's-eye view feature; performing feature fusion on the first bird's-eye view feature and the second bird's-eye view feature to obtain a bird's-eye view feature of the multi-camera perspective image. The present method can effectively improve the conversion efficiency of the bird's-eye view.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment, storage medium and program product for converting a bird's-eye view. Background Art

[0002] With the continuous development of technologies such as intelligent driving and target detection, the algorithm research on bird's eye view (BEV) perspective conversion has become more and more extensive. BEV perspective conversion mainly projects the image features of the camera perspective to the BEV perspective, thereby applying it to downstream tasks.

[0003] The traditional lift-splat perspective conversion method mainly involves: inputting the surround multi-camera images into the backbone network for image feature extraction, and the depth estimation network uses the extracted multi-camera image features to perform depth estimation to obtain the depth estimation value of each pixel; then, the depth estimation value and the image feature are outer-producted to obtain the features of the view frustum point cloud, and the view frustum point cloud is hashed into the BEV grid according to the geometric position relationship. If the number of view frustum point clouds in a BEV grid is greater than 1, the features of all view frustum point clouds are added and summed to obtain the BEV features.

[0004] However, the lift-splat perspective conversion method requires the outer product of multi-camera image features first, which requires huge computational complexity. In addition, during the perspective conversion process, the neural network processing unit (NPU) is required to support operators such as cumsum, gather / gatherND, and scather / scatherND. Most NPUs cannot fully support these operators, and even if they support these operators, the execution speed is very slow, which greatly affects the BEV perspective conversion efficiency. Summary of the invention

[0005] Based on this, it is necessary to provide a bird's-eye view perspective conversion method, device, equipment, storage medium and program product to address the above-mentioned technical problems, which can effectively improve the conversion efficiency of the bird's-eye view perspective.

[0006] In a first aspect, the present application provides a method for converting a bird's-eye view perspective, the method comprising:

[0007] Extract image features and depth features from surround multi-camera view images;

[0008] Extract key information in the horizontal direction and the vertical direction for the image features and the depth features of the multi-camera perspective images, respectively, to obtain key image features and key depth features in the horizontal direction and key image features and key depth features in the vertical direction;

[0009] Performing matrix operations on the key image features and key depth features in the horizontal direction to obtain first bird's-eye view features;

[0010] Performing matrix operations on the key image features and key depth features in the vertical direction to obtain second bird's-eye view features;

[0011] The first bird's-eye view feature and the second bird's-eye view feature are fused to obtain the bird's-eye view feature of the multi-camera perspective image.

[0012] In one embodiment, extracting image features and depth features from the surround multi-camera view images includes:

[0013] Image features are extracted from surround multi-camera view images through a feature extraction network;

[0014] Depth features are extracted from image features of the multi-camera view images through a depth estimation network.

[0015] In one embodiment, performing matrix operation on the key image features and key depth features in the horizontal direction to obtain the first bird's-eye view features includes:

[0016] Performing matrix multiplication on the key image features and key depth features in the horizontal direction to obtain horizontal dimension-increased features;

[0017] Performing matrix multiplication on the horizontal dimension-raising feature and the horizontal ring matrix to obtain a horizontal intermediate feature;

[0018] The elements in the horizontal intermediate feature are multiplied element by element with the elements at corresponding positions in the horizontal radial matrix to obtain a first bird's-eye view feature.

[0019] In one embodiment, performing matrix operation on the key image features and key depth features in the vertical direction to obtain the second bird's-eye view features includes:

[0020] Performing matrix multiplication on the key image features and key depth features in the vertical direction to obtain vertical dimension-increased features;

[0021] Performing matrix multiplication on the vertical dimension-raising feature and the vertical ring matrix to obtain a vertical intermediate feature;

[0022] Performing an arithmetic operation on the elements in the vertical intermediate feature and the elements at corresponding positions in the vertical radial matrix to obtain a second bird's-eye view feature.

[0023] In one embodiment, the performing feature fusion on the first bird's-eye view feature and the second bird's-eye view feature to obtain the bird's-eye view feature of the multi-camera perspective image includes:

[0024] Perform an arithmetic operation on the elements in the first bird's-eye view feature and the elements in corresponding positions in the second bird's-eye view feature to obtain the bird's-eye view feature of the multi-camera perspective image.

[0025] In one of the embodiments, the multi-camera perspective image includes a multi-camera perspective driving image or a multi-camera perspective environment image;

[0026] After the first bird's-eye view feature and the second bird's-eye view feature are fused to obtain the bird's-eye view feature of the multi-camera perspective image, the method further includes:

[0027] Perform lane line detection on the multi-camera perspective driving image based on the bird's-eye view features of the multi-camera perspective image to obtain a lane line detection result of the vehicle's environment; or

[0028] Target detection is performed on the multi-camera perspective environment image based on the bird's-eye view features of the multi-camera perspective image to obtain a detection result of the target object in the environment.

[0029] In one embodiment, the image features and depth features of the multi-camera perspective images are subjected to key information extraction in the horizontal direction and the vertical direction, respectively, to obtain the key image features and key depth features in the horizontal direction and the key image features and key depth features in the vertical direction, including:

[0030] Compressing the image features of the multi-camera perspective images in a horizontal direction, and selecting feature values ​​that meet preset conditions in the horizontal direction during the compression process to obtain key image features in the horizontal direction; and compressing the image features of the multi-camera perspective images in a vertical direction, and selecting feature values ​​that meet preset conditions in the vertical direction during the compression process to obtain key image features in the vertical direction;

[0031] Predicting the compression coefficient of each element of the image features of the multi-camera perspective image in the horizontal direction and the compression coefficient of each element in the vertical direction; based on the compression coefficient of each element, performing weighted processing on the depth features of the multi-camera perspective image in the horizontal direction and the vertical direction to obtain the key depth features in the horizontal direction and the key depth features in the vertical direction.

[0032] In a second aspect, the present application further provides a bird's-eye view perspective conversion device, the device comprising:

[0033] A first extraction module is used to extract image features and depth features from the surround multi-camera view images;

[0034] A second extraction module is used to extract key information in the horizontal direction and the vertical direction from the image features and the depth features of the multi-camera perspective images, respectively, to obtain the key image features and the key depth features in the horizontal direction and the key image features and the key depth features in the vertical direction;

[0035] A first operation module, configured to perform a matrix operation on the key image features and the key depth features in the horizontal direction to obtain a first bird's-eye view feature;

[0036] A second operation module, used for performing matrix operation on the key image features and key depth features in the vertical direction to obtain a second bird's-eye view feature;

[0037] A fusion module is used to perform feature fusion on the first bird's-eye view feature and the second bird's-eye view feature to obtain the bird's-eye view feature of the multi-camera perspective image.

[0038] In one of the embodiments, the first extraction module is further used to extract image features from the surround multi-camera view images through a feature extraction network; and to extract depth features from the image features of the multi-camera view images through a depth estimation network.

[0039] In one embodiment, the first operation module is also used to perform matrix multiplication on the key image features and key depth features in the horizontal direction to obtain horizontal dimension-upgraded features; perform matrix multiplication on the horizontal dimension-upgraded features and the horizontal annular matrix to obtain horizontal intermediate features; and perform element-by-element multiplication on the elements in the horizontal intermediate features and the elements in the corresponding positions in the horizontal radial matrix to obtain the first bird's-eye view features.

[0040] In one embodiment, the second operation module is further used to perform matrix multiplication on the key image features and key depth features in the vertical direction to obtain vertical dimension-upgraded features; perform matrix multiplication on the vertical dimension-upgraded features and the vertical circumferential matrix to obtain vertical intermediate features; perform arithmetic operations on the elements in the vertical intermediate features and the elements at corresponding positions in the vertical radial matrix to obtain a second bird's-eye view feature.

[0041] In one of the embodiments, the fusion module is further used to perform arithmetic operations on elements in the first bird's-eye view feature and elements at corresponding positions in the second bird's-eye view feature to obtain the bird's-eye view features of the multi-camera perspective image.

[0042] In one embodiment, the multi-camera perspective image includes a multi-camera perspective driving image or a multi-camera perspective environment image; the device further includes:

[0043] A detection module is used to perform lane line detection on the multi-camera perspective driving image based on the bird's-eye view features of the multi-camera perspective image to obtain lane line detection results of the vehicle's environment; and perform target detection on the multi-camera perspective environment image based on the bird's-eye view features of the multi-camera perspective image to obtain detection results of target objects in the environment.

[0044] In one of the embodiments, the second extraction module is further used to compress the image features of the multi-camera perspective image in the horizontal direction, and select the feature values ​​that meet the preset conditions in the horizontal direction during the compression process to obtain the key image features in the horizontal direction; and compress the image features of the multi-camera perspective image in the vertical direction, and select the feature values ​​that meet the preset conditions in the vertical direction during the compression process to obtain the key image features in the vertical direction; predict the compression coefficient of each element of the image features of the multi-camera perspective image in the horizontal direction and the compression coefficient of each element in the vertical direction; based on the compression coefficient of each element, perform weighted processing on the depth features of the multi-camera perspective image in the horizontal direction and the vertical direction to obtain the key depth features in the horizontal direction and the key depth features in the vertical direction.

[0045] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the bird's-eye view perspective conversion method when executing the computer program.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the bird's-eye view perspective conversion method are implemented.

[0047] In a fifth aspect, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the bird's-eye view perspective conversion method are implemented.

[0048] The above-mentioned bird's-eye view perspective conversion method, device, computer equipment, storage medium and computer program product extract image features and depth features from the multi-camera perspective images of the surround view, and extract key information in the horizontal direction and the vertical direction for the image features and depth features of the multi-camera perspective images, respectively, so as to obtain key image features and key depth features in two different directions; in addition, matrix operations are performed on the key image features and key depth features in the horizontal direction to obtain the first bird's-eye view features; and matrix operations are performed on the key image features and key depth features in the vertical direction to obtain the second bird's-eye view features. The above-mentioned operations are pure matrix operations, do not require special operator support, and have fast operation speed; finally, considering that the first bird's-eye view features and the second bird's-eye view features are obtained based on the key image features and key depth features in two different directions, the first bird's-eye view features and the second bird's-eye view features are feature fused to obtain the bird's-eye view features of the multi-camera perspective images, so that the bird's-eye view features simultaneously contain information in two different directions, thereby effectively improving the conversion efficiency of the bird's-eye view perspective, which is conducive to improving the task effect of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A diagram of an application environment of a bird's-eye view perspective conversion method in one embodiment;

[0050] Figure 2 A schematic diagram of a process of converting a bird's-eye view perspective in one embodiment;

[0051] Figure 3 A schematic diagram of a flow chart of a method for converting a bird's-eye view perspective in another embodiment;

[0052] Figure 4 is a structural block diagram of a bird's-eye view perspective conversion device in one embodiment;

[0053] Figure 5 is a structural block diagram of a bird's-eye view perspective conversion device in another embodiment;

[0054] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] It should be noted that in the following description, the terms "first and second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first and second" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0057] The bird's-eye view perspective conversion method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0058] It should be noted that the data storage system can store data that needs to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0059] The terminal 102 may be an onboard device in a motor vehicle (including a fuel vehicle and a new energy vehicle), a smart phone, a tablet computer, a laptop computer, a desktop computer, an Internet of Things device, and a portable wearable device.

[0060] Server 104 can be an independent physical server or a service node in a blockchain system. Each service node in the blockchain system forms a peer-to-peer network. The peer-to-peer protocol is an application layer protocol running on the Transmission Control Protocol (TCP). In addition, server 104 can also be a server cluster composed of multiple physical servers, which can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0061] In one embodiment, Figure 2 As shown, a method for converting a bird's-eye view perspective is provided. The method can be Figure 1 The method is performed by a server or a terminal in the embodiment, or by a server and a terminal in collaboration. Figure 1 The following steps are used as an example to illustrate the terminal execution in:

[0062] S202, extracting image features and depth features from the surround multi-camera view images.

[0063] Among them, the multi-camera perspective image can refer to the image captured by multiple cameras with different perspectives, which can be called the surround multi-camera image. The above multiple cameras can be used to capture images of all directions under the position of the camera, such as collecting images in the four directions of the front, back, left, and right of the position, or even more directions to ensure that the environment around the position is captured; or, the multi-camera perspective image can be an image captured by a camera with different perspectives during the rotation along the target axis. In some actual application scenarios, the multi-camera perspective image can include at least one of a multi-camera perspective driving image or a multi-camera perspective environment image.

[0064] The depth feature may include a depth estimation value of each pixel in the image feature of the multi-camera view image.

[0065] In one embodiment, the terminal may extract image features from the surround multi-camera view images through a feature extraction network; and extract depth features from the image features of the multi-camera view images through a depth estimation network.

[0066] The feature extraction network may include a backbone network for extracting image features from the surround multi-camera view images. The depth estimation network may be a DeeperDepth Prediction with Fully Convolutional Residual Networks (FCRN) or a Deep Ordinal Regression Network for Monocular DepthEstimation.

[0067] Before extracting image features, the terminal may also perform image enhancement processing on the multi-camera view images, such as denoising, increasing brightness and contrast, etc.; then, the enhanced multi-camera view images are input into a feature extraction network, which can extract image features from the enhanced multi-camera view images, and then input the image features into a depth estimation network for depth estimation to obtain depth features.

[0068] For example, Figure 3 As shown, the terminal multi-camera view images are input into the feature extraction network. The backbone network of the feature extraction network can extract image features from the multi-camera view images to obtain a scale of Then, the image feature is input into the depth estimation network, and the depth estimation network uses the image feature to perform depth estimation, and obtains a scale of It should be noted that the above , , and Represents a dimension respectively, where It can be the number of cameras or the number of camera angles. and are the height and width of the image features, respectively, C represents the number of feature channels, Indicates the depth quantity (that is, the number of depth values).

[0069] S204, extracting key information in the horizontal direction and the vertical direction for the image features and the depth features of the multi-camera view images, respectively, to obtain key image features and key depth features in the horizontal direction and key image features and key depth features in the vertical direction.

[0070] Among them, key information extraction can refer to compressing information in the horizontal direction and the vertical direction respectively, removing other redundant information to obtain corresponding key information (ie main information).

[0071] In one embodiment, for extracting key image features, the terminal may compress the image features of the multi-camera perspective images in the horizontal direction, and select the feature values ​​that meet the preset conditions in the horizontal direction during the compression process to obtain the key image features in the horizontal direction; and compress the image features of the multi-camera perspective images in the vertical direction, and select the feature values ​​that meet the preset conditions in the vertical direction during the compression process to obtain the key image features in the vertical direction. For example, the largest feature value in the horizontal direction may be selected during the compression process to obtain the key image features in the horizontal direction; and the largest feature value in the vertical direction may be selected during the compression process to obtain the key image features in the vertical direction.

[0072] In one embodiment, for the extraction of key depth features, the terminal can predict the compression coefficient of each element of the image features of the multi-camera perspective images in the horizontal direction, as well as the compression coefficient of each element in the vertical direction; based on the compression coefficient of each element, the depth features of the multi-camera perspective images are weighted in the horizontal and vertical directions to obtain the key depth features in the horizontal direction and the key depth features in the vertical direction.

[0073] For example, Figure 3 As shown in the figure, the terminal extracts key information from the image features of the multi-camera view images in the width direction (i.e., the horizontal direction) and obtains a scale of In addition, the key information of the depth features of the multi-camera view images is extracted in the width direction, and the scale is In addition, the key information of the image features of the multi-camera perspective images is extracted in the height direction (i.e., vertical direction), and the scale is In addition, the key information of the depth features of the multi-camera perspective images is extracted in the height direction, and the scale is The key depth features.

[0074] S206, performing matrix operations on key image features and key depth features in the horizontal direction to obtain first bird's-eye view features.

[0075] In one embodiment, the terminal can perform matrix multiplication on the key image features and key depth features in the horizontal direction to obtain horizontal dimension-added features; perform matrix multiplication on the horizontal dimension-added features and the horizontal annular matrix to obtain horizontal intermediate features; and perform element-by-element multiplication on the elements in the horizontal intermediate features and the elements in the corresponding positions in the horizontal radial matrix to obtain the first bird's-eye view features.

[0076] The horizontal ring matrix may refer to the horizontal Ring matrix (referred to as the horizontal Ring matrix), and the horizontal radial matrix may refer to the horizontal Ray matrix (referred to as the horizontal Ray matrix).

[0077] For example, Figure 3 As shown, after obtaining the key image features and key depth features in the horizontal direction, the terminal performs matrix multiplication on the key image features and key depth features in the horizontal direction to obtain a scale of The horizontal dimension-raising feature of The horizontal Ring matrix is ​​multiplied to get the scale The horizontal intermediate feature of Multiply the horizontal Ray matrix element by element to obtain BEV feature 1.

[0078] The calculation process of the Ring matrix and Ray matrix in the horizontal direction may specifically include:

[0079] 1) Initialize the matrix to all zeros and initialize the matrix to all zeros ;

[0080] 2) Expand the coordinates of each point of the image feature in the image coordinate system by one dimension along the depth direction to obtain the view frustum point cloud of the image coordinate system;

[0081] 3) Average the view cone point cloud of the image coordinate system in the height direction to obtain the view cone point cloud after compression;

[0082] 4) With the origin of the ego-vehicle coordinate system as the center, divide the BEV grid in the front, back, left and right directions;

[0083] 5) Traverse each point in the view frustum point cloud after compression one by one, and its coordinates are (n, , d), project each point from the image coordinate system to the ego vehicle coordinate system through the camera parameters. If it falls on a certain BEV grid ( , ,1), then the matrix elements Set to 1, the matrix element Also set to 1 to obtain the horizontal Ring matrix and Ray matrix.

[0084] Among them, BEV grid ( 1) It can represent the height of BEV characteristics, Can represent the width of the BEV feature.

[0085] S208, performing matrix operations on key image features and key depth features in the vertical direction to obtain second bird's-eye view features.

[0086] In one embodiment, the terminal can perform matrix multiplication on the key image features and key depth features in the vertical direction to obtain the vertical dimension-upgraded features; perform matrix multiplication on the vertical dimension-upgraded features and the vertical circumferential matrix to obtain the vertical intermediate features; perform arithmetic operations on the elements in the vertical intermediate features and the elements in the corresponding positions in the vertical radial matrix to obtain the second bird's-eye view features.

[0087] The vertical annular matrix may refer to a vertical Ring matrix, and the vertical radial matrix may refer to a vertical Ray matrix.

[0088] For example, Figure 3 As shown, after obtaining the key image features and key depth features in the vertical direction, the terminal performs matrix multiplication on the key image features and key depth features in the vertical direction to obtain a scale of The vertical dimension-raising feature of The vertical ring matrix is ​​multiplied to obtain the scale The vertical intermediate feature of Multiply the vertical Ray matrix element by element to obtain BEV feature 2.

[0089] For the calculation process of the vertical Ring matrix and Ray matrix, you can refer to the following steps:

[0090] 1) Initialize the matrix to all zeros and initialize the matrix to all zeros ;

[0091] 2) Expand the coordinates of each point of the image feature in the image coordinate system by one dimension along the depth direction to obtain the view frustum point cloud of the image coordinate system;

[0092] 3) Average the view cone point cloud of the image coordinate system in the width direction to obtain the view cone point cloud after compressing the width;

[0093] 4) With the origin of the ego-vehicle coordinate system as the center, divide the BEV grid in the front, back, left and right directions;

[0094] 5) Traverse each point in the compressed view frustum point cloud one by one, and its coordinates are (n, , d), project each point from the image coordinate system to the ego vehicle coordinate system through the camera parameters. If it falls on a certain BEV grid ( , ,1), then the matrix elements Set to 1, the matrix element Also set to 1 to obtain the Ring matrix and Ray matrix in the vertical direction.

[0095] Among them, the coordinates (n, , n in d) represents the camera index (i.e. the sequence number or serial number of the camera), and d represent the width and depth of the frustum point cloud after the width is compressed, respectively.

[0096] S210, performing feature fusion on the first bird's-eye view feature and the second bird's-eye view feature to obtain a bird's-eye view feature of a multi-camera perspective image.

[0097] In one embodiment, the terminal performs an arithmetic operation on the elements in the first bird's-eye view feature and the elements in the corresponding positions in the second bird's-eye view feature to obtain the bird's-eye view features of the multi-camera perspective image. The arithmetic operation may be a sum operation, or a difference or product operation.

[0098] For example, Figure 3 As shown, after obtaining BEV feature 1 and BEV feature 2, BEV feature 1 and BEV feature 2 are added element by element to obtain the final bird's-eye view features of the multi-camera perspective image.

[0099] In one embodiment, when the multi-camera perspective image is a multi-camera perspective driving image, after obtaining the bird's-eye view features of the multi-camera perspective image, the terminal can perform lane line detection on the multi-camera perspective driving image based on the bird's-eye view features of the multi-camera perspective image to obtain the lane line detection result of the vehicle's environment, so as to determine whether the vehicle can change lanes based on the lane line detection result and the vehicle driving conditions of the lanes near the vehicle.

[0100] In the case where the multi-camera perspective image is a multi-camera perspective environment image, the terminal can perform target detection on the multi-camera perspective environment image based on the bird's-eye view features of the multi-camera perspective image to obtain detection results of target objects in the environment. Among them, the multi-camera perspective environment image can be a surround view image collected by a traffic monitoring device in an urban traffic scene, a surround view image collected by a monitoring device in a supermarket scene, or a surround view image collected by a camera of a drone. If the multi-camera perspective environment image is a surround view image collected by a camera of a drone, after obtaining the detection results of the target objects in the environment, the detection results can be used to control the drone to perform obstacle avoidance flight, such as controlling the flight of the drone during landing in a complex environment.

[0101] In addition, after obtaining the bird's-eye view features of the multi-camera perspective images, the terminal can also perform online mapping or occupancy grid prediction based on the bird's-eye view features of the multi-camera perspective images.

[0102] In order to more clearly understand the technical solution of this application, an explanation is given here in combination with the intelligent driving scenario, as follows:

[0103] (1) Collect driving images from multiple camera perspectives during vehicle driving;

[0104] (2) Input the multi-camera driving images into the feature extraction network to extract image features and obtain multi-camera driving image features;

[0105] (3) Input the multi-camera driving image features into the depth estimation network for depth estimation, and obtain the depth estimation value (i.e., depth feature) of each pixel of the multi-camera driving image features;

[0106] (4) The driving image features and depth estimation values ​​from multiple camera perspectives are used to extract key information in the width direction to obtain key image features and key depth estimation values ​​in the horizontal direction.

[0107] (5) Perform matrix multiplication on the key image features in the horizontal direction and the key depth estimation value to obtain the horizontal dimension-increased features.

[0108] (6) Perform matrix multiplication on the horizontal dimension-upgraded features and the horizontal Ring matrix to obtain the horizontal intermediate features.

[0109] (7) Multiply the horizontal intermediate feature and the horizontal Ray matrix element by element to obtain BEV feature 1.

[0110] (8) Repeat steps (3) to (6) in the height direction to obtain BEV feature 2.

[0111] (9) Add BEV feature 1 and BEV feature 2 element by element to obtain the final BEV feature.

[0112] (10) Lane line detection is performed on the multi-camera perspective driving image based on the bird's-eye view features of the multi-camera perspective image to obtain the lane line detection result of the vehicle's environment, so as to determine whether the vehicle can change lanes based on the lane line detection result and the vehicle driving conditions in the lanes near the vehicle.

[0113] In the above embodiment, image features and depth features are extracted from the surround multi-camera perspective images, and key information in the horizontal and vertical directions are extracted for the image features and depth features of the multi-camera perspective images, respectively, so as to obtain key image features and key depth features in two different directions; in addition, matrix operations are performed on the key image features and key depth features in the horizontal direction to obtain the first bird's-eye view features; and matrix operations are performed on the key image features and key depth features in the vertical direction to obtain the second bird's-eye view features. The above operations are pure matrix operations, do not require special operator support, and have fast operation speed; finally, considering that the first bird's-eye view features and the second bird's-eye view features are obtained based on the key image features and key depth features in two different directions, the first bird's-eye view features and the second bird's-eye view features are feature fused to obtain the bird's-eye view features of the multi-camera perspective images, so that the bird's-eye view features simultaneously contain information in two different directions, thereby effectively improving the conversion efficiency of the bird's-eye view, which is conducive to improving the task effects of downstream tasks.

[0114] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0115] Based on the same inventive concept, the embodiment of the present application also provides a bird's-eye view perspective conversion device for implementing the above-mentioned bird's-eye view perspective conversion method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more bird's-eye view perspective conversion device embodiments provided below can refer to the above-mentioned limitations on the bird's-eye view perspective conversion method, and will not be repeated here.

[0116] In one embodiment, Figure 4As shown, a bird's-eye view perspective conversion device is provided, comprising: a first extraction module 402, a second extraction module 404, a first operation module 406, a second operation module 408 and a fusion module 410, wherein:

[0117] A first extraction module 402 is used to extract image features and depth features from the surround multi-camera view images;

[0118] The second extraction module 404 is used to extract key information in the horizontal direction and the vertical direction from the image features and the depth features of the multi-camera perspective images, respectively, to obtain key image features and key depth features in the horizontal direction and key image features and key depth features in the vertical direction;

[0119] A first operation module 406 is used to perform a matrix operation on the key image features and the key depth features in the horizontal direction to obtain a first bird's-eye view feature;

[0120] A second operation module 408 is used to perform matrix operation on key image features and key depth features in the vertical direction to obtain second bird's-eye view features;

[0121] The fusion module 410 is used to perform feature fusion on the first bird's-eye view feature and the second bird's-eye view feature to obtain the bird's-eye view feature of the multi-camera perspective image.

[0122] In one embodiment, the first extraction module 402 is further configured to extract image features from the surround multi-camera view images through a feature extraction network; and extract depth features from the image features of the multi-camera view images through a depth estimation network.

[0123] In one embodiment, the first operation module 406 is also used to perform matrix multiplication on the key image features and key depth features in the horizontal direction to obtain horizontal dimension-upgraded features; perform matrix multiplication on the horizontal dimension-upgraded features and the horizontal annular matrix to obtain horizontal intermediate features; and perform element-by-element multiplication on the elements in the horizontal intermediate features and the elements in the corresponding positions in the horizontal radial matrix to obtain the first bird's-eye view features.

[0124] In one embodiment, the second operation module 408 is also used to perform matrix multiplication on the key image features and key depth features in the vertical direction to obtain the vertical dimension-upgraded features; perform matrix multiplication on the vertical dimension-upgraded features and the vertical circumferential matrix to obtain the vertical intermediate features; perform arithmetic operations on the elements in the vertical intermediate features and the elements in the corresponding positions in the vertical radial matrix to obtain the second bird's-eye view features.

[0125] In one embodiment, the fusion module 410 is further configured to perform arithmetic operations on elements in the first bird's-eye view feature and elements at corresponding positions in the second bird's-eye view feature to obtain bird's-eye view features of the multi-camera perspective image.

[0126] In one embodiment, the multi-camera perspective image includes a multi-camera perspective driving image or a multi-camera perspective environment image; Figure 5 As shown, the device also includes:

[0127] The detection module 412 is used to perform lane line detection on the multi-camera perspective driving images based on the bird's-eye view features of the multi-camera perspective images to obtain the lane line detection results of the vehicle's environment; perform target detection on the multi-camera perspective environment images based on the bird's-eye view features of the multi-camera perspective images to obtain the detection results of the target objects in the environment.

[0128] In one embodiment, the second extraction module 404 is further used to compress the image features of the multi-camera perspective images in the horizontal direction, and select feature values ​​that meet preset conditions in the horizontal direction during the compression process to obtain key image features in the horizontal direction; and to compress the image features of the multi-camera perspective images in the vertical direction, and select feature values ​​that meet preset conditions in the vertical direction during the compression process to obtain key image features in the vertical direction; predict the compression coefficient of each element of the image features of the multi-camera perspective images in the horizontal direction, and the compression coefficient of each element in the vertical direction; based on the compression coefficient of each element, perform weighted processing on the depth features of the multi-camera perspective images in the horizontal direction and the vertical direction to obtain key depth features in the horizontal direction and key depth features in the vertical direction.

[0129] In the above embodiment, image features and depth features are extracted from the surround multi-camera perspective images, and key information in the horizontal and vertical directions are extracted for the image features and depth features of the multi-camera perspective images, respectively, so as to obtain key image features and key depth features in two different directions; in addition, matrix operations are performed on the key image features and key depth features in the horizontal direction to obtain the first bird's-eye view features; and matrix operations are performed on the key image features and key depth features in the vertical direction to obtain the second bird's-eye view features. The above operations are pure matrix operations, do not require special operator support, and have fast operation speed; finally, considering that the first bird's-eye view features and the second bird's-eye view features are obtained based on the key image features and key depth features in two different directions, the first bird's-eye view features and the second bird's-eye view features are feature fused to obtain the bird's-eye view features of the multi-camera perspective images, so that the bird's-eye view features simultaneously contain information in two different directions, thereby effectively improving the conversion efficiency of the bird's-eye view, which is conducive to improving the task effects of downstream tasks.

[0130] Each module in the above-mentioned bird's-eye view perspective conversion device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0131] In one embodiment, a computer device is provided, which may be a server, and includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data and traffic data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for converting a bird's-eye view perspective is implemented.

[0132] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for converting a bird's-eye view perspective is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0133] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0134] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned bird's-eye view perspective conversion method when executing the computer program.

[0135] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned bird's-eye view perspective conversion method are implemented.

[0136] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the above-mentioned bird's-eye view perspective conversion method when executed by a processor.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for converting a bird's-eye view, characterized in that: The method comprises: Extract image features and depth features from surround multi-camera view images; Compressing the image features of the multi-camera perspective images in a horizontal direction, and selecting feature values ​​that meet preset conditions in the horizontal direction during the compression process to obtain key image features in the horizontal direction; and compressing the image features of the multi-camera perspective images in a vertical direction, and selecting feature values ​​that meet preset conditions in the vertical direction during the compression process to obtain key image features in the vertical direction; Predicting the compression coefficient of each element of the image features of the multi-camera perspective images in the horizontal direction and the compression coefficient of each element in the vertical direction; performing weighted processing on the depth features of the multi-camera perspective images in the horizontal direction and the vertical direction based on the compression coefficient of each element to obtain the key depth features in the horizontal direction and the key depth features in the vertical direction; Performing matrix operations on the key image features and key depth features in the horizontal direction to obtain first bird's-eye view features; Performing matrix operations on the key image features and key depth features in the vertical direction to obtain second bird's-eye view features; The first bird's-eye view feature and the second bird's-eye view feature are fused to obtain the bird's-eye view feature of the multi-camera perspective image.

2. The method according to claim 1, characterized in that The extracting of image features and depth features from the surround multi-camera view images includes: Image features are extracted from surround multi-camera view images through a feature extraction network; Depth features are extracted from image features of the multi-camera view images through a depth estimation network.

3. The method according to claim 1, characterized in that The performing matrix operation on the key image features and the key depth features in the horizontal direction to obtain the first bird's-eye view features comprises: Performing matrix multiplication on the key image features and key depth features in the horizontal direction to obtain horizontal dimension-increased features; Performing matrix multiplication on the horizontal dimension-raising feature and the horizontal ring matrix to obtain a horizontal intermediate feature; The elements in the horizontal intermediate feature are multiplied element by element with the elements at corresponding positions in the horizontal radial matrix to obtain a first bird's-eye view feature.

4. The method according to claim 1, characterized in that: The performing matrix operation on the key image features and the key depth features in the vertical direction to obtain the second bird's-eye view features comprises: Performing matrix multiplication on the key image features and key depth features in the vertical direction to obtain vertical dimension-increased features; Performing matrix multiplication on the vertical dimension-raising feature and the vertical ring matrix to obtain a vertical intermediate feature; Performing an arithmetic operation on the elements in the vertical intermediate feature and the elements at corresponding positions in the vertical radial matrix to obtain a second bird's-eye view feature.

5. The method according to claim 1, characterized in that The step of fusing the first bird's-eye view feature and the second bird's-eye view feature to obtain the bird's-eye view feature of the multi-camera perspective image includes: Perform an arithmetic operation on the elements in the first bird's-eye view feature and the elements in corresponding positions in the second bird's-eye view feature to obtain the bird's-eye view feature of the multi-camera perspective image.

6. The method according to any one of claims 1 to 5, characterized in that: The multi-camera perspective image includes a multi-camera perspective driving image or a multi-camera perspective environment image; After the first bird's-eye view feature and the second bird's-eye view feature are fused to obtain the bird's-eye view feature of the multi-camera perspective image, the method further includes: Perform lane line detection on the multi-camera perspective driving image based on the bird's-eye view features of the multi-camera perspective image to obtain a lane line detection result of the vehicle's environment; or Target detection is performed on the multi-camera perspective environment image based on the bird's-eye view features of the multi-camera perspective image to obtain a detection result of the target object in the environment.

7. The method according to claim 6, characterized in that The method further comprises: If the multi-camera perspective environment image is a surround view image captured by a camera of a drone, after obtaining the detection result of the target object in the environment, the detection result is used to control the drone to perform obstacle avoidance flight.

8. A bird's-eye view perspective conversion device, characterized in that: The device comprises: A first extraction module is used to extract image features and depth features from the surround multi-camera view images; A second extraction module is used to compress the image features of the multi-camera perspective image in the horizontal direction, and select the feature values ​​that meet the preset conditions in the horizontal direction during the compression process to obtain the key image features in the horizontal direction; and to compress the image features of the multi-camera perspective image in the vertical direction, and select the feature values ​​that meet the preset conditions in the vertical direction during the compression process to obtain the key image features in the vertical direction; predict the compression coefficient of each element of the image features of the multi-camera perspective image in the horizontal direction and the compression coefficient of each element in the vertical direction; based on the compression coefficient of each element, perform weighted processing on the depth features of the multi-camera perspective image in the horizontal direction and the vertical direction to obtain the key depth features in the horizontal direction and the key depth features in the vertical direction; A first operation module, configured to perform a matrix operation on the key image features and the key depth features in the horizontal direction to obtain a first bird's-eye view feature; A second operation module, used for performing matrix operation on the key image features and key depth features in the vertical direction to obtain a second bird's-eye view feature; A fusion module is used to perform feature fusion on the first bird's-eye view feature and the second bird's-eye view feature to obtain the bird's-eye view feature of the multi-camera perspective image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images