An automatic driving BEV task learning method and related device

Abstract

The application provides an automatic driving BEV task learning method and related device, which mainly represents geometric depth information features through obtained bird's eye view (BEV) features. Since the BEV features are obtained by fusing, normalizing and aligning point cloud data estimated by time series stereo matching and depth estimation, the aligned point cloud data is obtained, and the BEV features are obtained by performing outer product operation and voxel pooling operation on the context features extracted from the image features of the multi-view image and the aligned point cloud data. Therefore, the BEV features can represent the perspective or coordinate system of the perceived world, so the BEV features can also represent the geometric depth information features. Thus, by using the BEV features to learn the machine vision task, the safety hazards of the automatic driving car during driving can be effectively avoided.

Description

Technical Field

[0001] This application relates to the technical fields of autonomous driving, simulation testing, task learning and model training, and more specifically, to an autonomous driving BEV task learning method and related apparatus. Background Technology

[0002] Currently, in autonomous driving applications, features extracted from images captured by single or multiple cameras are typically used for machine vision task learning and inference. For example, multiple cameras capture environmental images, and image features are extracted. Then, a model learned from these machine vision features is used to detect target objects (such as obstacles or pedestrians). However, in practice, while these image features can predict the relative position of a target object within the camera's field of view, it is difficult to accurately predict the distance (i.e., geometric depth information) between the target object and the camera, potentially posing safety hazards to autonomous vehicles. Summary of the Invention

[0003] The purpose of this application is to provide an autonomous driving BEV task learning method and related apparatus to improve the safety hazards of autonomous vehicles during driving.

[0004] This application provides a method for learning autonomous driving BEV tasks, including: acquiring a first multi-view image at the current moment in an autonomous driving scenario, and extracting image features from the first multi-view image, wherein the multi-view image is an image taken from multiple perspectives of an autonomous vehicle in the autonomous driving scenario at the same moment; acquiring image features from a second multi-view image at the previous moment, and performing temporal stereo matching on the image features of the first multi-view image and the image features of the second multi-view image to obtain matched point cloud data, wherein the shooting perspectives of the first multi-view image and the second multi-view image are the same; performing depth estimation on the image features of the first multi-view image to obtain estimated point cloud data, and fusing, normalizing, and aligning the estimated point cloud data and the matched point cloud data to obtain aligned point cloud data; extracting context features from the image features of the first multi-view image based on the camera parameters of the first multi-view image, and performing outer product operation and voxel pooling operation on the context features and the aligned point cloud data to obtain bird's-eye view BEV features, wherein the BEV features are used to learn at least one machine vision task. In the implementation of the above scheme, the obtained bird's-eye view BEV feature is used to represent geometric depth information features. Since the BEV feature is obtained by fusing, normalizing and aligning the point cloud data of temporal stereo matching and the point cloud data of depth estimation to obtain aligned point cloud data, and by performing outer product operation and voxel pooling operation on the context features extracted from the image features of multi-view images and aligned point cloud data, the BEV feature can represent the viewpoint or coordinate system of the perceived world. Therefore, the BEV feature can also represent geometric depth information features. Thus, by using the BEV feature to learn machine vision tasks, safety hazards of autonomous vehicles during driving can be effectively avoided.

[0005] Optionally, in this embodiment, temporal stereo matching of the image features of the first multi-view image and the image features of the second multi-view image includes: calculating the homography matrix between the image features of the first multi-view image and the image features of the second multi-view image; obtaining the depth information of the previous time step, using the homography matrix to estimate the depth information of the previous time step to obtain the depth information of the current time step; performing grid sampling on the depth information of the current time step and the image features of the first multi-view image to obtain a feature volume; calculating the group correlation similarity between the feature volume and the image features of the first multi-view image to obtain a cost volume; and regularizing the cost volume to obtain matched point cloud data. In the implementation of the above scheme, by using the homography matrix to estimate the depth information of the previous time step, more accurate matching point cloud data for depth perception is obtained, thus improving the accuracy of the finally obtained BEV features.

[0006] Optionally, in this embodiment, depth estimation of the image features of the first multi-view image includes: using a monocular depth estimation network to perform point cloud prediction estimation on the image features of the first multi-view image, obtaining estimated point cloud data. The monocular depth estimation network is trained using point cloud data acquired by LiDAR. In the implementation of the above scheme, by using a monocular depth estimation network to perform depth estimation, the calculation results of multi-view temporal stereo matching are compensated, thereby making the estimation of depth information more accurate and improving the estimation accuracy of point cloud data.

[0007] Optionally, in this embodiment, the estimated point cloud data and the matched point cloud data are fused, normalized, and aligned, including: fusing the estimated point cloud data and the matched point cloud data to obtain fused point cloud data; normalizing the fused point cloud data to obtain normalized point cloud data; and aligning the normalized point cloud data to obtain aligned point cloud data. In the implementation of the above scheme, by fusing, normalizing, and aligning the estimated point cloud data and the matched point cloud data, the robustness of point cloud data alignment in both the temporal and spatial domains is increased.

[0008] Optionally, in this embodiment, the camera parameters include: intrinsic parameters and extrinsic parameters; extracting context features from the image features of the first multi-view image based on the camera parameters of the first multi-view image includes: flattening and stitching the intrinsic and extrinsic parameters to obtain a parameter input vector, and expanding the parameter input vector into a multi-dimensional vector; using the multi-dimensional vector to perform weighted fusion of the image features to obtain context features. In the implementation of the above scheme, by using the parameter input vector after flattening and stitching the intrinsic and extrinsic parameters as part of the weighted fusion, the quality of depth estimation is further improved.

[0009] Optionally, in this embodiment, the outer product operation and voxel pooling operation are performed on the context features and the aligned point cloud data, including: performing an outer product operation on the context features and the aligned point cloud data to obtain point cloud features; and performing a voxel pooling operation on the point cloud features to obtain BEV features. In the implementation of the above scheme, by performing an outer product operation on the context features and the aligned point cloud data to obtain point cloud features, and performing a voxel pooling operation on the point cloud features, the robustness of the point cloud data alignment in the temporal domain is increased.

[0010] Optionally, in this embodiment, at least one machine vision task includes: object detection, map semantic segmentation, and / or motion trajectory prediction in an autonomous driving scenario; after obtaining the bird's-eye view BEV features, the method further includes: acquiring BEV features at multiple time points, including the BEV features corresponding to the current time point; concatenating and fusing the BEV features at multiple time points to obtain BEV fusion features; and using the BEV fusion features to learn the object detection, map semantic segmentation, and / or motion trajectory prediction tasks. In the implementation of the above scheme, by using BEV fusion features to simultaneously learn the object detection, map semantic segmentation, and motion trajectory prediction tasks, end-to-end multi-task learning is achieved, and mutual promotion and performance improvement between different tasks are realized, avoiding the situation where each task is trained separately and effectively saving computational resources for training multiple tasks.

[0011] This application embodiment also provides an autonomous driving BEV task learning device, including: an image feature extraction module, used to acquire a first multi-view image at the current moment in an autonomous driving scenario, and extract image features from the first multi-view image, wherein the multi-view image is an image taken from multiple perspectives of an autonomous vehicle in the autonomous driving scenario at the same moment; a temporal stereo matching module, used to acquire image features from a second multi-view image at the previous moment, and perform temporal stereo matching on the image features of the first multi-view image and the image features of the second multi-view image to obtain matched point cloud data, wherein the shooting perspective of the first multi-view image and the shooting perspective of the second multi-view image are the same; an image depth estimation module, used to perform depth estimation on the image features of the first multi-view image to obtain estimated point cloud data, and perform fusion, normalization and alignment on the estimated point cloud data and the matched point cloud data to obtain aligned point cloud data; and a BEV feature acquisition module, used to extract context features from the image features of the first multi-view image according to the camera parameters of the first multi-view image, and perform outer product operation and voxel pooling operation on the context features and the aligned point cloud data to obtain bird's-eye view BEV features, wherein the BEV features are used to learn at least one machine vision task.

[0012] Optionally, in this embodiment, the temporal stereo matching module includes: a homography matrix calculation submodule, used to calculate the homography matrix between the image features of the first multi-view image and the image features of the second multi-view image; a depth information acquisition submodule, used to acquire the depth information of the previous time step, and use the homography matrix to estimate the depth information of the previous time step to obtain the depth information of the current time step; a feature grid sampling submodule, used to perform grid sampling on the depth information of the current time step and the image features of the first multi-view image to obtain a feature volume; a correlation similarity calculation submodule, used to perform group correlation similarity calculation on the feature volume and the image features of the first multi-view image to obtain a cost volume; and a cost volume regularization submodule, used to regularize the cost volume to obtain matched point cloud data.

[0013] Optionally, in this embodiment of the application, the image depth estimation module includes: a point cloud prediction estimation submodule, used to perform point cloud prediction estimation on the image features of the first multi-view image using a monocular depth estimation network to obtain the estimated point cloud data. The monocular depth estimation network is trained using point cloud data acquired by LiDAR.

[0014] Optionally, in this embodiment, the image depth estimation module includes: a point cloud data fusion submodule, used to fuse the estimated point cloud data and the matched point cloud data to obtain fused point cloud data; a point cloud data normalization module, used to normalize the fused point cloud data to obtain normalized point cloud data; and a point cloud data alignment submodule, used to align the normalized point cloud data to obtain aligned point cloud data.

[0015] Optionally, in this embodiment, the camera parameters include: intrinsic parameters and extrinsic parameters; the BEV feature acquisition module includes: a vector acquisition and extension submodule, used to flatten and stitch the intrinsic and extrinsic parameters to obtain a parameter input vector, and to extend the parameter input vector into a multi-dimensional vector; and a feature weighted fusion submodule, used to use the multi-dimensional vector to perform weighted fusion of image features to obtain contextual features.

[0016] Optionally, in this embodiment, the BEV feature acquisition module includes: a data feature outer product submodule, used to perform an outer product operation on the context features and the aligned point cloud data to obtain point cloud features; and a voxel pooling operation submodule, used to perform a voxel pooling operation on the point cloud features to obtain BEV features.

[0017] Optionally, in this embodiment, at least one machine vision task includes: target detection, map semantic segmentation, and / or motion trajectory prediction in an autonomous driving scenario; the autonomous driving BEV task learning device further includes: a BEV feature acquisition module for acquiring BEV features at multiple times, the BEV features at multiple times including the BEV features corresponding to the current time; a BEV feature fusion module for splicing and fusing the BEV features at multiple times to obtain BEV fused features; and a BEV task learning module for using the BEV fused features to learn the target detection task, map semantic segmentation task, and / or motion trajectory prediction task.

[0018] This application also provides an electronic device, including a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions, when executed by the processor, perform the method described above.

[0019] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor to perform the methods described above.

[0020] Other features and advantages of embodiments of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of this application. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 The diagram shows a flowchart of the autonomous driving BEV task learning method provided in an embodiment of this application.

[0023] Figure 2 The diagram shown illustrates the process of extracting BEV features from a bird's-eye view provided in an embodiment of this application.

[0024] Figure 3 The illustration shows a flowchart of multi-task learning using BEV features provided in an embodiment of this application;

[0025] Figure 4 The diagram shown is a structural schematic of the autonomous driving BEV task learning device provided in an embodiment of this application.

[0026] Figure 5The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in the embodiments of this application are for illustrative and descriptive purposes only and are not intended to limit the protection scope of the embodiments of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the embodiments of this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of the embodiments of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0028] Furthermore, the described embodiments are merely a part of the embodiments of this application, and not all of them. The components of the embodiments of this application described and illustrated herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of this application, but merely to illustrate selected embodiments of this application.

[0029] It is understood that the terms "first" and "second" in the embodiments of this application are used to distinguish similar objects. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different. In the description of the embodiments of this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. The term "multiple" refers to two or more (including two), and similarly, "multiple groups" refers to two or more groups (including two groups).

[0030] Before introducing the autonomous driving BEV task learning method provided in the embodiments of this application, let's first introduce some concepts involved in the embodiments of this application:

[0031] Bird's-eye view (BEV) is a perspective or coordinate system (3D) used to describe the perceived world.

[0032] Multi-task learning (MTL) is an inferential transfer learning method that enables different tasks to mutually promote each other and improve performance. Compared with individual task learning, MTL can save computational costs.

[0033] It should be noted that the autonomous driving BEV task learning method provided in this application embodiment can be executed by an electronic device. Here, "electronic device" refers to a device terminal or server with the function of executing computer programs. Examples of device terminals include smartphones, personal computers, tablets, personal digital assistants, or mobile internet devices. A server refers to a device that provides computing services through a network. Examples of servers include x86 servers and non-x86 servers. Non-x86 servers include mainframes, minicomputers, and UNIX servers. Operating systems that can run on this electronic device include CentOS 7.6 or Ubuntu, etc.

[0034] The following describes the applicable scenarios for this autonomous driving BEV task learning method. These scenarios include, but are not limited to, autonomous vehicles, walking robots, drones, or special environment simulations. Special environment simulations include, but are not limited to, space simulations, underwater simulations, aviation simulations, or hazardous environment simulations. This autonomous driving BEV task learning method can be used to improve or upgrade machine vision tasks such as target detection, map semantic segmentation, and / or motion trajectory prediction performed on autonomous vehicles, walking robots, and / or drones. For ease of understanding and explanation, the following detailed explanations will use the autonomous vehicle application scenario as an example.

[0035] Please see Figure 1 The illustrated flowchart represents an autonomous driving BEV task learning method provided in this application embodiment. The main idea of ​​this method is to obtain BEV features that characterize the perspective or coordinate system of the perceived world by performing temporal stereo matching on multiple aligned consecutive video frames and multi-view images. These BEV features can also characterize geometric depth information features. Therefore, by using BEV features to learn machine vision tasks, safety hazards during autonomous driving can be effectively avoided. The implementation of the above-described autonomous driving BEV task learning method may include:

[0036] Step S110: Obtain the first multi-view image of the current moment in the autonomous driving scenario, and extract the image features of the first multi-view image. The multi-view image is an image taken from multiple perspectives of the autonomous vehicle in the autonomous driving scenario at the same moment.

[0037] Multi-view images refer to images taken from multiple perspectives. These multi-view images can be images taken by multiple cameras at the same time from multiple different perspectives. The multiple perspectives here include, but are not limited to: front view, left front view, right front view, rear view, left rear view, and right rear view.

[0038] The acquisition methods for the first multi-view image at the current moment in step S110 above can include: A first acquisition method, acquiring the first multi-view image at the current moment from multiple different cameras, specifically, acquiring the first multi-view image at the current moment from multiple different cameras installed on an autonomous vehicle, a walking robot, and / or a drone. The autonomous vehicle, walking robot, and / or drone can also send the first multi-view image to an electronic device, and then the electronic device receives the first multi-view image sent by the terminal device. The electronic device can store the first multi-view image in a file system, database, or mobile storage device. A second acquisition method, acquiring a pre-stored first multi-view image, specifically, acquiring the first multi-view image from a file system, database, or mobile storage device. A third acquisition method, acquiring the first multi-view image from the Internet using software such as a browser, or accessing the Internet using other applications to acquire the first multi-view image.

[0039] Please see Figure 2 The diagram shown illustrates the process of extracting BEV features from a bird's-eye view image according to an embodiment of this application. Assuming the current time is time t, then the previous time was time t-1. For example, the implementation of extracting image features from the first multi-view image in step S110 involves using an image feature extractor to extract the image features of the first multi-view image. The multi-view image is an image taken simultaneously from multiple perspectives of the autonomous vehicle in the autonomous driving scenario. This image feature can be represented as F... 2d , Where N is the number of cameras, C is the feature channel dimension, and H and W represent the height and width of the image features, respectively. The image feature extractor here is also called the image feature extraction network, which can use residual networks such as ResNet22, ResNet38, ResNet50, ResNet101, and ResNet152.

[0040] Step S120: Obtain the image features of the second multi-view image at the previous moment, and perform temporal stereo matching on the image features of the first multi-view image and the image features of the second multi-view image to obtain the matching point cloud data. The shooting angles of the first multi-view image and the second multi-view image are the same.

[0041] Step S130: Perform depth estimation on the image features of the first multi-view image to obtain the estimated point cloud data, and fuse, normalize and align the estimated point cloud data and the matched point cloud data to obtain the aligned point cloud data.

[0042] Step S140: Extract context features from the image features of the first multi-view image based on the camera parameters of the first multi-view image, and perform outer product operation and voxel pooling operation on the context features and aligned point cloud data to obtain bird's-eye view BEV features. The BEV features are used to learn at least one machine vision task.

[0043] It is understandable that camera parameters can include intrinsic parameters (also known as intrinsic K) and extrinsic parameters (also known as extrinsic E), among which, K is the intrinsic parameter of the camera parameters, which transforms the camera coordinate system to the pixel coordinate system. E is the extrinsic parameter of the camera parameters (i.e., the rotation and translation matrix), which transforms the world coordinate system to the camera coordinate system. It can be understood that the aforementioned context features are also called context-aware features. There are many ways to implement steps S120 to S140 above; therefore, the implementation methods of steps S120 to S140 will be described in detail below.

[0044] In the above implementation process, the obtained bird's-eye view BEV features are used to represent geometric depth information features. Since the BEV features are obtained by fusing, normalizing and aligning the point cloud data of temporal stereo matching and the point cloud data of depth estimation to obtain aligned point cloud data, and by performing outer product operation and voxel pooling operation on the context features extracted from the image features of multi-view images and aligned point cloud data, the BEV features can represent the viewpoint or coordinate system of the perceived world. Therefore, the BEV features can also represent geometric depth information features. Thus, by using the BEV features to learn machine vision tasks, safety hazards of autonomous vehicles during driving can be effectively avoided.

[0045] As an optional implementation of step S120 above, when performing temporal stereo matching on the image features of the first multi-view image and the image features of the second multi-view image, matching can be performed based on the homography matrix. This implementation may include:

[0046] Step S121: Calculate the homography matrix between the image features of the first multi-view image and the image features of the second multi-view image.

[0047] It is understandable that, assuming the current time is time t, then the previous time is time t-1. The image features of the first multi-view image are the same as the image features of the multi-view image at time t. Similarly, the image features of the second multi-view image are the same as the image features of the multi-view image at time t-1. (Refer to...) Figure 2 The multi-view temporal stereo matching operation in this context is used to estimate the depth information of the multi-view image at time t based on the image features and depth information of the multi-view image at time t-1.

[0048] The specific implementation of step S121 above is as follows: It can be understood that the formula can be used. To calculate the homography matrix between the image features of the first multi-view image and the image features of the second multi-view image; where d represents the sampling depth, H ij (d) represents the homography matrix between the image features of the j-th viewpoint image at time t-1 and the image features of the i-th viewpoint image at time t (with sampling depth d), K j K represents the intrinsic parameter matrix of the camera parameters for the j-th viewpoint image. i R represents the intrinsic parameter matrix of the camera parameters for the i-th viewpoint image. j R represents the rotation matrix from the camera coordinates of the j-th viewpoint image to the autonomous vehicle. i T represents the rotation matrix from the camera coordinates of the i-th viewpoint image to the autonomous vehicle. j T represents the translation matrix from the camera coordinates of the j-th viewpoint image to the autonomous vehicle. i Let I represent the translation matrix from the camera coordinates of the i-th viewpoint image to the autonomous vehicle, where I represents the identity matrix and n1 represents the principal optical axis direction of the reference camera.

[0049] Step S122: Obtain the depth information of the previous time step, use the homography matrix to estimate the depth information of the previous time step, and obtain the depth information of the current time step.

[0050] For example, in implementing step S122 above: assuming the current time is time t and the previous time is time t-1, the multi-view image at time t-1 can be used as the source image, and the multi-view image at time t as the reference image. Then, the depth information of the corresponding sampling position of the j-th view image at time t-1 relative to the i-th view image at time t can be represented as... Where D represents the number of depth divisions (e.g., depth information in monocular detection or binocular detection), and H and W represent the height and width of the feature, respectively. Then, the formula can be used... The homography matrix and the depth information from the previous time step are used to calculate the depth information for the current time step; where, H represents the depth information of the corresponding sampling position of the i-th viewpoint image at time t. ij (d) represents the homography matrix between the image features of the j-th viewpoint image at time t-1 and the image features of the i-th viewpoint image at time t (with sampling depth d). This represents the depth information of the i-th viewpoint image at time t-1 (i.e., the previous time).

[0051] Step S123: Perform grid sampling on the depth information at the current moment and the image features of the first multi-view image to obtain the feature volume.

[0052] The implementation of step S123 above is as follows: It can be understood that since a pixel in the multi-view image at time t, used as a reference image, may only partially appear in the multi-view image at time t-1, used as the source image, the depth information of a portion of the pixels in the depth information of the corresponding sampling position of the i-th view image at time t is invalid. Therefore, it is necessary to sample the depth information of the valid pixels. Finally, grid sampling technology is used to sample the depth information at the current time (which can be represented as...). The image features of the first multi-view image (which can be represented as F) and the image features of the first multi-view image (which can be represented as F) 2d The feature volume obtained by performing grid sampling can be represented as: C represents the feature channel dimension, D represents the number of depth divisions (e.g., depth information in monocular detection or depth information in binocular detection), and H and W represent the height and width of the feature volume, respectively.

[0053] Step S124: Calculate the group correlation similarity between the feature volume and the image features of the first multi-view image to obtain the cost volume.

[0054] For example, the cost body of step S124 above can be calculated using a group-wise correlation similarity measure. Specifically, a formula can be used. Group correlation similarity is calculated on the image features of the feature body and the first multi-view image to obtain the cost body; where, This represents the g-th cost volume of the i-th viewpoint image, where G is the number of groups and C is a preset constant. This represents the image features of the g-th group in the i-th view image, where ⊙ represents the inner product operation. This represents the g-th feature body of the v-th viewpoint image.

[0055] Step S125: Regularize the cost volume to obtain matching point cloud data.

[0056] For example, the implementation of step S125 above is based on the encoder-decoder principle. A 3D convolutional neural network is used to regularize the cost volume, reducing the G-channel dimension of the cost volume to 1 channel, and obtaining matching point cloud data generated based on multi-view temporal stereo matching. The 3D convolutional neural network mentioned above can adopt the 3D Unet network model.

[0057] As an optional implementation of step S130 above, when performing depth estimation on the image features of the first multi-view image, in order to perform more accurate depth estimation on textureless regions and moving objects in the image, a monocular depth estimation network can be used to compensate for the calculation results of multi-view temporal stereo matching. This implementation may include:

[0058] Step S131: Use a monocular depth estimation network to predict and estimate the point cloud features of the first multi-view image to obtain the estimated point cloud data. The monocular depth estimation network is trained using point cloud data acquired by LiDAR.

[0059] For example, step S131 described above can be implemented by using a monocular depth estimation network to predict and estimate the point cloud features of the first multi-view image, thereby obtaining the estimated point cloud data. The monocular depth estimation network can employ residual network (ResNet) models such as ResNet22, ResNet38, ResNet50, ResNet101, and ResNet152. Since this monocular depth estimation network uses radar to collect 3D point cloud data of the target object, and can use formula P... img′ =KTP is obtained by training ground truth point clouds on 3D point cloud data of the target object after performing min-pooling, one-hot encoding, and alignment operations. Here, P represents the original 3D point cloud data, T represents the data transformed to pixel coordinates, and K represents the point cloud data transformed to 2.5D image coordinates. The point cloud data can be represented as P. img (u, v, d); where u and v represent pixel positions in the pixel coordinate system, and d represents the depth information of the point cloud. Therefore, after inputting the image features of the first multi-view image into the monocular depth estimation network, the predicted point cloud data can be obtained. The predicted point cloud data can be represented as: Where N is the number of cameras, D represents the number of depth divisions (e.g., monocular depth or binocular depth, etc.), and H and W represent the height and width of the point cloud data, respectively.

[0060] As an optional implementation of step S130 above, the implementation of fusing, normalizing, and aligning the estimated point cloud data and the matched point cloud data may include:

[0061] Step S132: Fuse the estimated point cloud data and the matched point cloud data to obtain fused point cloud data.

[0062] Step S133: Normalize the fused point cloud data to obtain normalized point cloud data.

[0063] An example of implementing steps S132 to S133 above is as follows: For the estimated point cloud data (which can be represented as D...) mono The point cloud data is added to and fused with the matched point cloud data to obtain fused point cloud data. Then, the Softmax operation is used to normalize the fused point cloud data to obtain normalized point cloud data.

[0064] Step S134: Align the normalized point cloud data to obtain aligned point cloud data.

[0065] A specific example of the implementation of step S134 above is as follows: It can be understood that formula P can be used. cur =T global2cur ·T post2global ·P post Align the normalized point cloud data to obtain aligned point cloud data. Where P cur T represents the aligned point cloud data. global2cur T represents the coordinate system from the global coordinate system to the point cloud data. post2global This indicates that the coordinate system of the point cloud data from the past time step is converted to the global coordinate system.

[0066] As an optional implementation of step S140 above, the camera parameters may include: intrinsic parameters and extrinsic parameters; when extracting context features from the image features of the first multi-view image based on the camera parameters of the first multi-view image, the features can be obtained by weighted fusion based on the flattened and stitched vectors of the intrinsic and extrinsic parameters. This implementation may include:

[0067] Step S141: Flatten and concatenate the internal and external parameters to obtain the parameter input vector, and expand the parameter input vector into a multidimensional vector.

[0068] For example, the implementation of step S141 above is as follows: For the camera parameters' internal parameters (also referred to as intrinsic parameters K) and external parameters (also referred to as extrinsic parameters E), the internal parameters and external parameters are first flattened and spliced ​​to obtain the parameter input vector, and then the parameter input vector is extended to C dimensions using a multi-layer perceptron (MLP) to obtain a C-dimensional multi-dimensional vector.

[0069] Step S142: Use multidimensional vectors to perform weighted fusion of image features to obtain contextual features.

[0070] An example implementation of step S142 above is as follows: using a Squeeze-and-Excitation Network (SENet) to utilize C-dimensional multidimensional vectors to represent image features (which can be represented as F). 2d We perform weighted fusion to obtain context features (also known as context-aware features).

[0071] It is understandable that the processing steps S141 to S142 described above can be expressed by the formula F. ctx =SE(F 2d MLP(concat(flatten(E), flatten(K)))), where K represents the intrinsic parameters of the camera, E represents the extrinsic parameters of the camera, flatten is the flattening operation, concat is the stitching operation, MLP represents the multilayer perceptron processing operation, and F 2d The image features represent multi-view images, SE represents the processing operation of the extrusion network, and F represents the image features. ctx These are the obtained contextual features (also known as context-aware features), and the size of the contextual features is related to F. 2d The same applies. In the above implementation process, the parameter input vector, obtained by flattening and concatenating the intrinsic and extrinsic parameters, is used as part of the weighted fusion, thereby further improving the quality of depth estimation and the accuracy of object detection.

[0072] As an optional implementation of step S140 above, when performing outer product and voxel pooling operations on the context features and aligned point cloud data, the features can be obtained based on the point cloud features of the outer product operation. This implementation may include:

[0073] Step S143: Perform an outer product operation on the context features and the aligned point cloud data to obtain the point cloud features.

[0074] Step S144: Perform voxel pooling on the point cloud features to obtain BEV features.

[0075] For example, the implementation of steps S143 to S144 above involves using an executable program compiled or interpreted using a preset programming language to obtain aligned point cloud data, and then using formulas... Perform an outer product operation on the context features and the aligned point cloud data to obtain the point cloud features, which can be represented as follows: Among them, F pc F represents point cloud features. ctx P represents contextual features. cur This indicates the alignment of point cloud data. Let N represent the outer product operation, C be the number of cameras, H be the feature channel dimension, and W be the height and width of the feature, respectively. Then, the PointPillars method is used to process the point cloud features, assigning each point to its nearest pillar. Further, a summation voxel pooling operation is performed on the point cloud features to obtain the current time-stack bird's-eye view features (i.e., BEV features). These BEV features can be represented as... Where N is the number of cameras, C is the feature channel dimension, and X and Y represent the length and width of the BEV feature, respectively.

[0076] Please see Figure 3 The illustrated embodiment of this application provides a flowchart for multi-task learning using BEV features. As an optional implementation of the aforementioned autonomous driving BEV task learning method, the at least one machine vision task may include: target detection tasks, map semantic segmentation tasks, and / or motion trajectory prediction tasks in an autonomous driving scenario, etc. After obtaining the bird's-eye view BEV features, multi-task learning can also be performed using the BEV features. This implementation may include:

[0077] Step S210: Obtain BEV features at multiple time points, including the BEV features corresponding to the current time point.

[0078] For example, the implementation of step S210 above can be described as follows: It can be understood that for each past moment, steps S110 to S140 above can be used to generate BEV features, thereby obtaining BEV features for multiple past moments. Therefore, the BEV features for multiple past moments include the BEV features corresponding to the current moment.

[0079] Step S220: The BEV features at multiple time points are spliced ​​and fused to obtain the BEV fused features.

[0080] The specific implementation of step S220 above is as follows: It can be understood that after obtaining BEV features from multiple past time points (e.g., BEV features from T time points), the BEV features from the T time points can be spliced ​​and fused to obtain fused BEV features. A formula can be used... BEV features from multiple time points are stitched together and fused to obtain BEV fused features. Among them, F bev-ts This indicates the features of the merged bird's-eye view. This represents the bird's-eye view features at time t after alignment. This represents the bird's-eye view features at time t-1 after alignment, and concat represents the stitching and merging operation.

[0081] Step S230: Use BEV fusion features to learn for the object detection task, map semantic segmentation task, and / or motion trajectory prediction task.

[0082] For example, after obtaining the BEV fusion features based on spatiotemporal fusion, the BEV fusion features can be used to learn and train one or more tasks such as object detection, map semantic segmentation, and / or motion trajectory prediction.

[0083] Understandably, when using BEV fusion features for multi-task learning training of object detection, map semantic segmentation, and motion trajectory prediction tasks, a multi-task learning loss function can be adopted. This allows for simultaneous training on object detection, map semantic segmentation, and motion trajectory prediction tasks. This represents the total loss value. This represents the loss value for the object detection task. This represents the loss value for the map semantic segmentation task. The loss value represents the loss value for the motion trajectory prediction task, where α, β, and γ represent the loss values ​​for the object detection task, map semantic segmentation task, and motion trajectory prediction task, respectively.

[0084] Please see Figure 4 The diagram shown is a structural schematic of the autonomous driving BEV task learning device provided in an embodiment of this application; this application provides an autonomous driving BEV task learning device 300, including:

[0085] The image feature extraction module 310 is used to acquire the first multi-view image at the current moment in the autonomous driving scenario and extract the image features of the first multi-view image. The multi-view image is an image taken from multiple perspectives of the autonomous vehicle in the autonomous driving scenario at the same moment.

[0086] The temporal stereo matching module 320 is used to acquire the image features of the second multi-view image at the previous moment, and perform temporal stereo matching on the image features of the first multi-view image and the image features of the second multi-view image to obtain matched point cloud data. The shooting angles of the first multi-view image and the second multi-view image are the same.

[0087] The image depth estimation module 330 is used to perform depth estimation on the image features of the first multi-view image to obtain the estimated point cloud data, and to fuse, normalize and align the estimated point cloud data and the matched point cloud data to obtain aligned point cloud data.

[0088] BEV feature acquisition module 340 is used to extract context features from the image features of the first multi-view image based on the camera parameters of the first multi-view image, and to perform outer product operation and voxel pooling operation on the context features and aligned point cloud data to obtain bird's-eye view BEV features. The BEV features are used to learn at least one machine vision task.

[0089] Optionally, in this embodiment of the application, the temporal stereo matching module includes:

[0090] The homography matrix calculation submodule is used to calculate the homography matrix between the image features of the first multi-view image and the image features of the second multi-view image.

[0091] The depth information acquisition submodule is used to obtain the depth information of the previous time step. It uses the homography matrix to estimate the depth information of the previous time step and obtains the depth information of the current time step.

[0092] The feature grid sampling submodule is used to perform grid sampling on the depth information and image features of the first multi-view image at the current moment to obtain the feature volume.

[0093] The related similarity calculation submodule is used to perform group correlation similarity calculation on the image features of the feature body and the first multi-view image to obtain the cost body.

[0094] The cost volume regularization submodule is used to regularize the cost volume to obtain matching point cloud data.

[0095] Optionally, in this embodiment of the application, the image depth estimation module includes:

[0096] The point cloud prediction estimation submodule is used to perform point cloud prediction estimation on the image features of the first multi-view image using a monocular depth estimation network to obtain the estimated point cloud data. The monocular depth estimation network is trained using point cloud data acquired by LiDAR.

[0097] Optionally, in this embodiment of the application, the image depth estimation module includes:

[0098] The point cloud data fusion submodule is used to fuse the estimated point cloud data and the matched point cloud data to obtain fused point cloud data.

[0099] The point cloud data normalization module is used to normalize the fused point cloud data to obtain normalized point cloud data.

[0100] The point cloud data alignment submodule is used to align normalized point cloud data to obtain aligned point cloud data.

[0101] Optionally, in this embodiment, the camera parameters include: intrinsic parameters and extrinsic parameters; the BEV feature acquisition module includes:

[0102] The vector acquisition extension submodule is used to flatten and concatenate the internal and external parameters to obtain the parameter input vector, and then expand the parameter input vector into a multidimensional vector.

[0103] The feature weighted fusion submodule is used to perform weighted fusion of image features using multi-dimensional vectors to obtain contextual features.

[0104] Optionally, in this embodiment of the application, the BEV feature acquisition module includes:

[0105] The data feature outer product submodule is used to perform an outer product operation on the context features and the aligned point cloud data to obtain point cloud features.

[0106] The voxel pooling operation submodule is used to perform voxel pooling operations on point cloud features to obtain BEV features.

[0107] Optionally, in embodiments of this application, at least one machine vision task includes: object detection task, map semantic segmentation task, and / or motion trajectory prediction task in an autonomous driving scenario; the autonomous driving BEV task learning device further includes:

[0108] The BEV feature acquisition module is used to acquire BEV features at multiple time points, including the BEV features corresponding to the current time point.

[0109] The BEV feature fusion module is used to stitch together and fuse BEV features from multiple time points to obtain BEV fused features.

[0110] The BEV task learning module is used to learn object detection tasks, map semantic segmentation tasks, and / or motion trajectory prediction tasks using BEV fusion features.

[0111] It should be understood that this device corresponds to the above-described embodiment of the autonomous driving BEV task learning method and is capable of executing the various steps involved in the above method embodiment. The specific functions of this device can be found in the description above, and detailed descriptions are omitted here to avoid repetition. The device includes at least one software functional module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0112] Please see Figure 5 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 400 provided in this application includes a processor 410 and a memory 420. The memory 420 stores machine-readable instructions executable by the processor 410. When the machine-readable instructions are executed by the processor 410, the method described above is performed.

[0113] This application embodiment also provides a computer-readable storage medium 430, on which a computer program is stored. This computer program is executed by a processor 410 to perform the methods described above. The computer-readable storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0114] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0115] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, as provided in the embodiments of this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending primarily on the functions involved.

[0116] Furthermore, the functional modules of each embodiment in this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," "some examples," etc., means that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0117] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A method for learning autonomous driving BEV tasks, characterized in that, include: Acquire a first multi-view image of the current moment in the autonomous driving scenario, and extract the image features of the first multi-view image. The multi-view image is an image taken from multiple perspectives of the autonomous vehicle in the autonomous driving scenario at the same moment. The image features of the second multi-view image at the previous moment are obtained, and the image features of the first multi-view image and the image features of the second multi-view image are matched in a temporal stereo process to obtain the matched point cloud data. The shooting angle of the first multi-view image and the shooting angle of the second multi-view image are the same. Depth estimation is performed on the image features of the first multi-view image to obtain estimated point cloud data. The estimated point cloud data and the matched point cloud data are then fused, normalized, and aligned to obtain aligned point cloud data. Based on the camera parameters of the first multi-view image, context features are extracted from the image features of the first multi-view image, and the context features and the aligned point cloud data are subjected to outer product operation and voxel pooling operation to obtain bird's-eye view BEV features. The BEV features are used to learn at least one machine vision task. The step of performing temporal stereo matching of the image features of the first multi-view image and the image features of the second multi-view image includes: calculating the homography matrix between the image features of the first multi-view image and the image features of the second multi-view image; obtaining the depth information of the previous time step, using the homography matrix to estimate the depth information of the previous time step to obtain the depth information of the current time step; performing grid sampling on the depth information of the current time step and the image features of the first multi-view image to obtain a feature volume; performing group correlation similarity calculation on the feature volume and the image features of the first multi-view image to obtain a cost volume; and performing regularization on the cost volume to obtain the matched point cloud data. The camera parameters include: internal parameters and external parameters; the step of extracting context features from the image features of the first multi-view image based on the camera parameters of the first multi-view image includes: flattening and stitching the internal parameters and the external parameters to obtain a parameter input vector, and expanding the parameter input vector into a multi-dimensional vector; using the multi-dimensional vector to perform weighted fusion of the image features to obtain the context features.

2. The method according to claim 1, characterized in that, The depth estimation of the image features of the first multi-view image includes: A monocular depth estimation network is used to predict and estimate the point cloud features of the first multi-view image to obtain the estimated point cloud data. The monocular depth estimation network is trained using point cloud data acquired by LiDAR.

3. The method according to claim 1, characterized in that, The process of fusing, normalizing, and aligning the estimated point cloud data and the matched point cloud data includes: The estimated point cloud data and the matched point cloud data are fused to obtain fused point cloud data; The fused point cloud data is normalized to obtain normalized point cloud data; The normalized point cloud data is aligned to obtain the aligned point cloud data.

4. The method according to claim 1, characterized in that, The outer product operation and voxel pooling operation on the context features and the aligned point cloud data include: Perform an outer product operation on the context features and the aligned point cloud data to obtain point cloud features; Voxel pooling is performed on the point cloud features to obtain the BEV features.

5. The method according to any one of claims 1-4, characterized in that, The at least one machine vision task includes: object detection in autonomous driving scenarios, map semantic segmentation, and / or motion trajectory prediction; after obtaining the bird's-eye view BEV features, it also includes: Obtain BEV features at multiple time points, wherein the BEV features at multiple time points include: the BEV features corresponding to the current time point; The BEV features at the multiple time points are spliced ​​and fused to obtain the BEV fused features; The BEV fusion features are used to learn for object detection, map semantic segmentation, and / or motion trajectory prediction tasks.

6. An autonomous driving BEV task learning device, characterized in that, include: The image feature extraction module is used to acquire the first multi-view image at the current moment in the autonomous driving scenario and extract the image features of the first multi-view image. The multi-view image is an image taken from multiple perspectives of the autonomous vehicle in the autonomous driving scenario at the same moment. The temporal stereo matching module is used to acquire the image features of the second multi-view image at the previous moment, and perform temporal stereo matching on the image features of the first multi-view image and the image features of the second multi-view image to obtain matched point cloud data. The shooting angle of the first multi-view image and the shooting angle of the second multi-view image are the same. The image depth estimation module is used to perform depth estimation on the image features of the first multi-view image to obtain estimated point cloud data, and to fuse, normalize and align the estimated point cloud data and the matched point cloud data to obtain aligned point cloud data. The BEV feature acquisition module is used to extract context features from the image features of the first multi-view image based on the camera parameters of the first multi-view image, and to perform outer product operation and voxel pooling operation on the context features and the aligned point cloud data to obtain bird's-eye view BEV features. The BEV features are used to learn at least one machine vision task. The temporal stereo matching module includes: The homography matrix calculation submodule is used to calculate the homography matrix between the image features of the first multi-view image and the image features of the second multi-view image; The depth information acquisition submodule is used to acquire the depth information of the previous time step, use the homography matrix to estimate the depth of the depth information of the previous time step, and obtain the depth information of the current time step. The feature grid sampling submodule is used to perform grid sampling on the depth information at the current moment and the image features of the first multi-view image to obtain the feature volume; The relevant similarity calculation submodule is used to perform group correlation similarity calculation on the image features of the feature body and the first multi-view image to obtain the cost body; The cost volume regularization submodule is used to regularize the cost volume to obtain the matched point cloud data; The camera parameters include: internal parameters and external parameters; the BEV feature acquisition module includes: The vector acquisition extension submodule is used to flatten and concatenate the internal parameters and the external parameters to obtain the parameter input vector, and to extend the parameter input vector into a multi-dimensional vector; The feature weighted fusion submodule is used to perform weighted fusion of the image features using the multidimensional vector to obtain the context features.

7. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when executed by the processor, perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method as described in any one of claims 1 to 5.

Images