Bird's eye view based multi-modal perception method, apparatus, device and storage medium

Abstract

The application relates to the technical field of computer vision, in particular to a multi-modal perception method and device based on a bird's-eye view, equipment and a storage medium, wherein the method comprises the following steps: acquiring a laser radar point cloud and a perspective image of a current environment; extracting features of the perspective image through a point cloud feature extraction network to obtain pseudo point cloud bird's-eye view features; extracting features of the perspective image after inverse perspective transformation through a deep neural network to obtain bird's-eye view features; extracting features of the laser radar point cloud through the point cloud feature extraction network to obtain laser radar point cloud bird's-eye view features; fusing the pseudo point cloud bird's-eye view features, the bird's-eye view features and the laser radar point cloud bird's-eye view features; and extracting the fused features through a deep neural network to obtain bird's-eye view multi-modal features. The method disclosed by the application can solve the technical problem that an existing multi-modal perception algorithm based on a bird's-eye view is complex and cannot be deployed in a platform with limited computing capacity.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a multimodal perception method, apparatus, device and storage medium based on bird's-eye view. Background Technology

[0002] In the field of mobile robotics, most algorithms related to localization, mapping, and navigation are based on bird's-eye view space. Bird's-eye view-based perception systems obtain environmental perception results within that space, making them easily usable by localization, mapping, and navigation algorithms, thus reducing the complexity of mobile robot algorithm systems. LiDAR and cameras are crucial sensors in mobile robotics, providing ample environmental information. LiDAR can acquire 3D point clouds of the environment, making bird's-eye view perception relatively easy to achieve, as seen with PointPillar. However, due to technological and cost limitations, LiDAR point clouds are relatively sparse. Therefore, fusing camera image information is a crucial method to compensate for the sparseness of LiDAR point clouds. Cameras perform perspective transformations on the 3D environmental information to obtain 2D image data, but the perception results obtained from these images are not in bird's-eye view space, making them difficult to directly use for mapping, localization, and navigation algorithms. Therefore, the images or extracted features from the camera are usually transformed into a bird's-eye view space before being input into the perception system. For example, BEVDet and BEVFormer have gradually become mainstream in recent years. However, such algorithms are difficult to deploy on platforms with limited computing power, such as embedded neural network processors.

[0003] Therefore, existing multimodal perception algorithms based on bird's-eye view are complex and cannot be deployed on platforms with limited computing power. Summary of the Invention

[0004] The main objective of this application is to provide a multimodal perception method, apparatus, device, and storage medium based on bird's-eye view, aiming to solve the technical problem that existing multimodal perception algorithms based on bird's-eye view are complex and cannot be deployed on platforms with limited computing power.

[0005] To achieve the aforementioned objectives, this application proposes a multimodal perception method based on bird's-eye view, the method comprising:

[0006] Acquire lidar point cloud and perspective images of the current environment;

[0007] The features of the perspective image are extracted using a point cloud feature extraction network to obtain pseudo point cloud bird's-eye view features;

[0008] Perform inverse perspective transformation on the perspective image to obtain a bird's-eye view of the perspective image;

[0009] The features of the bird's-eye view are obtained by extracting features from the bird's-eye view using a deep neural network;

[0010] The features of the lidar point cloud are extracted by the point cloud feature extraction network to obtain the lidar point cloud bird's-eye view features.

[0011] The pseudo-point cloud bird's-eye view features, the bird's-eye view features, and the lidar point cloud bird's-eye view features are fused together, and the fused features are extracted through the deep neural network to obtain the multimodal features of the bird's-eye view.

[0012] Further, before the step of extracting features from the perspective image through a point cloud feature extraction network to obtain pseudo-point cloud bird's-eye view features, the following steps are included:

[0013] Obtain the camera intrinsic parameters used to capture the perspective image;

[0014] The pixel depth of the perspective image is calculated by a depth estimation network, and the pseudo point cloud in the camera coordinate system is obtained by combining the camera intrinsic parameters.

[0015] The pseudo-point cloud in the camera coordinate system is transformed into a pseudo-point cloud in the bird's-eye view reference coordinate system by using camera extrinsic parameters.

[0016] Further, the step of extracting features from the perspective image through a point cloud feature extraction network to obtain pseudo-point cloud bird's-eye view features includes:

[0017] Based on the coordinates of the pseudo-point cloud, the pseudo-point cloud is divided into regions, and each pseudo-point cloud is converted into a vector.

[0018] The PointNet model is used to extract features from the pseudo point cloud of each region to generate a pseudo image of the pseudo point cloud.

[0019] The pseudo-image features are further extracted using the backbone model to obtain pseudo-point cloud bird's-eye view features.

[0020] Further, before the step of extracting features from the lidar point cloud through the point cloud feature extraction network to obtain the lidar point cloud bird's-eye view features, the following steps are included:

[0021] The lidar point cloud is transformed into a lidar point cloud in the bird's-eye view reference coordinate system using an extrinsic parameter matrix.

[0022] Further, the step of extracting features from the lidar point cloud through the point cloud feature extraction network to obtain the lidar point cloud bird's-eye view features includes:

[0023] Based on the coordinates of the lidar point cloud, the lidar point cloud is divided into regions, and each lidar point cloud is converted into a vector.

[0024] The PointNet model is used to extract features from the LiDAR point cloud of each region to generate a pseudo image of the LiDAR point cloud.

[0025] The pseudo-image features are further extracted using the backbone model to obtain the aerial view features of the LiDAR point cloud.

[0026] Furthermore, the feature extraction processes for the pseudo point cloud bird's-eye view features, the bird's-eye view features, and the lidar point cloud bird's-eye view features are independent of each other.

[0027] Furthermore, the point cloud feature extraction network uses the pointpillar backbone model.

[0028] This application also provides a multimodal sensing device based on a bird's-eye view, the device comprising:

[0029] The acquisition module is used to acquire the lidar point cloud and perspective image of the current environment;

[0030] The first feature extraction module is used to extract features of the perspective image through a point cloud feature extraction network to obtain pseudo point cloud bird's-eye view features.

[0031] The inverse perspective transformation module is used to perform inverse perspective transformation on the perspective image to obtain a bird's-eye view of the perspective image;

[0032] The second feature extraction module is used to extract features from the bird's-eye view through a deep neural network to obtain bird's-eye view features;

[0033] The third feature extraction module is used to extract the features of the lidar point cloud through the point cloud feature extraction network to obtain the lidar point cloud bird's-eye view features.

[0034] The feature fusion module is used to fuse the pseudo point cloud bird's-eye view features, the bird's-eye view features, and the lidar point cloud bird's-eye view features, and extract the fused features through the deep neural network to obtain the multimodal features of the bird's-eye view.

[0035] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0036] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0037] This application provides a multimodal perception method based on bird's-eye view, which realizes the fusion of pseudo point cloud bird's-eye view features, bird's-eye view features and lidar point cloud bird's-eye view features, reduces the computational difficulty of bird's-eye view feature fusion, and realizes the fusion of multimodal features in bird's-eye view space in an easy-to-deploy manner, so that the multimodal perception system in bird's-eye view space can be applied to platforms with limited computing power and supporting operators, such as mobile robots. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating a multimodal perception method based on a bird's-eye view according to an embodiment of this application;

[0039] Figure 2 This is a schematic block diagram of the structure of a multimodal sensing device based on a bird's-eye view according to an embodiment of this application;

[0040] Figure 3 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application.

[0041] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any modules and all combinations of one or more associated listed items.

[0044] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0045] Reference Figure 1 This invention provides a multimodal perception method based on a bird's-eye view, including steps S1-S6, specifically:

[0046] S1. Obtain the lidar point cloud and perspective image of the current environment;

[0047] LiDAR can acquire 3D point clouds of the environment, i.e., LiDAR point clouds. A camera performs perspective transformation on the environmental information in three-dimensional space to obtain image data in 2D space, i.e., perspective images. In this application, the perspective images can be captured by one or more surround-mounted cameras.

[0048] S2. Extract the features of the perspective image through a point cloud feature extraction network to obtain pseudo point cloud bird's-eye view features;

[0049] First, the perspective image is transformed into a coordinate system. A depth estimation network is used to obtain the pixel depth of the perspective image from one or more viewpoints. This depth is then combined with camera intrinsic parameters to transform it into a pseudo-point cloud in the camera coordinate system. Next, an extrinsic parameter matrix is ​​used to transform the pseudo-point cloud in the camera coordinate system into a pseudo-point cloud in the bird's-eye view reference coordinate system. After the coordinate transformation, the bird's-eye view features of the pseudo-point cloud are extracted by a point cloud feature extraction network.

[0050] S3. Perform inverse perspective transformation on the perspective image to obtain a bird's-eye view of the perspective image;

[0051] S4. Extract the features of the bird's-eye view using a deep neural network to obtain the bird's-eye view features;

[0052] The inverse perspective transformation method is used to convert perspective images from one or more viewpoints into a bird's-eye view. Then, feature extraction is performed using a deep neural network to obtain the bird's-eye view features.

[0053] S5. Extract the features of the lidar point cloud through the point cloud feature extraction network to obtain the lidar point cloud bird's-eye view features.

[0054] LiDAR point cloud data is a dataset of spatial points obtained by scanning with a 3D LiDAR device. Each point contains information such as its 3D coordinates. First, the coordinates of the LiDAR point cloud are transformed using an extrinsic parameter matrix. Then, the bird's-eye view features of the LiDAR point cloud are extracted by a point cloud feature extraction network.

[0055] S6. The pseudo point cloud bird's-eye view features, the bird's-eye view features, and the lidar point cloud bird's-eye view features are fused together, and the fused features are extracted through the deep neural network to obtain the multimodal features of the bird's-eye view.

[0056] Due to the sparsity of LiDAR point clouds, it is necessary to supplement them with image information from cameras. Feature fusion methods include, but are not limited to, concatenation, addition, and multiplication. After fusion, a deep neural network is used to extract features from the fused features, obtaining multimodal features of the bird's-eye view. Based on multimodal features, visual tasks in the bird's-eye view space can be achieved, such as object detection, semantic segmentation, and lane detection.

[0057] As described above, the fusion of pseudo point cloud bird's-eye view features, bird's-eye view features, and lidar point cloud bird's-eye view features has been achieved, reducing the computational difficulty of bird's-eye view feature fusion. This enables the fusion of multimodal features in the bird's-eye view space in an easy-to-deploy manner, allowing the multimodal perception system in the bird's-eye view space to be applied to platforms with limited computing power and supporting operators, such as mobile robots.

[0058] In one embodiment, before the step of extracting features from the perspective image using a point cloud feature extraction network to obtain pseudo-point cloud bird's-eye view features, the following steps are included:

[0059] S201. Obtain the camera intrinsic parameters for capturing the perspective image;

[0060] S202. Calculate the pixel depth of the perspective image through a depth estimation network, and combine it with the camera intrinsic parameters to obtain the pseudo point cloud in the camera coordinate system.

[0061] S203. Convert the pseudo point cloud in the camera coordinate system into a pseudo point cloud in the bird's-eye view reference coordinate system using camera extrinsic parameters.

[0062] A depth estimation network extracts features from the input perspective image, and then performs regression on each pixel to predict the distance of each pixel in the perspective image from the camera, calculating the pixel depth of each pixel. The pixel depth is then calculated in conjunction with camera intrinsic parameters to transform the pseudo-point cloud in the image coordinate system into a pseudo-point cloud in the camera coordinate system. Combined with camera extrinsic parameter calculations, the pseudo-point cloud in the camera coordinate system is transformed into a pseudo-point cloud in the bird's-eye view reference coordinate system. Camera intrinsic parameters are related to internal camera parameters, such as focal length, optical center, and distortion coefficients, and do not change with the object's position. Camera extrinsic parameters are only related to external camera parameters, such as camera height, pitch angle, and yaw angle.

[0063] In one embodiment, the step of extracting features from the perspective image using a point cloud feature extraction network to obtain pseudo-point cloud bird's-eye view features includes:

[0064] S204. Based on the coordinates of the pseudo-point cloud, divide the pseudo-point cloud into regions and convert each pseudo-point cloud into a vector.

[0065] S205. Use the PointNet model to extract features from the pseudo point cloud of each region and generate a pseudo image of the pseudo point cloud.

[0066] S206. Further extract the features of the pseudo image using the backbone model to obtain the pseudo point cloud bird's-eye view features.

[0067] The point cloud feature extraction network used in this application is the pointpillar backbone model. Based on the X and Y axes (excluding the Z axis) of the pseudo-point cloud data, the pseudo-point cloud data is divided into grids, and the pseudo-point cloud data falling into a grid forms a pillar.

[0068] Each pseudo-point cloud is represented by a vector, namely (x, y, z, r, x). c y c , z c x p y p Where x, y, z are the true 3D coordinates of the pseudo-point cloud, and r is the reflection intensity; x c y c , z c x is the geometric center of all points in the pillar containing the pseudo-point cloud; p For xx c y p For yy c This reflects the relative position of the pseudo-point cloud to the aforementioned geometric center.

[0069] Assuming each sample contains P non-empty pillars, and each pillar contains N pseudo-point cloud data points, then this sample can be represented by a (D, P, N) tensor. If each pillar contains more than N pseudo-point cloud data points, then the number of pseudo-point cloud data points in that pillar is randomly sampled down to N; if each pillar contains fewer than N pseudo-point cloud data points, then the fewer pseudo-point cloud data points are padded with 0, until the number of pseudo-point cloud data points in that pillar equals N, thus achieving tensor quantization of the point cloud data. PointNet processes and extracts features from the tensor-quantized pseudo-point cloud data. PointNet can directly process the pseudo-point cloud. For each point in the input pseudo-point cloud, it learns its corresponding spatial encoding, and then uses the features of all points to obtain a global pseudo-point cloud feature, generating a pseudo-image of the pseudo-point cloud. The backbone uses two networks. One network continuously reduces the resolution of the feature map while increasing its dimension, thus obtaining three feature maps with different resolutions. The other network upsamples the three feature maps to the same size and then concatenates them to obtain the pseudo-point cloud bird's-eye view feature.

[0070] In one embodiment, before the step of extracting features from the lidar point cloud using the point cloud feature extraction network to obtain the lidar point cloud bird's-eye view features, the following steps are included:

[0071] S501. The lidar point cloud is converted into a lidar point cloud in the bird's-eye view reference coordinate system by using the extrinsic parameter matrix.

[0072] In order to project the points measured in the lidar coordinate system onto the bird's-eye view reference coordinate system, an additional transformation, namely the extrinsic parameter matrix, needs to be added to the mapping operation so that the lidar point cloud data can be associated with the bird's-eye view reference coordinate system.

[0073] In one embodiment, the step of extracting features from the lidar point cloud using the point cloud feature extraction network to obtain the lidar point cloud bird's-eye view features includes:

[0074] S502. Based on the coordinates of the lidar point cloud, divide the lidar point cloud into regions and convert each lidar point cloud into a vector.

[0075] S503. Use the PointNet model to extract features from the LiDAR point cloud of each region and generate a pseudo image of the LiDAR point cloud.

[0076] S504. Further extract the features of the pseudo image using the backbone model to obtain the aerial view features of the LiDAR point cloud.

[0077] The point cloud feature extraction network used in this application is the pointpillar backbone model. Based on the X and Y axes (excluding the Z axis) of the LiDAR point cloud data, the LiDAR point cloud data is divided into grids, and the LiDAR point cloud data falling into a grid form a pillar.

[0078] Each lidar point cloud is represented by a vector, namely (x, y, z, r, x). c y c , z c x p y p Where x, y, z are the actual three-dimensional coordinates of the lidar point cloud, and r is the reflection intensity; x c y c , z c x is the geometric center of all points in the pillar containing the lidar point cloud; p For xx c y p For yy c This reflects the relative position of the lidar point cloud with the aforementioned geometric center.

[0079] Assuming each sample contains P non-empty pillars, and each pillar contains N LiDAR point cloud data points, then this sample can be represented by a (D, P, N) tensor. If each pillar contains more than N LiDAR point cloud data points, then the number of LiDAR point cloud data points in that pillar is randomly sampled down to N; if each pillar contains fewer than N LiDAR point cloud data points, then the fewer LiDAR point cloud data points are padded with 0, until the number of LiDAR point cloud data points in that pillar equals N, thus achieving tensor quantization of the LiDAR point cloud data. PointNet processes and extracts features from the tensor quantized LiDAR point cloud data. PointNet can directly process the LiDAR point cloud. For each point in the input LiDAR point cloud, it learns its corresponding spatial encoding, and then uses the features of all points to obtain a global LiDAR point cloud feature, generating a pseudo image of the LiDAR point cloud. The backbone uses two networks. One network continuously reduces the resolution of the feature map while increasing the dimension of the feature map, thus obtaining three feature maps with different resolutions. Another network upsamples the three feature maps to the same size and then concatenates them to obtain the LiDAR point cloud bird's-eye view features.

[0080] In one embodiment, the feature extraction processes for the pseudo point cloud bird's-eye view features, the bird's-eye view features, and the lidar point cloud bird's-eye view features described above are independent of each other.

[0081] Feature information of two modalities is obtained from the perspective image information captured by the camera: pseudo-point cloud bird's-eye view features and bird's-eye view features. Feature information of another modality is obtained from the LiDAR point cloud. The feature extraction process of the three modalities is carried out separately, resulting in three different modal feature information. This makes the fused bird's-eye view features rich and accurate, which is convenient for subsequent tasks such as object detection, semantic segmentation, and lane line detection based on the bird's-eye view.

[0082] In one embodiment, the point cloud feature extraction network described above is a pointpillar backbone model.

[0083] The PointPillar Backbone model transforms points in point cloud data into individual pillars, creating pseudo-images. The backbone model then further extracts features from these pseudo-images. Point cloud feature detection processed by the PointPillar Backbone model offers high speed and accuracy.

[0084] Reference Figure 2 This is a structural block diagram of a multimodal sensing device based on a bird's-eye view according to an embodiment of this application. The device includes:

[0085] The acquisition module 100 is used to acquire the lidar point cloud and perspective image of the current environment;

[0086] The first feature extraction module 200 is used to extract features of the perspective image through a point cloud feature extraction network to obtain pseudo point cloud bird's-eye view features.

[0087] The inverse perspective transformation module 300 is used to perform inverse perspective transformation on the perspective image to obtain a bird's-eye view of the perspective image;

[0088] The second feature extraction module 400 is used to extract features of the bird's-eye view through a deep neural network to obtain bird's-eye view features;

[0089] The third feature extraction module 500 is used to extract the features of the lidar point cloud through the point cloud feature extraction network to obtain the lidar point cloud bird's-eye view features.

[0090] The feature fusion module 600 is used to fuse the pseudo point cloud bird's-eye view features, the bird's-eye view features, and the lidar point cloud bird's-eye view features, and extract the fused features through the deep neural network to obtain multimodal features of the bird's-eye view.

[0091] In one embodiment, the above-mentioned multimodal sensing device based on bird's-eye view further includes:

[0092] The first coordinate transformation module is used to obtain the camera intrinsic parameters of the perspective image; calculate the pixel depth of the perspective image through a depth estimation network, and obtain a pseudo point cloud in the camera coordinate system by combining the camera intrinsic parameters; and transform the pseudo point cloud in the camera coordinate system into a pseudo point cloud in the bird's-eye view reference coordinate system by using camera extrinsic parameters.

[0093] In one embodiment, the above-mentioned multimodal sensing device based on bird's-eye view further includes:

[0094] The first feature extraction submodule is used to divide the pseudo point cloud into regions based on the coordinates of the pseudo point cloud and convert each pseudo point cloud into a vector; to extract features from the pseudo point cloud of each region using the PointNet model to generate a pseudo image of the pseudo point cloud; and to further extract features from the pseudo image using the backbone model to obtain the bird's-eye view features of the pseudo point cloud.

[0095] In one embodiment, the above-mentioned multimodal sensing device based on bird's-eye view further includes:

[0096] The second coordinate transformation module is used to transform the lidar point cloud into a lidar point cloud in the bird's-eye view reference coordinate system through an external parameter matrix.

[0097] In one embodiment, the above-mentioned multimodal sensing device based on bird's-eye view further includes:

[0098] The third feature extraction submodule is used to divide the LiDAR point cloud into regions based on the coordinates of the LiDAR point cloud and convert each LiDAR point cloud into a vector; to extract features from the LiDAR point cloud in each region using the PointNet model to generate a pseudo image of the LiDAR point cloud; and to further extract features from the pseudo image using the backbone model to obtain the aerial view features of the LiDAR point cloud.

[0099] Reference Figure 3 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 3As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores usage data in the multimodal perception method based on bird's-eye view. The network interface allows communication with external terminals via a network connection. Furthermore, the computer device may also include input devices and a display screen. When the aforementioned computer program is executed by a processor, it employs a multimodal perception method based on a bird's-eye view, comprising the following steps: acquiring a LiDAR point cloud and a perspective image of the current environment; extracting features from the perspective image using a point cloud feature extraction network to obtain pseudo-point cloud bird's-eye view features; performing an inverse perspective transformation on the perspective image to obtain a bird's-eye view of the perspective image; extracting features from the bird's-eye view using a deep neural network to obtain bird's-eye view features; extracting features from the LiDAR point cloud using the point cloud feature extraction network to obtain LiDAR point cloud bird's-eye view features; fusing the pseudo-point cloud bird's-eye view features, the bird's-eye view features, and the LiDAR point cloud bird's-eye view features, and extracting the fused features using the deep neural network to obtain multimodal bird's-eye view features.

[0100] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.

[0101] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a multimodal perception method based on a bird's-eye view, including the following steps: acquiring a LiDAR point cloud and a perspective image of the current environment; extracting features from the perspective image using a point cloud feature extraction network to obtain pseudo-point cloud bird's-eye view features; performing an inverse perspective transformation on the perspective image to obtain a bird's-eye view of the perspective image; extracting features from the bird's-eye view using a deep neural network to obtain bird's-eye view features; extracting features from the LiDAR point cloud using the point cloud feature extraction network to obtain LiDAR point cloud bird's-eye view features; fusing the pseudo-point cloud bird's-eye view features, the bird's-eye view features, and the LiDAR point cloud bird's-eye view features, and extracting the fused features using the deep neural network to obtain multimodal bird's-eye view features. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0102] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0103] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0104] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multimodal perception method based on bird's-eye view, characterized in that, The method includes: Acquire lidar point cloud and perspective images of the current environment; The features of the perspective image are extracted using a point cloud feature extraction network to obtain pseudo point cloud bird's-eye view features; Perform inverse perspective transformation on the perspective image to obtain a bird's-eye view of the perspective image; The features of the bird's-eye view are obtained by extracting features from the bird's-eye view using a deep neural network; The features of the lidar point cloud are extracted by the point cloud feature extraction network to obtain the lidar point cloud bird's-eye view features. The pseudo point cloud bird's-eye view features, the bird's-eye view features, and the lidar point cloud bird's-eye view features are fused together, and the fused features are extracted through the deep neural network to obtain the multimodal features of the bird's-eye view. The feature extraction processes for the pseudo-point cloud bird's-eye view features, the bird's-eye view features, and the lidar point cloud bird's-eye view features are independent of each other.

2. The multimodal perception method based on bird's-eye view according to claim 1, characterized in that, Before the step of extracting features from the perspective image using a point cloud feature extraction network to obtain pseudo-point cloud bird's-eye view features, the following steps are included: Obtain the camera intrinsic parameters used to capture the perspective image; The pixel depth of the perspective image is calculated by a depth estimation network, and the pseudo point cloud in the camera coordinate system is obtained by combining the camera intrinsic parameters. The pseudo-point cloud in the camera coordinate system is transformed into a pseudo-point cloud in the bird's-eye view reference coordinate system by using camera extrinsic parameters.

3. The multimodal perception method based on bird's-eye view according to claim 2, characterized in that, The step of extracting features from the perspective image using a point cloud feature extraction network to obtain pseudo-point cloud bird's-eye view features includes: Based on the coordinates of the pseudo-point cloud, the pseudo-point cloud is divided into regions, and each pseudo-point cloud is converted into a vector. The PointNet model is used to extract features from the pseudo point cloud of each region to generate a pseudo image of the pseudo point cloud. The pseudo-image features are further extracted using the backbone model to obtain pseudo-point cloud bird's-eye view features.

4. The multimodal perception method based on bird's-eye view according to claim 1, characterized in that, Before the step of extracting features from the lidar point cloud through the point cloud feature extraction network to obtain the lidar point cloud bird's-eye view features, the following steps are included: The lidar point cloud is transformed into a lidar point cloud in the bird's-eye view reference coordinate system using an extrinsic parameter matrix.

5. The multimodal perception method based on bird's-eye view according to claim 4, characterized in that, The step of extracting features from the lidar point cloud through the point cloud feature extraction network to obtain the lidar point cloud bird's-eye view features includes: Based on the coordinates of the lidar point cloud, the lidar point cloud is divided into regions, and each lidar point cloud is converted into a vector. The PointNet model is used to extract features from the LiDAR point cloud of each region to generate a pseudo image of the LiDAR point cloud. The pseudo-image features are further extracted using the backbone model to obtain the aerial view features of the LiDAR point cloud.

6. The multimodal perception method based on bird's-eye view according to claim 1, characterized in that, The point cloud feature extraction network uses the pointpillar backbone model.

7. A multimodal sensing device based on bird's-eye view, characterized in that, The device includes: The acquisition module is used to acquire the lidar point cloud and perspective image of the current environment; The first feature extraction module is used to extract features of the perspective image through a point cloud feature extraction network to obtain pseudo point cloud bird's-eye view features. The inverse perspective transformation module is used to perform inverse perspective transformation on the perspective image to obtain a bird's-eye view of the perspective image; The second feature extraction module is used to extract features from the bird's-eye view through a deep neural network to obtain bird's-eye view features; The third feature extraction module is used to extract the features of the lidar point cloud through the point cloud feature extraction network to obtain the lidar point cloud bird's-eye view features. The feature fusion module is used to fuse the pseudo point cloud bird's-eye view features, the bird's-eye view features, and the lidar point cloud bird's-eye view features, and extract the fused features through the deep neural network to obtain the multimodal features of the bird's-eye view. The feature extraction processes for the pseudo-point cloud bird's-eye view features, the bird's-eye view features, and the lidar point cloud bird's-eye view features are independent of each other.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images