Multi-modal fusion perception method and device for vehicle, vehicle and storage medium

Abstract

This application relates to the field of autonomous driving technology, and in particular to a multimodal fusion perception method, device, vehicle, and storage medium for vehicles. The method includes: acquiring laser point cloud data collected by an onboard LiDAR and image data collected by an onboard camera; placing all laser point clouds from the laser point cloud data into voxels on a target overhead view, and extracting laser voxel features from the laser point cloud data; extracting two-dimensional image features from each image in the image data, projecting them onto voxels on the target overhead view, and extracting visual voxel features from the image data; aligning the laser voxel features and visual voxel features in time and space, and then fusing them on the target overhead view to obtain the multimodal fusion perception result of the vehicle using the fused features. This solves the problems of related technologies that perform target-level fusion based on single-sensor detection results, which cannot fuse data from multiple sensors, resulting in large errors and low perception accuracy.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a multimodal fusion perception method, device, vehicle, and storage medium for a vehicle. Background Technology

[0002] Autonomous driving perception technology is a crucial module for driverless vehicles. Perception mainly includes tasks such as object detection, segmentation, depth estimation, and prediction, enabling vehicles to perceive road environment information through sensors. Currently, the mainstream approach primarily uses LiDAR point cloud data and image data separately for road perception. However, individual sensors have inherent limitations. For example, in complex scenarios, occlusion or truncation of the target can significantly hinder object detection or semantic segmentation algorithms. Furthermore, due to mechanical limitations, LiDAR has varying resolutions at different distances, especially in foggy or overcast weather, where the detection range becomes extremely short. While different sensors should be placed in different locations, combining different sensor data, such as LiDAR and image data, in the development of perception algorithms can significantly improve perception results.

[0003] Therefore, multimodal fusion perception algorithms are a crucial task for autonomous driving perception systems. However, data noise, insufficient utilization of perception data, and calibration errors between multiple sensors can lead to significant errors in the fusion process. Furthermore, in engineering applications, data annotation and the challenges of multi-sensor synchronization pose substantial difficulties. Therefore, algorithm design must consider and mitigate potential engineering challenges as much as possible.

[0004] Multi-sensor fusion sensing technology, also known as multimodal sensing technology, is traditionally categorized into three main types in multimodal sensor methodologies: data-level fusion (early fusion), feature-level fusion (deep fusion), and target-level fusion (late fusion). Data-level fusion requires spatial alignment, feature-level fusion is a fusion across feature spaces, and target-level fusion is the fusion of the sensor's detected target.

[0005] In autonomous driving perception algorithms, fusion perception algorithms are rarely deployed in practical engineering projects. More often, they are used for weak fusion or later-stage decision-level fusion to fuse targets. Decision-level fusion, based on sensor detection results, performs certain discrimination, classification, and simple logical operations, making higher-level decisions according to application requirements; this is advanced fusion. Decision-level fusion is application-oriented, improving overall perception recall by fusing target-level data based on single-sensor detection results, but it cannot fuse data from multiple sensors. Summary of the Invention

[0006] This application provides a multimodal fusion perception method, device, vehicle, and storage medium for vehicles, in order to solve the problems of related technologies that, by performing target-level fusion based on single sensor detection results, cannot fuse data from multiple sensors, resulting in large errors and low perception accuracy.

[0007] The first aspect of this application provides a multimodal fusion perception method for a vehicle, comprising the following steps: acquiring laser point cloud data collected by an onboard LiDAR and image data collected by an onboard camera; placing all laser point clouds in the laser point cloud data into voxels on a pre-constructed target overhead view, and extracting laser voxel features from the voxels on the target overhead view; extracting two-dimensional image features from each image in the image data, and projecting all two-dimensional image features onto voxels on the target overhead view, and extracting visual voxel features from the voxels on the target overhead view; aligning the laser voxel features and the visual voxel features in time and space, and fusing the aligned laser voxel features and the visual voxel features on the target overhead view to obtain fused features; and using the fused features to obtain the multimodal fusion perception result of the vehicle.

[0008] Based on the aforementioned technical means, this embodiment of the application can input the laser point cloud data collected by the vehicle-mounted LiDAR and the image data collected by the vehicle-mounted camera into voxels on the target overhead view, and extract the corresponding laser voxel features and visual voxel features. Then, the laser and visual voxel features are aligned and fused to obtain the multimodal fusion perception result of the vehicle. This solves the problems of difficulty in synchronizing the camera's exposure time and the LiDAR's sampling time, as well as the calibration errors between the camera and the LiDAR. It improves the accuracy of perception while compensating for the shortcomings of both the visual and LiDAR sensors.

[0009] Optionally, in one embodiment of this application, the step of extracting laser voxel features from the voxels of the target top view includes: discretizing the point cloud data in the laser point cloud data into a uniformly spaced grid in a two-dimensional plane, and adjusting the grid size according to the density of the point cloud distribution to create one or more columnar sets; adding preset features to the point cloud in each columnar set in multiple dimensions, so that each point cloud has multiple dimensional features; tensorizing each point cloud according to the number of non-empty columns in each columnar set, the number of point clouds in each columnar set, and the multiple dimensional features of each point cloud to obtain tensorized point clouds; inputting the tensorized point clouds into a preset multilayer perceptron to obtain deep point cloud features, and performing max pooling on the deep point cloud features to output feature vectors; encoding feature maps according to the feature vectors; sequentially traversing all point clouds in all columnar sets, and distributing all feature vectors to the positions corresponding to the feature maps to obtain the laser voxel features of the laser point cloud data.

[0010] Based on the above technical means, the embodiments of this application can discretize point clouds into a uniformly spaced grid in a two-dimensional plane and create a set of columnar sets. By increasing the features of the point clouds, each point cloud has multiple dimensional features, which is more conducive to the extraction of point cloud features from the network model. After obtaining a large number of non-empty columnar sets, restrictions are imposed on the number of non-empty columns and the number of points in each column for each sample, thereby creating a tensor of size normalized. By performing multi-layer sensing operations on each point, deep-level point cloud features are obtained. After encoding and traversal, all feature vectors are distributed to the corresponding positions of the feature map to obtain the laser voxel features of the laser point cloud data, making the lidar sensing data more accurate, avoiding the loss of data features, and further improving the sensing results.

[0011] Optionally, in one embodiment of this application, before the tensor quantization process is performed on each point cloud based on the number of non-empty pillars in each column set, the number of point clouds in each column set, and the multiple dimensional features of each point cloud to obtain the tensor quantized point cloud, the method further includes: if the amount of data in the column set is greater than a preset value, then multiple data are randomly sampled from the column set as the point cloud for tensor quantization processing of the column set; otherwise, the column set is filled with a preset identifier.

[0012] Based on the above technical means, the embodiments of this application can perform random sampling when there is too much data in the columnar column, and fill it with preset labels when there is too little data, thereby further improving the accuracy of data feature extraction.

[0013] Optionally, in one embodiment of this application, the step of extracting the two-dimensional image features of each image in the image data and projecting all the two-dimensional image features onto the voxels on the target overhead view includes: extracting the image features of each image in the image data; using the image features as an index to query a preset depth vector table to obtain the depth vector corresponding to the image features, and projecting the depth vector onto the target overhead view to fuse each image in the image data onto the target overhead view.

[0014] Based on the above technical means, the embodiments of this application can extract image features captured by vehicle-mounted cameras, use the image features to query the corresponding depth in a preset depth vector table, project and fuse the depth onto the plane of the corresponding overhead view, so as to extract the visual voxel features of the image data, which is beneficial to data fusion.

[0015] Optionally, in one embodiment of this application, before querying the preset depth vector table, the method further includes: acquiring offline image data, wherein the offline image data includes image features and multiple pixels; associating preset data with discrete depths with each pixel, and generating a new image based on the image features, wherein the new image includes image features and depth features; converting the depth features of each new image to the target coordinate system through camera extrinsic parameters to obtain the correspondence between depth vectors and image features, and constructing the preset depth vector table based on the correspondence.

[0016] Based on the above technical means, the embodiments of this application can extract image features and depth features of pixels, and transform the depth data in each image to the same coordinate system through camera extrinsic parameters, thereby obtaining a series of virtual point cloud depth information and image position lookup tables, so as to query the corresponding depth vector by the position of the pixel.

[0017] Optionally, in one embodiment of this application, aligning the laser voxel features and the visual voxel features in time and space, and fusing the aligned laser voxel features and the visual voxel features on the target top-view to obtain fused features includes: inputting the laser voxel features and the visual voxel features into a learnable alignment module with three fully connected layers, outputting a first feature value of the laser voxel features and a second and third feature value of the visual voxel features; calculating a similarity matrix based on the first and second feature values, using the similarity matrix to determine the coordinate relationship between the vehicle-mounted camera and the vehicle-mounted LiDAR, performing normalization processing using operators of a preset activation function, and using attention and a matrix to weight and aggregate the third feature value to obtain a fusion matrix; concatenating the fusion matrix and the first feature value to obtain a concatenated matrix, using the concatenated matrix to extract features of a preset level from the second and third feature values, and concatenating the features of the preset level with the laser voxel features to obtain the fused features.

[0018] Based on the above technical means, the embodiments of this application can use three fully connected layers to convert the voxels of the laser point cloud to obtain the first feature value, the second feature value, and the third feature value of the laser voxel features. A similarity matrix is ​​calculated based on the first and second feature values. The similarity between the laser features and the visual features is aggregated to extract the features that are common to both laser and visual features. The aggregated camera features are processed by the fully connected layer and matrix-concatenated with the features of the point cloud to extract deeper features. Finally, it is connected with the original lidar features to complete the feature fusion. This can not only correct spatially misaligned data but also temporally misaligned data, effectively solving the problem of temporal and spatial mismatch between data.

[0019] Optionally, in one embodiment of this application, before fusing and aligning the laser voxel features and the visual voxel features on the target top view, the method further includes: given a preset three-dimensional key point of the laser point cloud data, performing geometrically related data augmentation on the laser point cloud data using preset enhancement parameters to obtain augmented data, thereby enhancing the data. During fusion, all augmented data are inverted using the preset enhancement parameters to obtain the original coordinates of the preset three-dimensional key point of the laser point cloud data, and the corresponding camera features are located in two-dimensional space using the original coordinates.

[0020] Based on the above technical means, the embodiments of this application can use a reverse augmentation data augmentation method. When point cloud data is augmented, a data-augmented 3D key point is given to realize data augmentation. When fusing, the original coordinates of the 3D key point are obtained by reversing all the data. Then, the corresponding 2D coordinates are found in the camera space. This not only aligns different types of key points, but also processes the data before and after augmentation.

[0021] A second aspect of this application provides a multimodal fusion perception device for a vehicle, comprising: a first acquisition module for acquiring laser point cloud data collected by an onboard LiDAR and image data collected by an onboard camera; a first extraction module for placing all laser point clouds in the laser point cloud data into voxels on a pre-constructed target top-view image, and extracting laser voxel features from the voxels on the target top-view image; a second extraction module for extracting two-dimensional image features from each image in the image data, projecting all two-dimensional image features onto voxels on the target top-view image, and extracting visual voxel features from the voxels on the target top-view image; and a perception module for aligning the laser voxel features and the visual voxel features in time and space, fusing the aligned laser voxel features and the visual voxel features on the target top-view image to obtain fused features, and using the fused features to obtain a multimodal fusion perception result for the vehicle.

[0022] Optionally, in one embodiment of this application, the first extraction module is further configured to: discretize the point cloud in the laser point cloud data into a uniformly spaced grid in a two-dimensional plane, and adjust the grid size according to the density of the point cloud distribution to create one or more columnar sets; add preset features to the point cloud in each columnar set in multiple dimensions, so that each point cloud has multiple dimensional features; tensile process each point cloud according to the number of non-empty columns in each columnar set, the number of point clouds in each columnar set, and the multiple dimensional features of each point cloud to obtain tensile-quantized point clouds; input the tensile-quantized point clouds into a preset multilayer perceptron to obtain deep point cloud features, and perform max pooling operation on the deep point cloud features to output feature vectors; generate feature maps based on the feature vector encoding; sequentially traverse all point clouds in all columnar sets, and distribute all feature vectors to the positions corresponding to the feature maps to obtain laser voxel features of the laser point cloud data.

[0023] Optionally, in one embodiment of this application, it further includes: a judgment module, configured to, before the tensor quantization processing of each point cloud based on the number of non-empty pillars in each column set, the number of point clouds in each column set, and the multiple dimensional features of each point cloud to obtain the tensor quantized point cloud, if the amount of data in the column set is greater than a preset value, randomly sample multiple data from the column set as the point cloud for tensor quantization processing of the column set; otherwise, fill the column set with a preset identifier.

[0024] Optionally, in one embodiment of this application, the second extraction module is further configured to extract image features of each image in the image data; use the image features as an index to query a preset depth vector table to obtain the depth vector corresponding to the image features, and project the depth vector onto the target overhead view to fuse each image in the image data onto the target overhead view.

[0025] Optionally, in one embodiment of this application, it further includes: a second acquisition module, configured to acquire offline image data before querying a preset depth vector table, wherein the offline image data includes image features and multiple pixels; a generation module, configured to associate preset data with discrete depths with each pixel, and generate a new image based on the image features, wherein the new image includes image features and depth features; and a construction module, configured to convert the depth features of each new image to a target coordinate system through camera extrinsic parameters, obtain the correspondence between depth vectors and image features, and construct the preset depth vector table based on the correspondence.

[0026] Optionally, in one embodiment of this application, the perception module is further configured to input the laser voxel features and the visual voxel features into a learnable alignment module having three fully connected layers, and output a first feature value of the laser voxel features and a second and third feature value of the visual voxel features; calculate a similarity matrix based on the first and second feature values, use the similarity matrix to determine the coordinate relationship between the vehicle camera and the vehicle lidar, perform normalization processing using operators of a preset activation function, and use attention and a matrix to weight and aggregate the third feature value to obtain a fusion matrix; concatenate the fusion matrix and the first feature value to obtain a concatenation matrix, use the concatenation matrix to extract features of a preset level from the second and third feature values, and concatenate the features of the preset level with the laser voxel features to obtain the fused features.

[0027] Optionally, in one embodiment of this application, it further includes: a processing module, configured to, before fusing and aligning the laser voxel features and the visual voxel features on the target top view, provide a preset three-dimensional key point to the laser point cloud data, and use preset enhancement parameters to perform geometrically related data augmentation on the laser point cloud data to obtain augmented data, thereby enhancing the data. During fusion, all augmented data is inverted using the preset enhancement parameters to obtain the original coordinates of the preset three-dimensional key point of the laser point cloud data, and the corresponding camera features are located in two-dimensional space using the original coordinates.

[0028] A third aspect of this application provides a vehicle, comprising: an onboard lidar for acquiring lidar point cloud data within a sensing area; an onboard camera for acquiring image data of the vehicle's surrounding environment; and a processor for acquiring the lidar point cloud data acquired by the onboard lidar and the image data acquired by the onboard camera; placing all lidar point clouds in the lidar data into voxels on a pre-constructed target top-view image, and extracting lidar voxel features from the voxels on the target top-view image; extracting two-dimensional image features from each image in the image data, and projecting all two-dimensional image features onto voxels on the target top-view image, and extracting visual voxel features from the voxels on the target top-view image; aligning the lidar voxel features and the visual voxel features in time and space, and fusing the aligned lidar voxel features and the visual voxel features on the target top-view image to obtain fused features; and using the fused features to obtain the vehicle's multimodal fusion perception result.

[0029] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the multimodal fusion perception method for vehicles as described in the above embodiments.

[0030] Therefore, this application has at least the following beneficial effects:

[0031] 1. This application embodiment can input laser point cloud data collected by vehicle-mounted LiDAR and image data collected by vehicle-mounted camera into voxels on the target overhead view, and extract the corresponding laser voxel features and visual voxel features. Then, the laser and visual voxel features are aligned and fused to obtain the multimodal fusion perception result of the vehicle. This solves the problems of difficulty in synchronizing camera exposure time and LiDAR sampling time, as well as the calibration errors between the camera and LiDAR. It improves the perception accuracy while compensating for the shortcomings of each sensor, namely vision and LiDAR.

[0032] 2. In this embodiment, point clouds can be discretized into a uniformly spaced grid on a two-dimensional plane and a set of columnar columns can be created. By increasing the features of the point clouds, each point cloud has multiple dimensional features, which is more conducive to the extraction of point cloud features from the network model. After obtaining a large number of non-empty columnar columns, the number of non-empty columns and the number of points in each column of each sample are restricted to create a tensor of size normalized. By performing multi-layer sensing operations on each point, deep point cloud features are obtained. After encoding and traversal, all feature vectors are distributed to the corresponding positions of the feature map to obtain the laser voxel features of the laser point cloud data. This makes the LiDAR sensing data more accurate, avoids the loss of data features, and further improves the sensing results.

[0033] 3. In this embodiment of the application, if there is too much data in the column, random sampling is performed; if there is too little data, it is filled with preset labels, which further improves the accuracy of data feature extraction.

[0034] 4. The embodiments of this application can extract image features captured by the vehicle-mounted camera, use the image features to query the corresponding depth in a preset depth vector table, project the depth and fuse it onto the plane of the corresponding top-down view, so as to extract the visual voxel features of the image data, which is beneficial to data fusion.

[0035] 5. The embodiments of this application can extract image features and depth features of pixels, and transform the depth data in each image to the same coordinate system through camera extrinsic parameters, thereby obtaining a series of virtual point cloud depth information and image position lookup tables, so as to query the corresponding depth vector by the position of the pixel.

[0036] 6. In this embodiment, three fully connected layers can be used to convert the voxels of the laser point cloud to obtain the first, second, and third feature values ​​of the laser voxel features. A similarity matrix is ​​calculated based on the first and second feature values. The similarity between the laser features and the visual features is aggregated to extract features that are common to both laser and visual features. The aggregated camera features are processed by the fully connected layer and matrix-concatenated with the features of the point cloud to extract deeper features. Finally, the features are connected with the original LiDAR features to complete feature fusion. This not only corrects spatially misaligned data but also temporally misaligned data, effectively solving the problem of temporal and spatial mismatch between data.

[0037] 7. This application embodiment can use an inverse augmentation data augmentation method. When point cloud data is augmented, a data-augmented 3D key point is given to realize data augmentation. When fusing, the original coordinates of the 3D key point are obtained by inverting all the data. Then, the corresponding 2D coordinates are found in the camera space. This not only aligns different types of key points, but also processes the data before and after augmentation.

[0038] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0039] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0040] Figure 1 This is a flowchart of a multimodal fusion perception method for a vehicle provided according to an embodiment of this application;

[0041] Figure 2 This is a schematic diagram of a visual laser fusion network model provided according to an embodiment of this application;

[0042] Figure 3 This is a schematic diagram of lidar feature extraction according to an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of image feature extraction according to an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of point cloud image feature fusion according to an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of the inverse data augmentation method provided according to an embodiment of this application;

[0046] Figure 7 This is a block diagram of a multimodal fusion sensing device for a vehicle according to an embodiment of this application;

[0047] Figure 8 This is a block diagram of a vehicle provided according to an embodiment of this application.

[0048] Explanation of reference numerals in the attached drawings: First acquisition module-100, First extraction module-200, Second extraction module-300, Sensing module-400, Vehicle-mounted LiDAR-500, Vehicle-mounted camera-600, Processor-700. Detailed Implementation

[0049] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0050] The following description, with reference to the accompanying drawings, outlines a multimodal fusion perception method, apparatus, vehicle, and storage medium for vehicles according to embodiments of this application. Addressing the problems mentioned in the background section, this application provides a multimodal fusion perception method for vehicles. In this method, laser point cloud data acquired by an onboard LiDAR and image data acquired by an onboard camera are input into voxels on a target top-down view. Corresponding laser voxel features and visual voxel features are extracted. Then, the laser and visual voxel features are aligned and fused to obtain the multimodal fusion perception result for the vehicle. This avoids the difficulty in synchronizing camera exposure time and LiDAR sampling time, as well as calibration errors between the camera and LiDAR. It compensates for the shortcomings of both the visual and LiDAR sensors while improving perception accuracy. Therefore, it solves the problems of related technologies that perform target-level fusion based on single-sensor detection results, which cannot fuse data from multiple sensors, resulting in large errors and low perception accuracy.

[0051] Specifically, Figure 1 This is a flowchart illustrating a multimodal fusion perception method for vehicles provided in an embodiment of this application.

[0052] The feature-level fusion algorithm based on point cloud LiDAR and multiple cameras in this application embodiment can effectively avoid the algorithm failure caused by the failure of a certain sensor. That is to say, the vision algorithm can still work effectively when the LiDAR fails, and the overall algorithm can still work effectively when a certain camera fails. It can also be designed according to different task requirements.

[0053] Specifically, such as Figure 2 As shown, both LiDAR and vision systems use their respective feature extractors to extract features and obtain the features of the embedding layer. Then, the features from the LiDAR and the embedding layer are aligned and fused. After the fusion module is completed, a detection head or a semantic segmentation head can be inserted to predict the target or the semantics of the image and point cloud.

[0054] like Figure 1 As shown, the multimodal fusion perception method for this vehicle includes the following steps:

[0055] In step S101, laser point cloud data collected by the vehicle-mounted lidar and image data collected by the vehicle-mounted camera are acquired.

[0056] It is understood that the embodiments of this application can acquire laser point cloud data collected by vehicle-mounted LiDAR and image data collected by vehicle-mounted cameras. By combining different sensors, such as LiDAR and image data, to develop perception algorithms, the accuracy of perception can be improved, thereby helping vehicles perceive road environment information through sensors.

[0057] In step S102, all laser point clouds in the laser point cloud data are placed into voxels on a pre-constructed target top view, and laser voxel features of the laser point cloud data are extracted from the voxels on the target top view.

[0058] The point cloud representation obtained from the lidar in this application embodiment is generally in the form of three-dimensional coordinates x, y, z and reflection intensity i. Since point cloud data is usually discrete, this application embodiment can put the discrete data into a three-dimensional grid, which is called a voxel. Laser voxel features are obtained by processing and feature extraction of the laser point cloud data.

[0059] In one embodiment of this application, extracting laser voxel features from laser point cloud data from voxels in a target top-view image includes: discretizing the point cloud data into a uniformly spaced grid on a two-dimensional plane, and adjusting the grid size according to the density of the point cloud distribution to create one or more columnar sets; adding preset features to the point cloud in each columnar set in multiple dimensions, so that each point cloud has multiple dimensional features; tensorizing each point cloud according to the number of non-empty columns in each columnar set, the number of point clouds in each columnar set, and the multiple dimensional features of each point cloud to obtain tensorized point clouds; inputting the tensorized point clouds into a preset multilayer perceptron to obtain deep point cloud features, and performing max pooling on the deep point cloud features to output feature vectors; generating feature maps based on feature vector encoding; sequentially traversing all point clouds in all columnar sets, and distributing all feature vectors to the positions corresponding to the feature maps to obtain laser voxel features of the laser point cloud data.

[0060] Specifically, for extracting laser voxel features from laser point cloud data, such as Figure 3As shown, this embodiment of the application can discretize the point cloud into a uniformly spaced grid in the xy plane. The grid size can be adjusted according to the density of the point cloud distribution. Then, a set of columnar sets P can be created. The z-axis does not require parameter control. Therefore, there are usually unevenly distributed point clouds in the columnar set P. Because the point cloud is discrete, most columnar sets are empty sets. Typical point cloud data only has four-dimensional features. This embodiment of the application can add features to the point cloud, which is more conducive to the extraction of point cloud features from the network model. In this embodiment of the application, xc, yc, zc, xp, and yp are added to the points in each column (where the subscript c represents the distance to the arithmetic mean of all points in the column, and the subscript p represents the offset from the x and y centers of the column). Each point in the lidar then has multi-dimensional features (D).

[0061] Furthermore, after obtaining a large number of non-empty column sets, this embodiment can impose restrictions on the number of non-empty columns (P) and the number of points (N) in each column for each sample, thereby creating a tensor of normalized size (D, P, N) for processing and feature extraction of the tensorized point cloud data. In actual implementation, this embodiment can perform multilayer perceptron + BN layer + ReLU layer operation on each point. The multilayer perceptron is used to obtain deep point cloud features, thereby generating a tensor of (C, P, N). Max pooling is performed on the deep point cloud features to output a feature vector of (C, P). After encoding, a feature map is generated and set to zero. This process is repeated for all P values, distributing all feature vectors to the corresponding positions in the feature map, thereby obtaining the laser voxel features of the laser point cloud data. The created feature map is a pseudo-image of (C, W, H).

[0062] It should be noted that, for the bird's-eye view network of point clouds, the embodiments of this application can use the backbone network combination of PointPillars to extract point cloud data features.

[0063] In step S103, the two-dimensional image features of each image in the image data are extracted, and all two-dimensional image features are projected onto voxels on the target top view. The visual voxel features of the image data are extracted from the voxels on the target top view.

[0064] It is understood that, in this embodiment, the two-dimensional image features of each image in the extracted image data can be projected onto voxels in the target overhead view for visual voxel feature extraction. Similar to the extraction in the lidar overhead view, this embodiment can use a sub-network to extract features from the target, ultimately obtaining a Cn×Wn×Hn matrix. Here, Cn is the feature channel output by the feature network, Wn is the width of the network output feature map, and Hn is the height of the network output feature map. In actual execution, the image input in this embodiment can typically be multiple images or a single image. The images must ensure a large overlap with the laser, and extrinsic parameters between images need to be obtained to perform coordinate transformation between images.

[0065] It should be noted that for bird's-eye view networks of images, the embodiments of this application may use a combination of ResNet blocks similar to PointPillars, without specific limitations.

[0066] In one embodiment of this application, extracting two-dimensional image features of each image in the image data and projecting all two-dimensional image features onto voxels on the target overhead view includes: extracting image features of each image in the image data; querying a preset depth vector table using the image features as an index to obtain the depth vector corresponding to the image features, and projecting the depth vector onto the target overhead view to fuse each image in the image data onto the target overhead view.

[0067] Specifically, such as Figure 4 As shown, this embodiment of the application can extract features from each image in the image data, mainly extracting image features of pixels and features with depth, i.e., obtaining a matrix [N,C,D,H,W]. This embodiment of the application can query the corresponding depth according to the matrix INDEX, i.e., obtain a series of points with features, i.e., M×D (M=N×C×W×H), where C is the feature of the image in the top view, W is the width of the tensor in the top view, and H is the height of the tensor in the top view. The data structure of the points in this embodiment of the application is similar to the feature data structure of the LiDAR point cloud module. The depth can be projected onto the plane of the corresponding top view, i.e., C×W'×H', further fusing multiple frames of images into a single top view plane. Here, W' is the width in the top view, and H' is the height in the top view. W' can be different from W in the image, and similarly, H' can be different from H. The size of W' and H' is determined by the voxel size of the point.

[0068] It should be noted that in this embodiment, the voxel size of the plane used to construct the bird's-eye view is generally consistent with that of the LiDAR plane, which is beneficial for data fusion. For each image individually, this embodiment can use a two-dimensional matrix to generate a three-dimensional voxel matrix, thereby generating feature maps on the bird's-eye view from a virtual depth lookup table. For the network operating on each image individually, this embodiment can use an EfficientNet-B0 backbone network pre-trained on ImageNet as the baseline network. EfficientNets are network architectures found through extensive architecture search under limited resources; their depth, width, and resolution can be increased proportionally according to the size requirements of the network model.

[0069] In step S104, the laser voxel features and visual voxel features are aligned in time and space, and the aligned laser voxel features and visual voxel features are fused on the target top view to obtain fused features. The fused features are then used to obtain the multimodal fusion perception result of the vehicle.

[0070] Because it is difficult to synchronize the camera's exposure time with the LiDAR's sampling time, and because there are calibration errors between the camera and LiDAR, the main challenge in camera-LiDAR fusion lies in temporal and spatial mismatches. To better align information from the LiDAR function with the most relevant camera function, this application introduces a learnable alignment method. This method utilizes a cross-attention mechanism to dynamically capture the correlation between the two modes, correcting not only spatially misaligned data but also temporally misaligned data, effectively resolving the problem of temporal and spatial mismatches between data.

[0071] In one embodiment of this application, laser voxel features and visual voxel features are aligned in time and space, and the aligned laser voxel features and visual voxel features are fused on a target top-view to obtain fused features. This includes: inputting the laser voxel features and visual voxel features into a learnable alignment module with three fully connected layers, outputting a first feature value of the laser voxel features and a second and third feature value of the visual voxel features; calculating a similarity matrix based on the first and second feature values, using the similarity matrix to determine the coordinate relationship between the vehicle-mounted camera and the vehicle-mounted LiDAR, performing normalization processing using operators of a preset activation function, and weighting and aggregating the third feature value using attention and a matrix to obtain a fusion matrix; concatenating the fusion matrix and the first feature value to obtain a concatenated matrix; extracting features of a preset level from the second and third feature values ​​using the concatenated matrix, and concatenating the features of the preset level with the laser voxel features to obtain the fused features.

[0072] Specifically, embodiments of this application can input features of voxels on a top-view image of a laser point cloud and features of voxels from multiple cameras on the top-view image. Three fully connected layers are used to convert the voxels of the laser point cloud into a query ql (first feature value), the camera features into kc (second feature value), and the value vc (third feature value), respectively. For each query (i.e., voxel unit), such as... Figure 5 As shown, in this embodiment, the similarity matrix for each time step can be obtained by calculating the inner product of q1 and kc between the visual features and the laser radar features. The relationship between the coordinates of the visual and laser radar points can then be obtained through the similarity matrix. Normalization is then performed using the softmax operator. Finally, attention and a matrix are used to weight and aggregate the value vc containing camera information to obtain the final fusion matrix. The weighted aggregation aggregates the similarity between the laser and visual features, identifying common features between them. The aggregated camera features are processed by a fully connected layer and matrix-concatenated with the point cloud features to extract deeper features. Finally, this is connected to the original laser radar features to complete feature fusion. This not only overcomes calibration errors between sensors in engineering applications but also overcomes errors caused by asynchronous operation between sensors, improving perception accuracy.

[0073] In one embodiment of this application, before fusing and aligning the laser voxel features and visual voxel features on the target top view, the method further includes: given a preset three-dimensional key point of the laser point cloud data, performing geometrically related data augmentation on the laser point cloud data using preset enhancement parameters to obtain augmented data, thereby enhancing the data. During fusion, all augmented data are inverted using the preset enhancement parameters to obtain the original coordinates of the preset three-dimensional key point of the laser point cloud data, and the corresponding camera features are located in two-dimensional space using the original coordinates.

[0074] The preset 3D key points and preset enhancement parameters can be determined according to the actual situation and are not specifically limited.

[0075] To address the alignment issues caused by the increase in geometrically related data, such as Figure 6 As shown, this application embodiment can use an inverse augmentation data augmentation method. When point cloud data is augmented, given an augmented 3D key point (which can be any 3D coordinate, such as a LiDAR point, voxel center, etc.), this application embodiment, when using geometry-related data augmentation, can first save the augmentation parameters (such as rotation angle). During fusion, it will reverse all the data to obtain the original coordinates of the 3D key point, and then find the corresponding 2D coordinates in the camera space. This not only aligns different types of key points, but also processes the data before and after augmentation.

[0076] In one embodiment of this application, before querying the preset depth vector table, the method further includes: acquiring offline image data, wherein the offline image data includes image features and multiple pixels; associating preset data with discrete depths with each pixel, and generating a new image based on the image features, wherein the new image includes image features and depth features; transforming the depth features of each new image into the target coordinate system through camera extrinsic parameters to obtain the correspondence between depth vectors and image features, and constructing the preset depth vector table based on the correspondence.

[0077] It is understood that this application embodiment can generate an offline virtual depth vector table for an image. If the image data is N×3×H×W, where N is the number of cameras, W is the width of the image, H is the height, E is the extrinsic parameter of the camera, I is the intrinsic parameter of the camera, and p is a pixel (h, w) on the image. This application embodiment associates a set of data {d0+Δ,...,d0+|D|Δ} with each pixel. Each pixel can be represented as {(h, w, d)∈R 3|d∈D}, where D is a set of discrete depths. Therefore, this application embodiment can generate a new image, i.e., D×H×W data, and there are no parameters that need to be trained in this expression.

[0078] If there are multiple images, this embodiment of the application can convert the depth D in each image to the same coordinate system through extrinsic parameters to obtain a series of virtual point cloud depth information and a lookup table of image positions. The size of the lookup table is [D,H,W,3], where 3 corresponds to the three dimensions of x, y, and z coordinates, so that the corresponding depth vector can be found by the position of the pixel.

[0079] In one embodiment of this application, before tensorizing each point cloud according to the number of non-empty pillars in each column set, the number of point clouds in each column set, and the multiple dimensional features of each point cloud to obtain the tensorized point cloud, the method further includes: if the amount of data in the column set is greater than a preset value, then randomly sample multiple data from the column set as a column set for tensorizing point cloud processing; otherwise, fill the column set with a preset identifier.

[0080] It is understood that, in this embodiment of the application, if the amount of data in the column is greater than a preset value, random sampling can be performed; otherwise, zeros are used for padding, further improving the accuracy of data feature extraction. The preset value can be determined according to actual conditions; for ease of calculation, the preset flag can be set to zero, without specific limitations.

[0081] The multimodal fusion perception method for vehicles proposed in this application involves inputting laser point cloud data collected by an onboard LiDAR and image data collected by an onboard camera into voxels on a target overhead view, extracting corresponding laser voxel features and visual voxel features, and then aligning and fusing the laser and visual voxel features to obtain the multimodal fusion perception result of the vehicle. This avoids the problem of difficulty in synchronizing camera exposure time and LiDAR sampling time, as well as calibration errors between the camera and LiDAR. It can compensate for the shortcomings of each sensor (visual and LiDAR) while improving perception accuracy. Therefore, it solves the problems of related technologies that perform target-level fusion based on single-sensor detection results, which cannot fuse data from multiple sensors, resulting in large errors and low perception accuracy.

[0082] Next, referring to the accompanying drawings, a multimodal fusion sensing device for a vehicle according to an embodiment of this application is described.

[0083] Figure 7 This is a block diagram of a multimodal fusion sensing device for a vehicle according to an embodiment of this application.

[0084] like Figure 7 As shown, the multimodal fusion perception device 10 of the vehicle includes: a first acquisition module 100, a first extraction module 200, a second extraction module 300, and a perception module 400.

[0085] The system comprises the following modules: a first acquisition module 100, which acquires laser point cloud data collected by the vehicle-mounted LiDAR and image data collected by the vehicle-mounted camera; a first extraction module 200, which places all laser point clouds in the laser point cloud data into voxels on a pre-constructed target top-view image and extracts laser voxel features from the voxels on the target top-view image; a second extraction module 300, which extracts two-dimensional image features from each image in the image data and projects all two-dimensional image features onto voxels on the target top-view image and extracts visual voxel features from the voxels on the target top-view image; and a perception module 400, which aligns the laser voxel features and visual voxel features in time and space, fuses the aligned laser voxel features and visual voxel features on the target top-view image to obtain fused features, and uses the fused features to obtain the multimodal fusion perception result of the vehicle.

[0086] Optionally, in one embodiment of this application, the first extraction module 200 is further configured to discretize the point cloud in the laser point cloud data into a uniformly spaced grid in a two-dimensional plane, and adjust the grid size according to the density of the point cloud distribution to create one or more columnar sets; add preset features to the point cloud in each columnar set in multiple dimensions, so that each point cloud has multiple dimensional features; tensile process each point cloud according to the number of non-empty columns in each columnar set, the number of point clouds in each columnar set, and the multiple dimensional features of each point cloud to obtain tensile-quantized point clouds; input the tensile-quantized point clouds into a preset multilayer perceptron to obtain deep point cloud features, and perform max pooling operation on the deep point cloud features to output feature vectors; generate feature maps based on feature vector encoding; sequentially traverse all point clouds in all columnar sets, and distribute all feature vectors to the positions corresponding to the feature maps to obtain laser voxel features of the laser point cloud data.

[0087] Optionally, in one embodiment of this application, the apparatus 10 of this application embodiment further includes: a judgment module, configured to, before performing tensor quantization processing on each point cloud based on the number of non-empty pillars in each column set, the number of point clouds in each column set, and the multiple dimensional features of each point cloud to obtain the tensor quantized point cloud, if the amount of data in the column set is greater than a preset value, randomly sample multiple data from the column set as the column set for tensor quantization processing; otherwise, fill the column set with a preset identifier.

[0088] Optionally, in one embodiment of this application, the second extraction module 300 is further used to extract image features of each image in the image data; using the image features as an index, querying a preset depth vector table to obtain the depth vector corresponding to the image features, and projecting the depth vector onto the target overhead view, so as to fuse each image in the image data into the target overhead view.

[0089] Optionally, in one embodiment of this application, the apparatus 10 of this application embodiment further includes: a second acquisition module, a generation module, and a construction module.

[0090] The second acquisition module is used to acquire offline image data before querying the preset depth vector table. The offline image data includes image features and multiple pixels. The generation module is used to associate the preset data with discrete depths with each pixel and generate a new image based on the image features. The new image includes image features and depth features. The construction module is used to transform the depth features of each new image into the target coordinate system through camera extrinsic parameters, obtain the correspondence between depth vectors and image features, and construct the preset depth vector table based on the correspondence.

[0091] Optionally, in one embodiment of this application, the perception module 400 is further configured to input the laser voxel features and visual voxel features into a learnable alignment module with three fully connected layers, and output the first feature value of the laser voxel features and the second and third feature values ​​of the visual voxel features; calculate a similarity matrix based on the first and second feature values, use the similarity matrix to determine the coordinate relationship between the vehicle camera and the vehicle lidar, perform normalization processing using operators of a preset activation function, and use attention and a matrix to weight and aggregate the third feature value to obtain a fusion matrix; concatenate the fusion matrix and the first feature value to obtain a concatenated matrix, use the concatenated matrix to extract features of a preset level from the second and third feature values, and concatenate the features of the preset level with the laser voxel features to obtain fused features.

[0092] Optionally, in one embodiment of this application, the apparatus 10 of this application embodiment further includes: a processing module, configured to, before fusing and aligning the laser voxel features and visual voxel features on the target top view, provide a preset three-dimensional key point to the laser point cloud data, and use preset enhancement parameters to perform geometrically related data augmentation on the laser point cloud data to obtain augmented data, thereby achieving data enhancement. In the process of fusion, all augmented data are inverted using the preset enhancement parameters to obtain the original coordinates of the preset three-dimensional key point of the laser point cloud data, and the corresponding camera features are located in two-dimensional space using the original coordinates.

[0093] It should be noted that the foregoing explanation of the multimodal fusion perception method embodiment for vehicles also applies to the multimodal fusion perception device for vehicles in this embodiment, and will not be repeated here.

[0094] The multimodal fusion perception device for vehicles proposed in this application integrates laser point cloud data collected by an onboard LiDAR and image data collected by an onboard camera into voxels on a target overhead view. Corresponding laser voxel features and visual voxel features are extracted, and then the laser and visual voxel features are aligned and fused to obtain the multimodal fusion perception result of the vehicle. This avoids the problems of inconsistent camera exposure time and LiDAR sampling time, as well as calibration errors between the camera and LiDAR. It compensates for the shortcomings of both the visual and LiDAR sensors while improving perception accuracy. Therefore, it solves the problems of related technologies that perform target-level fusion based on single-sensor detection results, which cannot fuse data from multiple sensors, resulting in large errors and low perception accuracy.

[0095] Figure 8 This is a block diagram of a vehicle provided in an embodiment of this application. The vehicle 20 includes: an onboard LiDAR 500, an onboard camera 600, and a processor 700.

[0096] The system comprises: a vehicle-mounted lidar 500 for collecting lidar point cloud data within the sensing area; a vehicle-mounted camera 600 for collecting image data of the vehicle's surrounding environment; and a processor 700 for acquiring the lidar point cloud data and image data from the vehicle-mounted camera. The system integrates all lidar point clouds into voxels on a pre-constructed target top-down view, extracting lidar voxel features from these voxels. It also extracts two-dimensional image features from each image in the image data and projects all these features onto voxels on the target top-down view, extracting visual voxel features from these voxels. Finally, it aligns the lidar and visual voxel features in time and space, fuses the aligned features on the target top-down view to obtain fused features, and uses these fused features to obtain the vehicle's multimodal fusion perception results.

[0097] According to the vehicle proposed in this application, laser point cloud data collected by the vehicle-mounted LiDAR and image data collected by the vehicle-mounted camera are input into voxels on the target overhead view, and corresponding laser voxel features and visual voxel features are extracted. Then, the laser and visual voxel features are aligned and fused to obtain the multimodal fusion perception result of the vehicle. This avoids the problem of the camera's exposure time and LiDAR sampling time being difficult to synchronize, as well as the calibration error between the camera and LiDAR. It can compensate for the shortcomings of the visual and LiDAR sensors respectively, while improving the perception accuracy. Thus, it solves the problems of related technologies that perform target-level fusion based on single-sensor detection results, which cannot fuse data from multiple sensors, resulting in large errors and low perception accuracy.

[0098] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described multimodal fusion perception method for vehicles.

[0099] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0100] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0101] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0102] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0103] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0104] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A multi-modal fusion perception method of a vehicle, characterized in that, Includes the following steps: Acquire laser point cloud data collected by the vehicle-mounted LiDAR and image data collected by the vehicle-mounted camera; All laser point clouds in the laser point cloud data are placed into voxels on a pre-constructed target top view, and the laser voxel features of the laser point cloud data are extracted from the voxels on the target top view. Two-dimensional image features are extracted from each image in the image data, and all two-dimensional image features are projected onto voxels on the target top view. Visual voxel features of the image data are extracted from the voxels on the target top view. The laser voxel features and the visual voxel features are aligned in time and space, and the aligned laser voxel features and the visual voxel features are fused on the target top view to obtain fused features. The fused features are then used to obtain the multimodal fusion perception results of the vehicle. The process of aligning the laser voxel features and the visual voxel features in time and space, and fusing the aligned laser voxel features and the visual voxel features on the target top-view to obtain the fused features includes: The laser voxel features and the visual voxel features are input into a learnable alignment module with three fully connected layers, and the first feature value of the laser voxel features and the second and third feature values ​​of the visual voxel features are output. A similarity matrix is ​​calculated based on the first and second eigenvalues. The coordinate relationship between the vehicle camera and the vehicle lidar is determined using the similarity matrix. Normalization is performed using operators of a preset activation function. The third eigenvalue is weighted and aggregated using attention and a matrix to obtain a fusion matrix. The fusion matrix and the first feature value are concatenated to obtain a concatenated matrix. The second feature value and the third feature value are extracted using the concatenated matrix at a preset level. The features at the preset level are then concatenated with the laser voxel features to obtain the fusion features.

2. The method of claim 1, wherein, The extraction of laser voxel features from the voxels of the target top view of the laser point cloud data includes: The point cloud data of the laser point cloud is discretized into a uniformly spaced grid in a two-dimensional plane, and the grid size is adjusted according to the density of the point cloud distribution to create one or more columnar sets. Preset features are added to the point cloud of each column set in multiple dimensions, so that each point cloud has multiple dimensional features. Tensor quantization is performed on each point cloud according to the number of non-empty columns in each column set, the number of point clouds in each column set, and the multiple dimensional features of each point cloud, to obtain the tensor quantized point cloud. The quantized point cloud is input into a preset multilayer perceptron to obtain deep point cloud features. Max pooling is then performed on the deep point cloud features to output a feature vector. A feature map is generated based on the feature vector encoding. The laser voxel features of the laser point cloud data are obtained by sequentially traversing all point clouds in all columnar sets and distributing all feature vectors to the corresponding positions of the feature maps.

3. The method of claim 2, wherein, Before performing tensor quantization on each point cloud based on the number of non-empty pillars in each columnar set, the number of point clouds in each columnar set, and the multiple dimensional features of each point cloud, to obtain the tensor quantized point cloud, the method further includes: If the amount of data in the column set is greater than a preset value, then multiple data points are randomly sampled from the column set as the point cloud for tensor quantization processing; otherwise, the column set is filled with a preset identifier.

4. The method according to claim 1, characterized in that, The step of extracting two-dimensional image features from each image in the image data and projecting all two-dimensional image features onto voxels in the target top-view image includes: Extract image features from each image in the image data; Using the image features as an index, a preset depth vector table is queried to obtain the depth vector corresponding to the image features, and the depth vector is projected onto the target overhead view to fuse each image in the image data into the target overhead view.

5. The method according to claim 4, characterized in that, Before querying the preset depth vector table, the following is also included: Acquire offline image data, wherein the offline image data includes image features and multiple pixels; Preset data with discrete depth is associated with each pixel, and a new image is generated based on the image features, wherein the new image includes image features and depth features; The depth features of each new image are transformed into the target coordinate system using camera extrinsic parameters to obtain the correspondence between depth vectors and image features, and the preset depth vector table is constructed based on the correspondence.

6. The method according to any one of claims 1-5, characterized in that, Before the laser voxel features and the visual voxel features fused and aligned on the target top view, the following is also included: Given a preset 3D key point in laser point cloud data, geometrically related data augmentation is performed on the laser point cloud data using preset enhancement parameters to obtain augmented data, thereby enhancing the data. During fusion, all augmented data is inverted using the preset enhancement parameters to obtain the original coordinates of the preset 3D key point in the laser point cloud data, and the corresponding camera features are located in 2D space using the original coordinates.

7. A multimodal fusion sensing device for a vehicle, characterized in that, The device is used to implement the multimodal fusion perception method for vehicles as described in any one of claims 1-6, comprising: The first acquisition module is used to acquire laser point cloud data collected by the vehicle-mounted lidar and image data collected by the vehicle-mounted camera; The first extraction module is used to put all the laser point clouds in the laser point cloud data into voxels on a pre-constructed target top view, and extract the laser voxel features of the laser point cloud data from the voxels on the target top view. The second extraction module is used to extract the two-dimensional image features of each image in the image data, and project all the two-dimensional image features onto the voxels on the target top view, and extract the visual voxel features of the image data from the voxels on the target top view; The perception module is used to align the laser voxel features and the visual voxel features in time and space, and fuse the aligned laser voxel features and the visual voxel features on the target top view to obtain fused features, and use the fused features to obtain the multimodal fusion perception result of the vehicle.

8. The apparatus according to claim 7, characterized in that, The first extraction module is further configured to: The point cloud data of the laser point cloud is discretized into a uniformly spaced grid in a two-dimensional plane, and the grid size is adjusted according to the density of the point cloud distribution to create one or more columnar sets. Preset features are added to the point cloud of each column set in multiple dimensions, so that each point cloud has multiple dimensional features. Tensor quantization is performed on each point cloud according to the number of non-empty columns in each column set, the number of point clouds in each column set, and the multiple dimensional features of each point cloud, to obtain the tensor quantized point cloud. The quantized point cloud is input into a preset multilayer perceptron to obtain deep point cloud features. Max pooling is then performed on the deep point cloud features to output a feature vector. A feature map is generated based on the feature vector encoding. The laser voxel features of the laser point cloud data are obtained by sequentially traversing all point clouds in all columnar sets and distributing all feature vectors to the corresponding positions of the feature maps.

9. The apparatus according to claim 8, characterized in that, Also includes: The judgment module is used to, before performing tensor quantization on each point cloud based on the number of non-empty pillars in each column set, the number of point clouds in each column set, and the multiple dimensional features of each point cloud to obtain the tensor quantized point cloud, randomly sample multiple data from the column set as the point cloud for tensor quantization processing if the amount of data in the column set is greater than a preset value; otherwise, fill the column set with a preset identifier.

10. The apparatus according to claim 7, characterized in that, The second extraction module is further used for: Extract image features from each image in the image data; Using the image features as an index, a preset depth vector table is queried to obtain the depth vector corresponding to the image features, and the depth vector is projected onto the target overhead view to fuse each image in the image data into the target overhead view.

11. The apparatus according to claim 10, characterized in that, Also includes: The second acquisition module is used to acquire offline image data before querying a preset depth vector table, wherein the offline image data includes image features and multiple pixels; A generation module is used to associate preset data with discrete depth to each pixel and generate a new image based on the image features, wherein the new image includes image features and depth features; The construction module is used to transform the depth features of each new image into the target coordinate system through camera extrinsic parameters, obtain the correspondence between depth vectors and image features, and construct the preset depth vector table based on the correspondence.

12. The apparatus according to claim 7, characterized in that, The sensing module is further used for: The laser voxel features and the visual voxel features are input into a learnable alignment module with three fully connected layers, and the first feature value of the laser voxel features and the second and third feature values ​​of the visual voxel features are output. A similarity matrix is ​​calculated based on the first and second eigenvalues. The coordinate relationship between the vehicle camera and the vehicle lidar is determined using the similarity matrix. Normalization is performed using operators of a preset activation function. The third eigenvalue is weighted and aggregated using attention and a matrix to obtain a fusion matrix. The fusion matrix and the first feature value are concatenated to obtain a concatenated matrix. The second feature value and the third feature value are extracted using the concatenated matrix at a preset level. The features at the preset level are then concatenated with the laser voxel features to obtain the fusion features.

13. The apparatus according to any one of claims 7-12, characterized in that, Also includes: The processing module is used to, before fusing and aligning the laser voxel features and the visual voxel features on the target top view, provide a preset 3D key point to the laser point cloud data, and use preset enhancement parameters to perform geometrically related data augmentation on the laser point cloud data to obtain augmented data, thereby enhancing the data. During fusion, all augmented data are inverted using the preset enhancement parameters to obtain the original coordinates of the preset 3D key point of the laser point cloud data, and the corresponding camera features are located in 2D space using the original coordinates.

14. A vehicle, characterized in that, The vehicle can implement the multimodal fusion perception method for vehicles as described in any one of claims 1-6, including: Vehicle-mounted lidar is used to collect laser point cloud data within the sensing area; Vehicle-mounted cameras are used to collect image data of the environment surrounding the vehicle. The processor is configured to acquire laser point cloud data collected by an onboard LiDAR and image data collected by an onboard camera; place all laser point clouds in the laser point cloud data into voxels on a pre-constructed target overhead view, and extract laser voxel features from the voxels on the target overhead view; extract two-dimensional image features from each image in the image data, and project all two-dimensional image features onto voxels on the target overhead view, and extract visual voxel features from the voxels on the target overhead view; align the laser voxel features and the visual voxel features in time and space, and fuse the aligned laser voxel features and the visual voxel features on the target overhead view to obtain fused features; and use the fused features to obtain the multimodal fusion perception result of the vehicle.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the multimodal fusion perception method for vehicles as described in any one of claims 1-6.

Images