Multi-sensor fusion method, device and equipment based on BEV vision and storage medium

Abstract

The invention discloses a BEV vision-based multi-sensor fusion method, which comprises the following steps of: acquiring perception input data of different sensors, the perception input data comprising 3D geometric structure data from a laser radar and semantic density characteristics from a camera; and unifying the modal features of the plurality of sensors in a sharable aerial view representation space. According to the method, the modal features of the multiple sensors are unified in the shared aerial view representation space to be used for distinguishing perception tasks of different sensors and carrying out learning training, meanwhile, the geometric structure and semantic density are reserved, three-dimensional perception tasks of different sensors can be supported, and the intelligent driving assistance system is improved. Therefore, the precision of fusing BEV features is improved, the accuracy of target detection is improved, higher-dimensional feature presentation is provided for target detection, and a more credible detection result is provided for 3D target detection.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a multi-sensor fusion method, device, equipment and storage medium based on BEV vision. Background Art

[0002] The intelligent driving assistance system (ADAS) of autonomous driving can not only provide assistance to drivers in complex traffic environments and vehicle control processes, but also change the behavioral habits of human driving vehicles over the years. It also promotes the development and progress of the overall society in terms of traffic safety, transportation costs, vehicle efficiency and air pollution. It is a definite development trend of the industry in the future.

[0003] As consumer demand continues to upgrade, the requirements for intelligent driving functions are also getting higher and higher. my country's intelligent driving technology is still in the process of continuous iteration and upgrading. The integrated development of vehicle, road and cloud has become the technical path for the development of China's autonomous driving technology, especially in scenarios such as mines, ports and airports, which have the characteristics of low speed, closed scenes, fixed routes and no passengers, and have strong implementation in the short term.

[0004] Multi-sensor fusion is crucial for accurate and reliable autonomous driving systems. The traditional method is to use camera features to further enhance the LiDAR point cloud level fusion. However, the projection from camera to LiDAR discards the semantic density of camera features, especially in semantic-oriented task fusion scenarios (such as 3D scene segmentation). Therefore, the accuracy of fused features is not high in the process of multi-sensor feature fusion, and the low accuracy of target detection is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the present invention is to provide a multi-sensor fusion method, device, equipment and storage medium based on BEV vision, aiming to solve the above-mentioned technical problems.

[0006] To achieve the above objectives, the present invention provides a multi-sensor fusion method based on BEV vision.

[0007] The multi-sensor fusion method based on BEV vision includes the following steps:

[0008] Acquire perception input data from different sensors, wherein the perception input data includes 3D geometric structure data from a lidar and semantic density features from a camera;

[0009] Unify the modal features of multiple sensors in a shareable bird's-eye view representation space;

[0010] The sparse perception input data of the laser radar is tiled along the height dimension, and the perception input data of the camera is projected back to a ray in the three-dimensional space, so that the perception input data of the sensor can be converted into the bird's-eye view representation space without losing accuracy.

[0011] In one embodiment, the method of unifying the modal features of multiple sensors in a shareable bird's-eye view representation space includes:

[0012] The conversion efficiency of the camera's semantic features to a bird's-eye view representation space is accelerated by discretizing the three-dimensional feature point cloud.

[0013] In one embodiment, the step of accelerating the conversion efficiency of the camera's semantic features to the bird's-eye view representation space by discretizing the three-dimensional feature point cloud is specifically as follows:

[0014] First, the discrete depth distribution of each pixel is predicted;

[0015] Each feature pixel is then dispersed into D discrete points along the camera ray, and the relevant features are scaled according to their corresponding depth probabilities to generate camera feature point cloud data;

[0016] The three-dimensional feature point cloud is discretized with a certain step size.

[0017] In one embodiment, the method of the multi-sensor fusion method based on BEV vision also includes:

[0018] By pre-calculating the 3D coordinates and bird's-eye view grid index of each point;

[0019] Sort all points according to the grid index and record the size of each point;

[0020] Reorder all feature points according to the pre-computed point size order.

[0021] In one embodiment, the method of the multi-sensor fusion method based on BEV vision also includes:

[0022] A separate GPU thread is allocated for each grid and calculates the intervals and writes the results back.

[0023] In one embodiment, in the BEV vision-based multi-sensor fusion method, unified BEV features are fused through a fully convolutional BEV encoder.

[0024] In addition, to achieve the above-mentioned purpose, the present invention also provides a multi-sensor fusion device based on BEV vision, and the multi-sensor fusion device based on BEV vision includes:

[0025] An acquisition module acquires perception input data from different sensors, wherein the perception input data includes 3D geometric structure data from a laser radar and semantic density features from a camera;

[0026] The processing module unifies the modal features of multiple sensors in a shareable bird's-eye view representation space.

[0027] In addition, to achieve the above-mentioned objectives, the present invention also provides a multi-sensor fusion device based on BEV vision, and the multi-sensor fusion device based on BEV vision includes: a memory, a processor, and a fusion program stored in the memory and executable on the processor, and when the fusion program is executed by the processor, the steps of the multi-sensor fusion method based on BEV vision as described above are implemented.

[0028] In addition, to achieve the above object, the present invention also provides a computer-readable storage medium, on which a fusion program is stored, and when the fusion program is executed by a processor, the steps of the data storage method described above are implemented.

[0029] Beneficial effects that can be achieved by the present invention: A multi-sensor fusion method based on BEV vision proposed in an embodiment of the present invention unifies the modal features of multiple sensors in a shared bird's-eye view representation space, which is used to distinguish the perception tasks of different sensors and conduct learning and training, while retaining the geometric structure and semantic density, and can support the three-dimensional perception tasks of different sensors, thereby improving the intelligent driving assistance system. The accuracy of the fused BEV features is improved, the accuracy of target detection is improved, a higher-dimensional feature presentation is provided for target detection, and a more reliable detection result is provided for 3D target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the structure of a device in a hardware operating environment involved in an embodiment of the present invention;

[0031] Figure 2 It is a flowchart of the first embodiment of the multi-sensor fusion method based on BEV vision of the present invention;

[0032] Figure 3 is a framework diagram of multi-sensor fusion based on BEV vision in an embodiment of the present invention;

[0033] Figure 4 is a schematic diagram of camera to BEV space conversion in an embodiment of the present invention;

[0034] Figure 5 Schematic diagram of improving BEV pooling efficiency in an embodiment of the present invention.

[0035] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0036] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0037] like Figure 1 As shown, Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention.

[0038] The terminal of the embodiment of the present invention can be a PC, or it can be a smart phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a portable computer, or other portable terminal devices with display function.

[0039] like Figure 1 As shown, the terminal may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0040] Optionally, the terminal may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; of course, the mobile terminal may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be repeated here.

[0041] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0042] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module and a fusion program.

[0043] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and perform data communication with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and perform data communication with the client; and the processor 1001 can be used to call the fusion program stored in the memory 1005 and perform the following operations:

[0044] Obtain sensory input data from different sensors;

[0045] Unify the modal features of multiple sensors in a shareable bird's-eye view representation space.

[0046] Further, the processor 1001 may call the fusion program stored in the memory 1005, and further perform the following operations:

[0047] The conversion efficiency of the camera's semantic features to a bird's-eye view representation space is accelerated by discretizing the three-dimensional feature point cloud.

[0048] Further, the processor 1001 may call the fusion program stored in the memory 1005, and further perform the following operations:

[0049] First, the discrete depth distribution of each pixel is predicted;

[0050] Each feature pixel is then dispersed into D discrete points along the camera ray, and the relevant features are scaled according to their corresponding depth probabilities to generate camera feature point cloud data;

[0051] The three-dimensional feature point cloud is discretized with a certain step size.

[0052] Further, the processor 1001 may call the fusion program stored in the memory 1005, and further perform the following operations:

[0053] By pre-calculating the 3D coordinates and bird's-eye view grid index of each point;

[0054] Sort all points according to the grid index and record the size of each point;

[0055] Reorder all feature points according to the pre-computed point size order.

[0056] Further, the processor 1001 may call the fusion program stored in the memory 1005, and further perform the following operations:

[0057] A separate GPU thread is allocated for each grid and calculates the intervals and writes the results back.

[0058] The specific embodiments of the data storage device of the present invention are basically the same as the embodiments of the data storage method described below, and will not be described in detail here.

[0059] Reference Figure 2-3 The first embodiment of the present invention provides a multi-sensor fusion method based on BEV vision, and the multi-sensor fusion method based on BEV vision includes:

[0060] Step S10, obtaining perception input data from different sensors;

[0061] In this embodiment, the perception input data includes 3D geometric structure data from a laser radar and semantic density features from a camera;

[0062] Step S20, unifying the modal features of multiple sensors in a shareable bird's-eye view representation space;

[0063] The sparse perception input data of the laser radar is tiled along the height dimension, and the perception input data of the camera is projected back to a ray in the three-dimensional space, so that the perception input data of the sensor can be converted into the bird's-eye view representation space without losing accuracy.

[0064] In this embodiment, the modal features of multiple sensors are unified in a shared bird's-eye view representation space to distinguish the perception tasks of different sensors and conduct learning and training, while retaining the geometric structure and semantic density, which can support the three-dimensional perception tasks of different sensors and improve the intelligent driving assistance system. This improves the accuracy of the fused BEV features, improves the accuracy of target detection, provides a higher-dimensional feature presentation for target detection, and provides more reliable detection results for 3D target detection.

[0065] The sensor may include: at least one of a CenterPoint detection head, a PointPillars detection head, and a TransFusion-L detection head.

[0066] For the perception inputs of different sensors, the modal features of multiple sensors are unified in a shared bird's-eye view representation space, and specific modal encoding techniques are applied to extract their complete features, so that the perception results of all sensors can be converted to the bird's-eye view representation space without losing accuracy. Specifically, the conversion to the BEV representation space retains the 3D geometric structure from the lidar and the semantic density features from the camera. The projection of LiDAR to BEV tiles the sparse LiDAR features along the height dimension, so there is no geometric distortion. On the other hand, the projection of the camera to BEV projects each camera feature pixel back to a ray in the three-dimensional space, which can retain all the semantic information of the camera. Since the output space is also in BEV, the representation of the BEV space is also suitable for different perception tasks.

[0067] For further information, please refer to Figure 4 First, the discrete depth distribution of each pixel is predicted, and then each feature pixel is dispersed into D discrete points along the camera light, and the relevant features are scaled according to their corresponding depth probabilities. In this way, a camera feature point cloud data can be generated, with N cameras and a feature map size of (H, W). This three-dimensional feature point cloud is further discretized along x and y with a step size of r = 0.5m. Finally, the BEV pooling operation is used to summarize all the features in each r*r grid, and flatten and compress them along the z axis, so as to accelerate the conversion efficiency of the camera's semantic features to the BEV space.

[0068] The first step of BEV pooling is to associate each point in the camera feature point cloud with the BEV grid. Unlike the lidar point cloud, the coordinates of the camera feature point cloud are fixed as long as the camera's intrinsic and extrinsic parameters remain unchanged. Therefore, in order to improve the efficiency of BEV pooling, this application optimizes it by pre-calculation and interval reduction.

[0069] Specifically, please refer to Figure 5, the application pre-calculates the 3D coordinates and BEV grid index of each point, sorts all points according to the grid index, and records the size of each point. During the detection reasoning process, it is only necessary to re-sort all feature points according to the pre-calculated point size order. This caching mechanism can greatly reduce the delay time of grid association.

[0070] After grid association, all points within the same BEV grid are continuous in the tensor representation. The next step of BEV pooling is to aggregate the features within each BEV grid through symmetric functions such as mean, maximum and cumulative sum. Then a GPU thread is assigned to each grid separately to calculate the interval and write the result back. The dependency between the outputs is eliminated, multi-level tree reduction is not required, and partial sums are avoided from being written to the cache (DRAM), thereby greatly reducing the latency of feature aggregation. Therefore, after two steps of pre-calculation and interval reduction, the conversion efficiency of the camera's semantic features to the BEV space can be greatly improved.

[0071] In addition, in this application, a fully convolutional BEV encoder is used to fuse unified BEV features to support different target recognition and segmentation tasks and adapt to different perception scenarios. After all perception features are converted into a shared BEV space representation, the features can be fused using element operators such as union. Although in the same BEV space, due to the inaccurate depth of view conversion, the LiDAR BEV features and the camera BEV features may still have local misalignment of features to a certain extent. Therefore, the present invention proposes a convolution-based BEV encoder to compensate for this local misalignment, thereby supporting different target recognition and segmentation tasks and adapting to different perception scenarios of multiple sensors.

[0072] Finally, the effectiveness of the multi-sensor fusion method based on BEV vision can be verified. The effectiveness of the multi-sensor fusion method based on BEV vision proposed in the present invention can be verified by using two example scenarios: 3D target detection and BEV map segmentation for the data set.

[0073] In addition, an embodiment of the present invention further proposes a data calculation device based on business type, and the data calculation device based on business type includes:

[0074] An acquisition module acquires perception input data from different sensors, wherein the perception input data includes 3D geometric structure data from a laser radar and semantic density features from a camera;

[0075] The processing module unifies the modal features of multiple sensors in a shareable bird's-eye view representation space.

[0076] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0077] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a fusion program is stored. When the fusion program is executed by a processor, the following operations are implemented:

[0078] Obtain sensory input data from different sensors;

[0079] Unify the modal features of multiple sensors in a shareable bird's-eye view representation space.

[0080] Furthermore, when the fusion program is executed by the processor, the following operations are also implemented:

[0081] The conversion efficiency of the camera's semantic features to a bird's-eye view representation space is accelerated by discretizing the three-dimensional feature point cloud.

[0082] Furthermore, when the fusion program is executed by the processor, the following operations are also implemented:

[0083] First, the discrete depth distribution of each pixel is predicted;

[0084] Each feature pixel is then dispersed into D discrete points along the camera ray, and the relevant features are scaled according to their corresponding depth probabilities to generate camera feature point cloud data;

[0085] The three-dimensional feature point cloud is discretized with a certain step size.

[0086] Furthermore, when the fusion program is executed by the processor, the following operations are also implemented:

[0087] By pre-calculating the 3D coordinates and bird's-eye view grid index of each point;

[0088] Sort all points according to the grid index and record the size of each point;

[0089] Reorder all feature points according to the pre-computed point size order.

[0090] Furthermore, when the fusion program is executed by the processor, the following operations are also implemented:

[0091] A separate GPU thread is allocated for each grid and calculates the intervals and writes the results back.

[0092] The specific embodiments of the computer-readable storage medium of the present invention are basically the same as the embodiments of the above-mentioned application software security vulnerability detection method, and will not be described in detail here.

[0093] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0094] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0096] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-sensor fusion method based on BEV vision, characterized in that: The multi-sensor fusion method based on BEV vision includes the following steps: Acquire perception input data from different sensors, wherein the perception input data includes 3D geometric structure data from a lidar and semantic density features from a camera; Unify the modal features of multiple sensors in a shareable bird's-eye view representation space; The sparse perception input data of the lidar is tiled along the height dimension, and the perception input data of the camera is projected back to a ray in the three-dimensional space, so that the perception input data of the sensor can be converted into a bird's-eye view representation space without losing accuracy.

2. The multi-sensor fusion method based on BEV vision as claimed in claim 1, characterized in that: The method of unifying the modal features of multiple sensors in a shareable bird's-eye view representation space includes: The conversion efficiency of the camera's semantic features to a bird's-eye view representation space is accelerated by discretizing the three-dimensional feature point cloud.

3. The multi-sensor fusion method based on BEV vision as claimed in claim 2, characterized in that: The steps of accelerating the conversion efficiency of the camera's semantic features into a bird's-eye view representation space by discretizing the three-dimensional feature point cloud are specifically as follows: First, the discrete depth distribution of each pixel is predicted; Each feature pixel is then dispersed into D discrete points along the camera ray, and the relevant features are scaled according to their corresponding depth probabilities to generate camera feature point cloud data; The three-dimensional feature point cloud is discretized with a certain step size.

4. The multi-sensor fusion method based on BEV vision according to claim 3 is characterized in that: The method of the multi-sensor fusion method based on BEV vision also includes: By pre-calculating the 3D coordinates and bird's-eye view grid index of each point; Sort all points according to the grid index and record the size of each point; Reorder all feature points according to the pre-computed point size order.

5. The multi-sensor fusion method based on BEV vision according to claim 4 is characterized in that: The method of the multi-sensor fusion method based on BEV vision also includes: A separate GPU thread is allocated for each grid and calculates the intervals and writes the results back.

6. The multi-sensor fusion method based on BEV vision according to claim 4 is characterized in that: In the method of the multi-sensor fusion method based on BEV vision, unified BEV features are fused through a fully convolutional BEV encoder.

7. A multi-sensor fusion device based on BEV vision, characterized in that: The multi-sensor fusion device based on BEV vision includes: An acquisition module acquires perception input data from different sensors, wherein the perception input data includes 3D geometric structure data from a laser radar and semantic density features from a camera; The processing module unifies the modal features of multiple sensors in a shareable bird's-eye view representation space.

8. A multi-sensor fusion device based on BEV vision, characterized in that: The multi-sensor fusion device based on BEV vision includes: a memory, a processor, and a fusion program stored in the memory and executable on the processor. When the fusion program is executed by the processor, the steps of the multi-sensor fusion method based on BEV vision are implemented as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a fusion program, and when the fusion program is executed by the processor, the steps of the multi-sensor fusion method based on BEV vision as described in any one of claims 1 to 6 are implemented.

Images