Perception fusion system, electronic device and storage medium
By converting the environmental perception data of unmanned vehicles into bird's-eye view space and integrating it, the algorithm complexity and accuracy problems caused by the independent processing of perception modules in the field of unmanned driving are solved, and the accuracy of perception tasks and process simplification are improved.
Patent Information
- Application Number
- PCT/CN2023/141394
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-16
- Filing Date
- 2023-12-25
- Publication Date
- 2024-11-21
AI Technical Summary
In the field of unmanned driving, in the prior art, camera algorithm, lidar algorithm and millimeter wave radar algorithm are processed separately by different perception modules, resulting in complex fusion algorithms and inaccurate outputs, which reduces the accuracy of subsequent perception tasks .
By converting different types of environmental perception data into a bird's-eye view space, using the bird's-eye view fusion features and task heads to perform perception tasks, the effective fusion of different sensors' perceptual data is achieved, and the process of perception fusion algorithm is simplified.
It improves the accuracy of perceived task results, and simplifies the process of perceived fusion algorithm, realizing the effective fusion of different sensor data.
Smart Images

Figure CN2023141394_21112024_PF_FP_ABST
Abstract
Description
Perception fusion system, electronic device and storage medium
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 16, 2023, with application number 202310553045.0. The entire contents of the above application are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of artificial intelligence technology, for example, to a perception fusion system, an electronic device, and a storage medium. Background Art
[0003] In the field of autonomous driving, the behavioral control of unmanned vehicles relies on the coordinated efforts of onboard sensing devices such as cameras, lidar, and millimeter-wave radar. However, camera algorithms, lidar algorithms, and millimeter-wave radar algorithms are each processed independently by different sensing modules. Simply fusing the outputs of these modules at the result level results in a complex fusion algorithm and inaccurate perception fusion output, reducing the accuracy of subsequent perception tasks.
[0004] Summary of the Invention
[0005] The present application provides a perception fusion system, electronic device and storage medium, which realizes the effective fusion of perception data from different sensors by converting different types of environmental perception data into a bird's-eye view space, and then uses the bird's-eye view fusion features and task head to perform perception tasks, thereby improving the accuracy of perception task results.
[0006] The present application provides a perception fusion system, which includes:
[0007] A data acquisition module is configured to acquire environmental perception data of the target vehicle, the environmental perception data including at least: surround view image data, lidar point cloud data, and millimeter wave radar point cloud data;
[0008] A data encoding module configured to process surround view image data, lidar point cloud data, and millimeter wave radar point cloud data according to different preset encoders to obtain corresponding bird's-eye view features;
[0009] The feature fusion module is configured to process each bird's-eye view feature according to a preset bird's-eye view encoder to obtain a bird's-eye view fusion feature, and generate a task result based on the bird's-eye view fusion feature and the task head.
[0010] The present application provides an electronic device, comprising:
[0011] at least one processor; and
[0012] a memory communicatively coupled to the at least one processor;
[0013] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can implement the perception fusion system described in any embodiment of the present application.
[0014] The present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the perception fusion system described in any embodiment of the present application is implemented.
[0015] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The following is a brief introduction to the drawings required for describing the embodiments.
[0017] FIG1 is a schematic diagram of a perception fusion system provided according to Embodiment 1 of the present application;
[0018] FIG2 is a schematic diagram of a perception fusion system provided according to Embodiment 2 of the present application;
[0019] FIG3 is a schematic diagram of a perception fusion system provided according to Embodiment 3 of the present application;
[0020] FIG4 is a schematic diagram of a framework of a perception fusion system provided according to Embodiment 3 of the present application;
[0021] FIG5 is a schematic diagram of sparse image feature extraction according to the third embodiment of the present application;
[0022] FIG6 is a schematic diagram of a geometric transformation based on line-of-sight projection according to the third embodiment of the present application;
[0023] FIG7 is a schematic diagram of a perception task processing according to the third embodiment of the present application;
[0024] FIG8 is a schematic diagram of the structure of an electronic device that implements the perception fusion system of an embodiment of the present application. DETAILED DESCRIPTION
[0025] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application, and the described embodiments serve as part of the embodiments of the present application.
[0026] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, in addition to the process, method, system, product or equipment comprising a series of steps or units shown in the embodiments of the present application, other processes, methods, systems, products or equipment of this series of steps or units not clearly listed may also be included, or other steps or units inherent to these processes, methods, systems, products or equipment.
[0027] Example 1
[0028] Figure 1 is a schematic diagram of a perception fusion system provided in Example 1 of this application. This embodiment is applicable to the case of fusing environmental perception data. As shown in Figure 1, the perception fusion system provided in Example 1 includes a data acquisition module 10, a data encoding module 20, and a feature fusion module 30.
[0029] The data acquisition module 10 is configured to acquire environmental perception data of the target vehicle, and the environmental perception data includes at least: surround view image data, laser radar point cloud data, and millimeter wave radar point cloud data.
[0030] Environmental perception data may refer to perception data collected by the target vehicle using sensors such as surround-view cameras, Light Detection and Ranging (LIDAR) (such as laser radar), and Radio Detection and Ranging (RADAR) (such as millimeter-wave radar). Environmental perception data may at least include: surround-view image data, lidar point cloud data, and millimeter-wave radar point cloud data. Surround-view image data may refer to image data collected by the surround-view camera installed on the target vehicle. Surround-view image data may include traffic light data and traffic sign data collected by the surround-view camera. Lidar point cloud data may refer to point cloud data collected by the laser radar (LIDAR) installed on the target vehicle. Millimeter-wave radar point cloud data may refer to point cloud data collected by the millimeter-wave radar (RADAR) installed on the target vehicle.
[0031] In the embodiment of the present application, the data acquisition module 10 can acquire environmental perception data based on the target vehicle, including, but not limited to, surround view image data collected by one or more surround view cameras installed on the target vehicle, lidar point cloud data collected by a lidar installed on the target vehicle, and millimeter wave radar point cloud data collected by a millimeter wave radar installed on the target vehicle. The above environmental perception data is merely exemplary and, in actual applications, may also include Global Positioning System (GPS) data, inertial measurement unit (IMU) data, and vehicle communication data, etc., which are not limited in the embodiment of the present application.
[0032] The data encoding module 20 is configured to process the surround view image data, the lidar point cloud data, and the millimeter wave radar point cloud data according to different preset encoders to obtain corresponding bird's-eye view features.
[0033] The preset encoder can be understood as an encoder for generating corresponding bird's-eye view features based on surround view image data, lidar point cloud data, and millimeter-wave radar point cloud data. The preset encoder can be composed of common convolutional neural networks, for example, it can include a 50-layer residual network (Residual Network-50, ResNet50) neural network and a 3D convolutional neural network. The bird's-eye view features can refer to features extracted from environmental perception data and converted to a 3D bird's-eye view space, where a bird's eye view (BEV) is a perspective or coordinate system (three-dimensional (3D) space) used to describe the perceived real world. Converting environmental perception data to a 3D bird's-eye view space can effectively preserve the semantic density of the surround view image data and the geometric structure of the lidar point cloud data and millimeter-wave radar point cloud data, and can provide a unified representation for the effective fusion of environmental perception data.
[0034] In an embodiment of the present application, the data encoding module 20 can call a pre-configured preset encoder to perform feature extraction on the acquired surround view image data, lidar point cloud data, and millimeter wave radar point cloud data respectively. For example, a ResNet50 neural network, a 3D convolutional neural network, or other types of neural network models can be used to extract bird's-eye view features from the above-mentioned environmental perception data. The extracted features are bird's-eye view features converted to a bird's-eye view 3D bird's-eye view space. Selecting the bird's-eye view as a unified representation for subsequent feature fusion can effectively preserve the semantic density of the surround view image data and the geometric structure of the lidar point cloud data and the millimeter wave radar point cloud data, providing an important basis for the execution of subsequent perception tasks.
[0035] The feature fusion module 30 is configured to process each bird's-eye view feature according to a preset bird's-eye view encoder to obtain a bird's-eye view fusion feature, and generate a task result according to the bird's-eye view fusion feature and the task header.
[0036] The preset bird's-eye view encoder may refer to a pre-configured encoder for performing feature fusion on the bird's-eye view features corresponding to the surround view image data, the lidar point cloud data, and the millimeter-wave radar point cloud data. The preset bird's-eye view encoder may be composed of a convolutional neural network or other types of neural network models, which is not limited in this embodiment of the present application. The bird's-eye view fusion feature may refer to the fusion feature obtained by fusing the bird's-eye view features corresponding to the surround view image data, the lidar point cloud data, and the millimeter-wave radar point cloud data. The task head may be understood as a functional unit for performing a specific perception task. The task head may include a 3D detection head, a map segmentation head, and a trajectory prediction head. The task head may be composed of a convolutional neural network or other types of neural network models. The task result may refer to the perception task result output by the task head. For example, the task result may include: 3D target detection results output by the 3D detection head, map segmentation results output by the map segmentation head, and target trajectory prediction results output by the trajectory prediction head.
[0037] In an embodiment of the present application, the feature fusion module 30 can call a pre-configured preset bird's-eye view encoder to perform feature fusion processing on the bird's-eye view corresponding to the extracted surround view image data, lidar point cloud data, and millimeter-wave radar point cloud data, and then input the obtained bird's-eye view fusion features into the corresponding task head according to actual needs, and the corresponding task head outputs the corresponding task result. Among them, the preset bird's-eye view encoder and each task head can be composed of a convolutional neural network or other types of neural network models. The task head can include but is not limited to a 3D detection head, a map segmentation head, and a trajectory prediction head. Accordingly, the task result can include but is not limited to 3D target detection results, map segmentation results, and target trajectory prediction results. The task head can be additionally configured according to actual needs. For example, the task head can also include a lane detection head, a traffic sign detection head, and a real-time positioning head, etc. This embodiment of the present application does not limit this.
[0038] The perception fusion system proposed in the embodiment of the present application is configured to acquire the environmental perception data of the target vehicle through a data acquisition module, wherein the environmental perception data includes at least: surround view image data, lidar point cloud data, and millimeter wave radar point cloud data; a data encoding module is configured to process the surround view image data, the lidar point cloud data, and the millimeter wave radar point cloud data according to different preset encoders to obtain corresponding bird's-eye view features; a feature fusion module is configured to process each of the bird's-eye view features according to a preset bird's-eye view encoder to obtain a bird's-eye view fusion feature, and generate a task result based on the bird's-eye view fusion feature and the task head. The perception fusion system proposed in the embodiment of the present application realizes the effective fusion of perception data from different sensors by converting different types of environmental perception data into a bird's-eye view space, and then using the bird's-eye view fusion feature and the task head to perform the perception task, thereby improving the accuracy of the perception task results and simplifying the process of the perception fusion algorithm.
[0039] Example 2
[0040] Figure 2 is a schematic diagram of a perception fusion system provided in Example 2 of the present application, which is optimized and expanded based on the above-mentioned embodiment and can be combined with the various optional technical solutions in the above-mentioned embodiment. As shown in Figure 2, a perception fusion system provided in Example 2 is a refinement of the data encoding module 20 and the feature fusion module 30, wherein the data encoding module 20 includes a first bird's-eye view feature extraction unit 21, a second bird's-eye view feature extraction unit 22, and a third bird's-eye view feature extraction unit 23, and the feature fusion module 30 includes a bird's-eye view feature fusion unit 31 and a task result acquisition unit 32. The perception fusion system includes:
[0041] The data acquisition module 10 is configured to acquire environmental perception data of the target vehicle, and the environmental perception data includes at least: surround view image data, laser radar point cloud data, and millimeter wave radar point cloud data.
[0042] The first bird's-eye view feature extraction unit 21 in the data encoding module 20 is configured to call the ResNet50 neural network of the first preset encoder to extract image features of at least one surround view image data, and horizontally splice the image features to obtain an image feature matrix, serialize the image feature matrix into an image feature sequence, input the feature sequence to the compression encoder of the first preset encoder for compression processing to obtain image sparse features, and convert the image sparse features into a 3D bird's-eye view space as the first bird's-eye view feature.
[0043] In an embodiment of the present application, the first preset encoder can be understood as an encoder for generating a first bird's-eye view feature based on the surround view image data. The first preset encoder can include a ResNet50 neural network and a compression encoder, among others. The ResNet50 neural network refers to a convolutional neural network that introduces the concept of residual learning, and the compression encoder can adopt a neural network model with a multi-layer self-attention mechanism, the number of which can be configured accordingly according to actual needs. The first bird's-eye view feature can refer to a bird's-eye view feature extracted from the surround view image data and converted to a 3D bird's-eye view space.
[0044] Exemplarily, one or more surround-view cameras installed on the target vehicle can be controlled to collect surround-view image data around the vehicle. The surround-view image data may include but is not limited to traffic light data and traffic sign data. The ResNet50 neural network of the first preset encoder is called to extract features from the above-mentioned surround-view image data, and the extracted image features are horizontally spliced to obtain an image feature matrix. Then, a serialization algorithm such as a convolutional neural network is called to process the image feature matrix into an image feature sequence. The processed image feature sequence is then input into a compression encoder of the first preset encoder for compression processing to obtain image sparse features corresponding to the surround-view image data. The compression encoder can adopt a neural network model with a multi-layer self-attention mechanism, and the number of layers can be configured accordingly according to actual needs. The perspective converter is used to convert the image sparse features from 2D space to 3D bird's-eye view space, thereby obtaining the first bird's-eye view features corresponding to the surround-view image data.
[0045] Optionally, in the first bird's-eye view feature extraction unit 21, the feature sequence obtained after processing is input into the compression encoder for compression processing to obtain image sparse features, which may include the following steps: calling a preset quantization scorer to determine the dependency score of each image feature in the feature sequence; eliminating image features in the feature sequence whose dependency score is lower than a preset filtering threshold as image sparse features.
[0046] In an embodiment of the present application, a preset quantitative scorer may refer to a pre-configured scorer for determining the dependency score of an image feature, and the sparse image features in each image feature can be extracted using the preset quantitative scorer. The dependency score can be understood as the attention score or weight corresponding to each image feature. The preset filtering threshold may refer to a filtering threshold pre-configured according to the dependency score corresponding to each image feature, and a preset filtering threshold may be comprehensively configured based on the dependency score corresponding to each image feature. The same number of preset filtering thresholds may also be configured based on the dependency score corresponding to each image feature, that is, one image feature corresponds to one preset filtering threshold, and the embodiment of the present application does not limit this. Image sparse features may refer to image features obtained after compression processing of image features.
[0047] Exemplarily, after obtaining the image feature sequence corresponding to the surround view image data, a pre-configured preset quantitative scorer can be called to determine the dependency score of each image feature in the feature sequence. The dependency score can refer to the attention score or weight corresponding to each image feature. Then, according to the pre-configured preset filtering threshold and the dependency score corresponding to each image feature, each image feature in the feature sequence is screened, and the image features in the feature sequence whose dependency score is lower than the preset filtering threshold are eliminated, and the screened image features are used as image sparse features. Among them, the number of preset filtering thresholds can be one or more, that is, a preset filtering threshold can be comprehensively configured according to the dependency scores corresponding to each image feature, or the corresponding preset filtering thresholds can be configured separately according to the dependency scores corresponding to each image feature. The embodiment of the present application does not limit this.
[0048] The second bird's-eye view feature extraction unit 22 in the data encoding module 20 is configured to perform gridding processing on the lidar point cloud data based on non-deterministic voxelization to obtain laser point cloud voxels, and then input the laser point cloud voxels into a second preset encoder composed of a 3D convolutional neural network to obtain a second bird's-eye view feature.
[0049] In the embodiments of the present application, a laser point cloud voxel may refer to a three-dimensional point cloud voxel obtained by performing voxel rasterization processing on the laser radar point cloud data. Since the number of spatial points in the laser radar point cloud data is very large, using voxel rasterization to obtain laser point cloud voxels can improve the efficiency of point cloud data processing. The second preset encoder can be understood as an encoder for generating a second bird's-eye view feature based on the laser radar point cloud data. The second preset encoder can be composed of a 3D convolutional neural network. The second bird's-eye view feature can refer to a bird's-eye view feature extracted from the laser radar point cloud data.
[0050] Exemplarily, the laser radar installed on the target vehicle can be controlled to collect laser radar point cloud data around the vehicle. In order to improve the processing speed of the subsequent laser radar point cloud data, the acquired laser radar point cloud data can be subjected to gridding processing based on non-deterministic voxelization to obtain corresponding laser point cloud voxels, wherein the gridding processing based on non-deterministic voxelization means that in the gridding processing process of the laser radar point cloud data, not all laser radar point cloud data are processed uniformly, but selective voxelization processing is performed according to the importance of each point cloud position in the laser radar point cloud data. After the laser point cloud voxels are obtained, they can be input into a second preset encoder composed of a 3D convolutional neural network to extract bird's-eye view features to obtain second bird's-eye view features corresponding to the laser radar point cloud data.
[0051] The third bird's-eye view feature extraction unit 23 in the data encoding module 20 is configured to perform deterministic gridding processing on the millimeter-wave radar point cloud data, and then input the processed millimeter-wave radar point cloud data into a third preset encoder composed of a 3D convolutional neural network to obtain a third bird's-eye view feature.
[0052] In this embodiment of the present application, the third preset encoder can be understood as an encoder for generating a third bird's-eye view feature based on millimeter-wave radar point cloud data. The third preset encoder can be composed of a 3D convolutional neural network. The third bird's-eye view feature can refer to a bird's-eye view feature extracted from the millimeter-wave radar point cloud data.
[0053] Exemplarily, the millimeter-wave radar installed on the target vehicle can be controlled to collect millimeter-wave radar point cloud data around the vehicle, and then the acquired millimeter-wave radar point cloud data can be subjected to deterministic voxelization and gridding processing, wherein the deterministic voxelization-based gridding processing refers to uniformly processing all millimeter-wave radar point cloud data during the gridding processing of the millimeter-wave radar point cloud data, so that the lidar point cloud data and the millimeter-wave radar point cloud data have consistent representation, and the processed millimeter-wave radar point cloud data is input into a third preset encoder composed of a 3D convolutional neural network for bird's-eye view feature extraction to obtain a third bird's-eye view feature corresponding to the millimeter-wave radar point cloud data.
[0054] Optionally, the first bird's-eye view feature, the second bird's-eye view feature, and the third bird's-eye view feature are stored in a gridded data matrix and have the same scale in the spatial dimension.
[0055] The bird's-eye view feature fusion unit 31 in the feature fusion module 30 is configured to horizontally splice the bird's-eye view features and then input them into a preset bird's-eye view encoder composed of a convolutional neural network to obtain a bird's-eye view fusion feature.
[0056] For example, after obtaining the first bird's-eye view feature, the second bird's-eye view feature and the third bird's-eye view feature corresponding to the surround view image data, the lidar point cloud data and the millimeter-wave radar point cloud data respectively, they can be concatenated horizontally and input into a pre-configured preset bird's-eye view encoder for feature fusion, thereby obtaining the bird's-eye view fusion feature.
[0057] The task result acquisition unit 32 in the feature fusion module 30 is configured to input the bird's-eye view fusion features into each task head composed of a convolutional neural network to perform the perception task and output the corresponding task result.
[0058] In this embodiment of the present application, a task head may include at least one of the following: a 3D detection head, a map segmentation head, a traffic sign detection head, a traffic light detection head, a real-time positioning head, and a trajectory prediction head. The 3D detection head and the map segmentation head are the basic task heads. Task results may include at least one of the following: 3D target detection results, map segmentation results, traffic sign detection results, traffic light status detection results, real-time positioning results, and target trajectory prediction results.
[0059] Exemplarily, according to the needs of the actual perception task, the acquired bird's-eye view fusion features are input into the task head composed of a convolutional neural network, and the corresponding task results are output, wherein the task head may include at least one of the following: 3D detection head, map segmentation head, traffic sign detection head, traffic light detection head, real-time positioning head, trajectory prediction head. Accordingly, the task results corresponding to the perception task may include 3D target detection results, map segmentation results, traffic sign detection results, traffic light status detection results, real-time positioning results, and target trajectory prediction results, etc.
[0060] Optionally, the task result acquisition unit 32 is configured to be at least one of the following:
[0061] The bird's-eye view fusion features are input into the 3D detection head in the task head, and the output task result is the 3D target information detection box;
[0062] The bird's-eye view fusion features are input into the map segmentation head in the task head, and the output task result is a real-time map element mask;
[0063] Input the 3D target information detection frame into the traffic sign detection head in the task head, and the output task result is the traffic sign information;
[0064] The 3D target information detection frame is input into the traffic light detection head in the task head, and the output task result is the traffic light status information;
[0065] The real-time map element mask is input into the real-time positioning head in the task head, and the output task result is the geographic location vector;
[0066] The 3D target information detection box, real-time map element mask, traffic sign information, traffic light status information and geographic location vector are input into the trajectory prediction head in the task head, and the output task result is the trajectory prediction information of the target object.
[0067] In a specific embodiment, if the perception task is pedestrian trajectory prediction, the bird's-eye view fusion features can be first input into the 3D detection head and the map segmentation head respectively to obtain the corresponding 3D target information detection frame and real-time map element mask, and then the 3D target information detection frame can be input into the traffic sign detection head and the traffic light detection head respectively to obtain the corresponding traffic sign information and traffic light status information, and the real-time map element mask can be input into the real-time positioning head to obtain the corresponding geographic location vector, and the 3D target information detection frame, real-time map element mask, traffic sign information, traffic light status information and geographic location vector can be input into the trajectory prediction head to output the pedestrian trajectory prediction information.
[0068] Optionally, the system further includes: an image conversion module configured to convert a 2D input frame into a 3D target information detection frame using a geometric transformation, where the geometric transformation is obtained by a parameter matrix of the relationship between the camera and the point cloud.
[0069] In an embodiment of the present application, since surround view image data such as traffic light data and traffic sign data is 2D, it is necessary to use geometric transformation to convert the 2D image data into 3D bird's-eye view features. This can not only retain complete semantic information, but also help surround view image data, lidar point cloud data and millimeter wave radar point cloud data to use a bird's-eye view as a unified representation for fusion.
[0070] The perception fusion system proposed in the embodiment of the present application is configured to obtain environmental perception data of the target vehicle through a data acquisition module, and the environmental perception data includes at least: surround view image data, lidar point cloud data, and millimeter wave radar point cloud data; the first bird's-eye view feature extraction unit, the second bird's-eye view feature extraction unit, and the third bird's-eye view feature extraction unit in the data acquisition module respectively obtain bird's-eye view features corresponding to the surround view image data, lidar point cloud data, and millimeter wave radar point cloud data; the bird's-eye view feature fusion unit in the feature fusion module is configured to horizontally splice each bird's-eye view feature and input it into a preset bird's-eye view encoder composed of a convolutional neural network to obtain a bird's-eye view fusion feature; the task result acquisition unit in the feature fusion module is configured to input the bird's-eye view fusion feature into each task head composed of a convolutional neural network to perform a perception task and output the corresponding task result. The perception fusion system proposed in the embodiment of the present application obtains bird's-eye view features corresponding to surround view image data, lidar point cloud data and millimeter-wave radar point cloud data through the first bird's-eye view feature extraction unit, the second bird's-eye view feature extraction unit and the third bird's-eye view feature extraction unit respectively, and then fuses the above bird's-eye view features through the bird's-eye view feature fusion unit. Finally, the task result acquisition unit executes the corresponding perception task and outputs the corresponding task result, thereby realizing the effective fusion of perception data from different sensors, improving the accuracy of perception task results, and simplifying the process of the perception fusion algorithm.
[0071] Example 3
[0072] Figure 3 is a schematic diagram of a perception fusion system provided in Example 3 of the present application, which is optimized and expanded based on the above-mentioned embodiment and can be combined with various optional technical solutions in the above-mentioned embodiment. As shown in Figure 3, a perception fusion system provided in Example 3 of the present application includes: an image backbone network 51, a laser point cloud backbone network 52, a radar point cloud backbone network 53, a feature fusion network 54, and a task head 55. Figure 4 is a schematic diagram of the framework of a perception fusion system provided in Example 3 of the present application, which is a refinement of the perception fusion system of Figure 3. The perception fusion system includes:
[0073] The image backbone network 51 takes as input surround view image data captured by multiple surround view cameras installed around the target vehicle and outputs visual bird's-eye view features. The captured surround view image data is processed by an image encoder to obtain depth image features. These image features are then input to a compression encoder for compression processing to obtain compressed image features. The compression encoder can utilize an efficient network with a sparse self-attention mechanism. Specifically, the architecture is a neural network model employing a multi-layer self-attention mechanism. The specific number of layers depends on deployment requirements and is not specifically limited in this embodiment. Exemplarily, as shown in FIG5 , the image encoder uses the convolutional neural network Resnet50 as an image feature extractor, and sequentially inputs six surround view image data into the image feature extractor. The six input image features are processed using image serialization to obtain serialized image features. The serialization process involves horizontally splicing the input image features to obtain a spliced image feature matrix, and then using a serialization algorithm to convert the feature matrix into a feature sequence. The feature sequence is scored by a quantization scorer to obtain a dependency score for each sequence feature. A filtering threshold is pre-set according to the filtering algorithm, and sequence features with dependency scores lower than the threshold are deleted to obtain sparse image features. The sparse image features are finally converted from 2D space to 3D bird's-eye view space through a camera-bird's-eye view conversion, thereby obtaining the bird's-eye view features corresponding to the surround view image data. The geometric transformation method based on line-of-sight projection, as shown in Figure 6, can be used to convert a 2D input frame into a 3D target information detection frame. The geometric transformation is obtained by the relationship parameter matrix between the camera and the point cloud. The geometric transformation calculates the position of any pixel point in 3D space in advance based on the physical characteristics of light propagation, thereby realizing 2D to 3D projection. In addition, during the conversion process, the thickness of targets such as traffic lights and traffic signs can be set to 0.5 meters.
[0074] The laser point cloud backbone network 52 takes as input the laser radar point cloud data acquired by a laser radar mounted on top of the target vehicle and outputs the laser point cloud bird's-eye view features. Exemplarily, the acquired laser radar point cloud data is gridded to obtain point cloud voxels, which are then passed through a laser point cloud encoder to obtain the laser point cloud bird's-eye view features. The voxelization process employs non-deterministic voxelization, meaning that all data points are no longer processed uniformly. Instead, voxelization is selectively performed based on the importance of the current point cloud position. The laser point cloud encoder is implemented using a 3D convolutional neural network.
[0075] The radar point cloud backbone network 53 takes as input millimeter-wave radar point cloud data acquired by millimeter-wave radars installed at the front and rear of the target vehicle, and outputs radar point cloud bird's-eye view features. Exemplarily, the acquired millimeter-wave radar point cloud data is gridded to obtain a radar point cloud, which is then passed through a radar point cloud encoder to obtain radar point cloud bird's-eye view features. Due to the sparsity of radar point clouds, the computational complexity is much lower than that of laser point clouds. Furthermore, to obtain target velocity, the point cloud data requires a consistent representation. Therefore, deterministic gridding is used to uniformly process each data point. The radar point cloud encoder is also implemented using a 3D convolutional neural network.
[0076] The visual bird's-eye view features, laser point cloud bird's-eye view features and radar point cloud bird's-eye view features output by the image backbone network, laser point cloud backbone network and radar point cloud backbone network are stored in the form of gridded data matrices and have the same scale in the spatial dimension.
[0077] The feature fusion network 54 takes as input the visual bird's-eye view features, the laser point cloud bird's-eye view features, and the radar point cloud bird's-eye view features, and outputs the bird's-eye view fusion features. For example, the visual bird's-eye view features, the laser point cloud bird's-eye view features, and the radar point cloud bird's-eye view features are concatenated horizontally and then fed into the bird's-eye view encoder for feature fusion, thereby generating the bird's-eye view fusion features.
[0078] The task head portion 35, as shown in Figures 4 and 7, is a parallel processing in terms of timing, but logically has a serial-parallel hybrid processing structure. The task head can include: a 3D detection head, a map segmentation head, a traffic sign detection head, a traffic light detection head, a real-time positioning head, and a trajectory prediction head. The 3D detection head and the map segmentation head are the basic task heads, and each task head is basically constructed as a convolutional neural network. The task heads work simultaneously but independently, and their output results are synchronized in time. The traffic sign detection head and the traffic light detection head are dependent on the 3D detection head, the real-time positioning head is dependent on the map segmentation head, and the trajectory prediction head is dependent on the 3D detection head, the map segmentation head, the traffic sign detection head, the traffic light detection head, and the real-time positioning head. Exemplarily, the input of the 3D detection head is the bird's-eye view fusion feature, and the output is the 3D target information detection frame; the input of the map segmentation head is the bird's-eye view fusion feature, and the output is the real-time map element mask; the input of the traffic sign detection head is the 3D target information detection frame, and the output is the traffic sign information; the input of the traffic light detection head is the 3D target information detection frame, and the output is the traffic light status information; the input of the real-time positioning head is the real-time map element mask, and the output is the geographic location vector; the input of the trajectory prediction head is the 3D target information detection frame, the real-time map element mask, the traffic sign information, the traffic light status information and the geographic location vector, and the output is the trajectory prediction information of the target object (pedestrians, vehicles, etc.).
[0079] The perception fusion system proposed in the embodiment of the present application includes an image backbone network, a laser point cloud backbone network, a radar point cloud backbone network, a feature fusion network, and a task head. Utilizing the perception fusion system proposed in the embodiment of the present application, the effective fusion of surround view image data, lidar point cloud data, and millimeter wave radar point cloud data can be achieved. Selecting a bird's-eye view as a unified representation for feature fusion can effectively preserve the semantic density of surround view image data and the geometric structure of lidar point cloud data and millimeter wave radar point cloud data; simplify the process of the perception fusion algorithm; and simultaneously realize a data-driven end-to-end perception system based on a bird's-eye view mode, thereby improving the utilization rate of perception data and realizing multi-sensor fusion at the feature level, effectively improving the accuracy of perception tasks.
[0080] Example 4
[0081] FIG8 shows a block diagram of an electronic device 40 that can be used to implement an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0082] As shown in Figure 8, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. Various programs and data required for the operation of the electronic device 40 can also be stored in the RAM 43. The processor 41, ROM 42 and RAM 43 are connected to each other via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.
[0083] Multiple components in the electronic device 40 are connected to the I / O interface 45, including an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a magnetic disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.
[0084] Processor 41 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. Processor 41 can implement the functions and processing of each system described above, for example, it can implement the functions of the perception fusion system.
[0085] In some embodiments, the functionality of the perception fusion system may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more of the functionality of the perception fusion system described above may be implemented. Alternatively, in other embodiments, processor 41 may be configured to implement the functionality of the perception fusion system in any other suitable manner (e.g., via firmware).
[0086] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0087] Computer programs for implementing the methods of the present application can be written in one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0088] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. A machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0089] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) configured to display information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be configured to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0090] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0091] A computing system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by running computer programs on the respective computers. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a host product within a cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and virtual private server (VPS) services.
[0092] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of this application can be achieved. This is not limited herein.
Claims
1. A perception fusion system, comprising: A data acquisition module, configured to acquire environmental perception data of a target vehicle, wherein the environmental perception data includes at least surround view image data, laser radar point cloud data, and millimeter wave radar point cloud data; A data encoding module, configured to process the surround view image data, the laser radar point cloud data, and the millimeter wave radar point cloud data according to different preset encoders to obtain corresponding bird's-eye view features; The feature fusion module is configured to process each of the bird's-eye view features according to a preset bird's-eye view encoder to obtain a bird's-eye view fusion feature, and generate a task result based on the bird's-eye view fusion feature and the task header.
2. The system according to claim 1, wherein: The data encoding module is configured as follows: Calling the 50-layer residual network ResNet50 neural network of the first preset encoder to extract image features of at least one of the surround view image data, and horizontally splicing the image features to obtain an image feature matrix, serializing the image feature matrix into an image feature sequence, inputting the feature sequence into a compression encoder of the first preset encoder for compression processing to obtain image sparse features, and converting the image sparse features into a 3D bird's-eye view space as the first bird's-eye view features; Performing a rasterization process based on non-deterministic voxelization on the laser radar point cloud data to obtain laser point cloud voxels, and then inputting the laser point cloud voxels into a second preset encoder composed of a 3D convolutional neural network to obtain a second bird's-eye view feature; The millimeter-wave radar point cloud data is subjected to deterministic gridding processing, and the processed millimeter-wave radar point cloud data is then input into a third preset encoder composed of a 3D convolutional neural network to obtain a third bird's-eye view feature.
3. The system according to claim 2, wherein: The data encoding module is configured to obtain image sparse features in the following manner: Calling a preset quantitative scorer to determine the dependency score of each of the image features in the feature sequence; The image features whose dependency scores in the feature sequence are lower than a preset filtering threshold are eliminated as the image sparse features.
4. The system according to claim 1, wherein: The task head includes at least one of the following: a 3D detection head, a map segmentation head, a traffic sign detection head, a traffic light detection head, a real-time positioning head, and a trajectory prediction head, wherein the 3D detection head and the map segmentation head are basic task heads.
5. The system according to claim 1, wherein: The feature fusion module is set to: The bird's-eye view features are horizontally spliced and input into the preset bird's-eye view encoder composed of a convolutional neural network to obtain the bird's-eye view fusion feature; The bird's-eye view fusion features are input into each of the task heads respectively composed of convolutional neural networks. Execute the perception task and output the corresponding task result.
6. The system according to claim 5, wherein: The feature fusion module is configured to output the corresponding task result in at least one of the following ways: Inputting the bird's-eye view fusion features into the 3D detection head in the task head, and outputting the task result as a 3D target information detection frame; Inputting the bird's-eye view fusion features into the map segmentation head in the task head, and outputting the task result as a real-time map element mask; Inputting the 3D target information detection frame into the traffic sign detection head in the task head, and outputting the task result as traffic sign information; Inputting the 3D target information detection frame into the traffic light detection head in the task head, and outputting the task result as traffic light status information; Inputting the real-time map element mask into the real-time positioning head in the task head, and outputting the task result as a geographic location vector; The 3D target information detection frame, the real-time map element mask, the traffic sign information, the traffic light status information and the geographic location vector are input into the trajectory prediction head in the task head, and the output task result is the trajectory prediction information of the target object.
7. The system according to claim 6, further comprising: The image conversion module is configured to convert a two-dimensional 2D input frame into the 3D target information detection frame using a geometric transformation, wherein the geometric transformation is obtained by a mutual relationship parameter matrix between the camera and the point cloud.
8. The system according to claim 2, wherein: The first bird's-eye view feature, the second bird's-eye view feature and the third bird's-eye view feature are stored in a grid-processed data matrix and have the same scale in the spatial dimension.
9. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can implement the perception fusion system according to any one of claims 1 to 8.
10. A computer-readable storage medium storing computer instructions, wherein the computer instructions, when executed by a processor, implement the perception fusion system according to any one of claims 1 to 8.
Citation Information
Patent Citations
Ground three-dimensional laser point cloud and ground penetrating radar image fusion method
CN107544095A
Perception fusion system, electronic equipment and storage medium
CN116664997A
Object detection device and object detection method
JP2018097765A
Method and apparatus for generating alignment matrix for camera-radar system
US20160116573A1
Target detection method based on fusion of vision, lidar, and millimeter wave radar
US20220277557A1
Cited By
Road tunnel environment traffic condition monitoring method and device
CN119600555A
Intelligent trajectory planning prediction method and device, equipment and storage medium
CN119647717A
Intelligent driving track prediction method and device, equipment and storage medium
CN119682788A
A high-precision map reconstruction method
CN119784960A
Multi-mode obstacle detection method and device, electronic equipment and storage medium
CN119810792A
