Multi-source heterogeneous data fusion and 3D target detection method for multi-axle vehicle
Through multi-source heterogeneous data fusion technology, combined with multi-view cameras and lidars, sparse tensors are established and fused network processing is solved, and the problems of degradation of long-distance detection performance and difficulty in obtaining 3D information are achieved, and higher detection accuracy and Information acquisition.
Patent Information
- Application Number
- CN202411430924.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2024-11-26
AI Technical Summary
The existing 3D object detection technology has deteriorated detection performance at long distances, making it difficult to directly obtain 3D information, especially the sparse data of the lidar point cloud, the camera imaging quality is reduced, and technical solutions that cannot effectively solve these problems have not yet appeared.
Using a multi-source heterogeneous data fusion method for multi-axis vehicles, deep features are extracted through convolutional layers, combined with multi-view cameras and lidar data, geometric and semantic consistency are established, sparse tensors are constructed, and ROI pooling and fusion networks are performed. Process, dynamically adjust weights to achieve the fusion of 2D and 3D detection results.
It improves detection performance, can directly obtain 3D information, make up for the limitations of single mode detection, and improves detection accuracy, especially in long-distance and occlusion scenarios.
Smart Images

Figure CN119027935A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of 3D target detection, and in particular to a multi-source heterogeneous data fusion and 3D target detection method for multi-axle vehicles. Background Art
[0002] Currently, 3D object detection is crucial in autonomous driving, and commonly used methods include single-modality-based detection, which refers to methods that use only one type of sensor data for 3D object detection. These sensors usually include cameras, lidars, etc. Lidars can provide high-precision 3D point cloud data, and cameras can provide high-resolution image data.
[0003] In the process of realizing the present invention, the inventors found that the current technical solution has at least the following deficiencies: at a long distance, the point cloud data of the lidar will become relatively sparse, its detection performance will be significantly reduced, the imaging quality of the camera will be significantly reduced, and it will be difficult to directly obtain 3D information.
[0004] At present, no technical solution to the above problems has been found. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a multi-source heterogeneous data fusion and 3D target detection method for multi-axle vehicles in view of the above shortcomings, which overcomes the defects of the prior art in that the detection performance is reduced and it is difficult to directly obtain 3D information; and achieves the purpose of improving the detection performance and being able to directly obtain 3D information.
[0006] In order to solve the above technical problems, the technical solution of the present invention is: The multi-source heterogeneous data fusion and 3D target detection method for multi-axle vehicles includes the following steps: Step 1. Obtain 2D multi-view camera photo detection information taken by a multi-view camera and 3D point cloud detection information collected by a lidar; Step 2. 2D multi-view camera photo detection information and 3D point cloud detection information are extracted through convolutional layers to extract deep features, establish geometric and semantic consistency, construct and define a tensor composed of non-empty elements, index the non-empty elements through sparse tensor processing technology, and establish a tensor T out ; Step 3. After processing the 2D multi-view camera photo detection information and the 3D point cloud detection information through the convolution layer, detection box and confidence level, the 2D detection results and 3D detection results are output respectively; Step 4. Input the 2D detection results and the 3D detection results into the ROI pooling module to filter out the detection results; Step 5. The screened 2D detection results and 3D detection results are fused through a fusion network to obtain the detection results.
[0007] Further, in step 2, each 2D multi-view camera photo detection information and 3D point cloud detection information is converted into a detection candidate set to form a sparse tensor; Implement a geometric consistency metric, usually using Intersection over Union (IoU) to represent the geometric consistency between 2D and 3D detections; Set the confidence score to reflect the trustworthiness of each detection candidate.
[0008] Furthermore, in step 3, the 3D point cloud detection information is projected onto a grid of a custom resolution, grid pixels are generated, and height encoding is performed; For each grid pixel, extract the point with the maximum height, and use the reflectivity value of the point as the intensity feature for intensity encoding; The 2D detection results include multi-view camera detection box (2D) and multi-view camera confidence (2D), which represent the 2D detection candidate boxes and their confidence scores at different view angles. The 3D detection results include bird's-eye view detection box (3D), bird's-eye view confidence (3D), front view detection box (3D) and front view confidence (3D). Multi-view camera detection box (2D), multi-view camera confidence (2D), lidar bird's-eye view detection box (3D), lidar front view detection box (3D), lidar bird's-eye view confidence (3D) and lidar front view confidence (3D) are candidates for subsequent input.
[0009] Further, in step 4, ROI pooling of the 2D detection results is performed to process the 2D detection information from the multi-view camera and generate a 2D ROI; ROI pooling of 3D detection results, processing the bird's-eye view 3D detection information from the lidar to generate the bird's-eye view ROI, processing the front view 3D detection information from the lidar to generate the front view ROI.
[0010] Further, in step 5, the multi-view camera detection frame (2D), the lidar bird's eye view detection frame (3D), and the lidar front view detection frame (3D) are combined in a fusion network; The fusion network combines the multi-view camera confidence (2D), the lidar bird's-eye view confidence (3D) and the lidar front view confidence (3D) for scoring, dynamically adjusts the weights, and obtains the adjusted detection results.
[0011] Furthermore, the multi-view camera obtains k multi-view camera photos, i.e., k 2D detection candidate points, and the laser radar obtains n laser point cloud information, i.e., n 3D candidate detection points; For k 2D detection candidate points in an image, they can be defined as follows:
[0012] P 2D is the set of all k 2D detection candidate points in an image. i 2D , x i1 ,y i1 、x i2 ,y i2 are the pixel coordinates of the top left and bottom right points in the 2D bounding box, S i 2D is the confidence score; For n 3D candidate detection points in a LiDAR scan, the definition is as follows:
[0013] Where P 3D is the set of all n 3D candidate detection points in a LiDAR scan. For the i-th detection P i 3D ,h i 、w i , l i 、x i ,y i 、z i ,θ i is the 7-bit vector of the 3D bounding box, S i 3D is the 3D confidence score; For k 2D detection candidate points and n 3D detection candidate points, a k×n×4 tensor T is constructed and defined; For each non-empty element (i, j), there are 4 channels; The non-empty element (i, j) is passed to the fusion network for processing, and the index of the non-empty element (i, j) is saved in the cache, generating a tensor of size 1×p×1, where p is the number of non-empty elements in the input tensor; Save the indices of these non-empty elements (i, j) and construct a tensor T of shape k×n×1 by filling p outputs based on the index non-empty elements (i, j) out .
[0014] The present invention adopts the above technical solution, which has the following advantages compared with the prior art: The present invention combines the data input of multi-view cameras and lidars, and by fusing the multi-view camera photos taken by the multi-view camera and the lidar point cloud, it makes up for the limitations of a single modality (such as using only a camera or lidar), and solves the problem that a single sensor is difficult to efficiently obtain three-dimensional information. Deep fusion is achieved through geometric and semantic consistency, ensuring the spatial and semantic alignment of 2D and 3D data, and improving detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 Schematic diagram of the network architecture of a multi-source heterogeneous data fusion and 3D target detection model for a multi-axle vehicle in an embodiment of the present invention; Figure 2 for Figure 1 A partially enlarged schematic diagram of . DETAILED DESCRIPTION
[0016] The specific embodiments of the present invention are further described below in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0017] Examples, such as Figure 1-2 As shown, this embodiment provides a multi-source heterogeneous data fusion and 3D target detection method for a multi-axle vehicle, including the following steps: Step 1. Obtain 2D multi-view camera photo detection information taken by a multi-view camera and 3D point cloud detection information collected by a lidar, which are multi-source heterogeneous data.
[0018] Multi-view cameras and lidar are used to read and calibrate parameters, and the 3D bounding box in the lidar coordinates is accurately projected onto the image plane.
[0019] The benefit of a 3D object candidate is that it can be projected into any view in 3D space (including bird’s-eye view, front view, etc.).
[0020] Step 2. The 2D multi-view camera photo detection information and the 3D point cloud detection information are extracted through the convolution layer to extract deep features, establish geometric and semantic consistency, construct and define a tensor composed of non-empty elements, index the non-empty elements through sparse tensor processing technology, and establish a tensor filling output.
[0021] Step 2.1 Sparse tensor representation: Convert each 2D multi-view camera photo detection information and 3D point cloud detection information into a detection candidate set to form a sparse tensor.
[0022] Step 2.2: Implement a geometric consistency metric, usually using IoU (Intersection over Union) to represent the geometric consistency between 2D and 3D detections.
[0023] Step 2.3 sets the confidence score to reflect the credibility of each detection candidate.
[0024] Step 3. After processing the 2D multi-view camera photo detection information and the 3D point cloud detection information through the convolution layer, detection box and confidence, the 2D detection results and 3D detection results are output respectively as candidate targets for the next step.
[0025] Step 3.1 Project the 3D point cloud detection information onto a grid with a custom resolution, generate grid pixels, and perform height encoding; Step 3.2 For each grid pixel, extract the point with the maximum height and use the reflectance value of the point as the intensity feature for intensity encoding.
[0026] Step 3.3 The 2D detection results include multi-view camera detection box (2D) and multi-view camera confidence (2D), which represent the 2D detection candidate boxes and their confidence scores at different view angles. The lidar 3D point cloud outputs the bird's-eye view detection box (3D), bird's-eye view confidence (3D), front view detection box (3D) and front view confidence (3D). The multi-view camera detection box (2D), multi-view camera confidence (2D), lidar bird's-eye view detection box (3D), lidar front view detection box (3D), lidar bird's-eye view confidence (3D) and lidar front view confidence (3D) are used as candidates for subsequent input.
[0027] Obtain multi-source heterogeneous data collected by multi-view cameras and lidars to generate 2D and 3D detection information respectively. Multi-view cameras obtain multi-view images, and the high-precision point cloud obtained by lidar is projected onto the bird's-eye view and the front view plane to present bird's-eye views from different perspectives.
[0028] First, the multi-view camera obtains k multi-view camera photos, that is, k 2D detection candidate points, and the lidar obtains n laser point cloud information, that is, n 3D candidate detection points.
[0029] For k 2D detection candidate points in an image, they can be defined as follows:
[0030] P 2D is the set of all k 2D detection candidate points in an image. i 2D , x i1 ,y i1 、x i2 ,y i2 are the pixel coordinates of the top left and bottom right points in the 2D bounding box, S i 2D is the confidence score; For n 3D candidate detection points in a LiDAR scan, they can be defined as follows:
[0031] Where P 3D is the set of all n 3D candidate detection points in a LiDAR scan. For the i-th detection Pi 3D ,h i 、w i , l i 、x i ,y i 、z i ,θ i is the 7-bit vector of the 3D bounding box, S i 3D is the 3D confidence score; For k 2D detection candidate points and n 3D detection candidate points, a k×n×4 tensor T is constructed and defined.
[0032] For each element (i, j), there are 4 channels.
[0033] The input tensor is sparse because for each projected 3D detection, there are only a few 2D detections that intersect it, so most elements are empty. This exploits the sparsity of the tensor T and makes the computation faster and feasible for large k and large n values.
[0034] Only non-empty elements (i, j) are passed to the fusion network for processing. The indices of non-empty elements (i, j) are important for further calculations, so the indices of these non-empty elements (i, j) are saved in the cache, resulting in a tensor of size 1×p×1, where p is the number of non-empty elements in the input tensor.
[0035] By saving the indices of these non-empty elements (i, j), we can construct a tensor T of shape k×n×1 by filling p outputs based on the index non-empty elements (i, j) out And put negative infinity somewhere else.
[0036] Finally, the tensor T is pooled by the maximum pooling of the first dimension. out Mapping to the desired learning target, a probability score map of size 1×n×1, plays an important role in candidate box generation.
[0037] Step 4. Input the 2D detection results and 3D detection results into the ROI (region of interest) pooling module to filter out the detection results.
[0038] Specifically, ROI pooling of 2D detection results processes 2D detection information from multi-view cameras and generates 2DROI; ROI pooling of 3D detection results, processing the bird's-eye view 3D detection information from the lidar to generate the bird's-eye view ROI, processing the front view 3D detection information from the lidar to generate the front view ROI.
[0039] Step 5. The screened 2D detection results and 3D detection results are fused through a fusion network to obtain the detection results.
[0040] Step 5.1 Combine the multi-view camera detection frame (2D), the lidar bird's eye view detection frame (3D), and the lidar front view detection frame (3D) in the fusion network.
[0041] Step 5.2 The fusion network combines the multi-view camera confidence (2D), the lidar bird's-eye view confidence (3D), and the lidar front view confidence (3D) for scoring, dynamically adjusts the weights, and obtains the adjusted detection results.
[0042] This embodiment sets a bird's-eye view representation, which mainly represents height, intensity, and density encoding. The high-precision point cloud projection point cloud is discretized into grid pixels with a self-defined resolution. For each unit, the height feature is calculated as the maximum height of the midpoint of the unit.
[0043] Set the front view representation, which provides supplementary information to the bird's eye view.
[0044] Although the bird's-eye view can provide a good bird's-eye view, due to the sparsity of point cloud data, directly projecting it onto the image plane will result in a sparse 2D point map, which makes it difficult to fully utilize the information.
[0045] Calculation of front view coordinates Given a 3D point p=(x,y,z), its coordinates in the front view pfv=(r,c) can be calculated.
[0046] In order to better represent the point cloud, it is projected onto a cylindrical plane to generate a dense front view. This projection method can make fuller use of the point cloud data and avoid the impact of sparsity.
[0047] Project the point cloud data onto a grid of custom resolution to generate grid pixels. For each grid cell, calculate the maximum height of the points in it as the height feature of the cell.
[0048] In order to capture more detailed height information, the point cloud is divided into multiple slices, and a height map is calculated for each slice to obtain a custom number of height maps.
[0049] The multi-view camera photos taken by a single multi-view camera and the 3D point cloud detection information detected by the lidar are converted into a set of consistent joint detection candidates (i.e., sparse tensors). The multi-view camera photos taken by the multi-view camera can enrich the detailed information of the environment, while the 3D point cloud detection information detected by the lidar can accurately locate the relative positions of surrounding objects in the three-dimensional space. The output is a three-dimensional oriented bounding box and confidence score based on the lidar coordinates.
[0050] The 3D bounding box is usually encoded using a 7-bit vector, which includes: 3D dimensions (height, width, length), 3D position (x, y, z), and rotation (yaw angle θ). This 7-bit vector is used to describe the geometric features of each 3D bounding box, and then the height is encoded, and the high-precision point cloud data is projected onto a grid of a custom resolution to generate grid pixels, which are then intensity encoded. For each grid pixel, the point with the maximum height is extracted, and the reflectivity value of the point is used as the intensity feature.
[0051] The goal of the encoding step is to integrate the scattered 2D and 3D detection results into a consistent set of detection candidates, which can be used as input to the subsequent fusion network.
[0052] By cooperating with multi-source heterogeneous data, the inadequacy of detection effects under a single modality can be compensated, especially in long-distance or occluded scenes, to maximize the probability of extracting all potential correct detections. The fusion network is data-driven and trains a discriminative network that receives as input the output scores and classification of a single candidate detection, as well as the spatial description of the candidate detection.
[0053] We learn from data how to best combine input detection candidates for a final output detection, combining gradient descent training with an auxiliary loss and showing superior performance over early / late fusion schemes.
[0054] Given the images acquired by the multi-view camera, the fusion network performs directional 3D candidate box regression to predict the exact position, size and orientation of the object in space, and further obtains a candidate detection set, which contains all the candidate detections in a lidar scan. Each detection contains a 7-bit vector and the corresponding 3D confidence score.
[0055] 3D Object Candidates,In the bird’s-eye view, objects maintain their physical size when projected,,which makes the size variation of objects in the image smaller.,Since in the bird’s-eye view, objects occupy different spatial regions,,the occlusion problem caused by object overlap can be avoided.
[0056] Accuracy of position prediction,In road scenes, objects are usually located on the ground plane and have little,variation in the vertical direction. Using explicit bird's-eye view as input,makes the prediction of 3D position more feasible and,accurate.
[0057] Fusion network architecture,The fusion network architecture processes the input sparse tensor through a series of 1×1 2D convolutional layers,,applies the ReLU activation function after each convolutional layer to enhance the,feature representation capability, and finally generates a probability score map through,the maximum pooling operation.
[0058] During the training process, a dataset containing 2D and 3D detection annotations (such as the KITTI dataset) is used, the cross entropy loss function is combined with focal loss to deal with the imbalance problem of positive and negative samples, and the Adam optimizer is used to gradually reduce the learning rate.
[0059] Finally, the fusion network can automatically learn and optimize the combination of 2D and 3D detectors.
[0060] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.
Claims
1. Multi-source heterogeneous data fusion and 3D target detection method for multi-axis vehicles, Features: The following steps are involved: Step 1. Obtain 2D multi-view camera photo detection information taken by a multi-view camera and 3D point cloud detection information collected by a lidar; Step 2. 2D multi-view camera photo detection information and 3D point cloud detection information are extracted through convolutional layers to extract deep features, establish geometric and semantic consistency, construct and define a tensor composed of non-empty elements, index the non-empty elements through sparse tensor processing technology, and establish a tensor T out ; Step 3. After processing the 2D multi-view camera photo detection information and the 3D point cloud detection information through the convolution layer, detection box and confidence level, the 2D detection results and 3D detection results are output respectively; Step 4. Input the 2D detection results and the 3D detection results into the ROI pooling module to filter out the detection results; Step 5. The screened 2D detection results and 3D detection results are fused through a fusion network to obtain the detection results.
2. The multi-source heterogeneous data fusion and 3D target detection method for multi-axle vehicles as claimed in claim 1, Features: In step 2, each 2D multi-view camera photo detection information and 3D point cloud detection information is converted into a detection candidate set to form a sparse tensor; Implement a geometric consistency metric, usually using Intersection over Union (IoU) to represent the geometric consistency between 2D and 3D detections; Set the confidence score to reflect the trustworthiness of each detection candidate.
3. The multi-source heterogeneous data fusion and 3D target detection method for multi-axle vehicles as claimed in claim 2, Features: In step 3, the 3D point cloud detection information is projected onto a grid of custom resolution, grid pixels are generated, and height encoding is performed; For each grid pixel, extract the point with the maximum height, and use the reflectivity value of the point as the intensity feature for intensity encoding; The 2D detection results include multi-view camera detection box (2D) and multi-view camera confidence (2D), which represent the 2D detection candidate boxes and their confidence scores at different view angles. The 3D detection results include bird's-eye view detection box (3D), bird's-eye view confidence (3D), front view detection box (3D) and front view confidence (3D). Multi-view camera detection box (2D), multi-view camera confidence (2D), lidar bird's-eye view detection box (3D), lidar front view detection box (3D), lidar bird's-eye view confidence (3D) and lidar front view confidence (3D) are candidates for subsequent input.
4. The multi-source heterogeneous data fusion and 3D target detection method for multi-axle vehicles as claimed in claim 3, Features: In step 4, ROI pooling of 2D detection results is performed to process 2D detection information from multi-view cameras and generate 2D ROI; ROI pooling of 3D detection results, processing the bird's-eye view 3D detection information from the lidar to generate the bird's-eye view ROI, processing the front view 3D detection information from the lidar to generate the front view ROI.
5. The multi-source heterogeneous data fusion and 3D target detection method for multi-axle vehicles as claimed in claim 4, Features: In step 5, the multi-view camera detection frame (2D), the lidar bird's eye view detection frame (3D), and the lidar front view detection frame (3D) are combined in a fusion network; The fusion network combines the multi-view camera confidence (2D), the lidar bird's-eye view confidence (3D) and the lidar front view confidence (3D) for scoring, dynamically adjusts the weights, and obtains the adjusted detection results.
6. The multi-source heterogeneous data fusion and 3D target detection method for multi-axle vehicles as claimed in claim 2, Features: The multi-view camera obtains k multi-view camera photos, i.e., k 2D detection candidate points, and the laser radar obtains n laser point cloud information, i.e., n 3D candidate detection points; For k 2D detection candidate points in an image, they can be defined as follows: 7.P 2D is the set of all k 2D detection candidate points in an image. i 2D , x i1 ,y i1 、x i2 ,y i2 are the pixel coordinates of the top left and bottom right points in the 2D bounding box, S i 2D is the confidence score; For n 3D candidate detection points in a LiDAR scan, the definition is as follows:
8. Among them P 3D is the set of all n 3D candidate detection points in a LiDAR scan. For the i-th detection P i 3D ,h i 、w i , l i 、x i ,y i 、z i ,θ i is the 7-bit vector of the 3D bounding box, S i 3D is the 3D confidence score; For k 2D detection candidate points and n 3D detection candidate points, a k×n×4 tensor T is constructed and defined; For each non-empty element (i, j), there are 4 channels; The non-empty element (i, j) is passed to the fusion network for processing, and the index of the non-empty element (i, j) is saved in the cache, generating a tensor of size 1×p×1, where p is the number of non-empty elements in the input tensor; Save the indices of these non-empty elements (i, j) and construct a tensor T of shape k×n×1 by filling p outputs based on the index non-empty elements (i, j) out .
