A multi-target detection method and device based on multimodal information fusion
Through the shared parameter convolution neural network and late fusion strategy, combined with dense tensors and hash tables, the problems of insufficient data correlation and high computational cost in the existing multimodal information fusion method are solved, and the problem of insufficient data correlation and high computational cost in autonomous driving scenarios is improved. Multi-objective detection accuracy and efficiency.
Patent Information
- Application Number
- CN202210049863.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-12-28
- Filing Date
- 2022-01-17
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-01-17
AI Technical Summary
The existing multimodal information fusion object detection method has shortcomings in utilizing multimodal data correlation. The deep fusion method is sensitive to data alignment and has a complex network structure. It cannot fully utilize the correlation between three-dimensional and two-dimensional data, resulting in accurate detection. Sex and inefficient.
Convolutional neural networks with shared parameters are used to forward pass the three-dimensional and two-dimensional features, train the network through cross-entropy loss function, combine the late fusion strategy to fuse the three-dimensional and two-dimensional object detection boxes, and use dense tensors and hashing Tables improve point cloud processing efficiency and build multi-objective detection methods suitable for autonomous driving scenarios.
It improves the robustness of features and detection accuracy, reduces the calculation cost, makes up for the limitations of single-modal methods, and realizes efficient multi-objective detection.
Smart Images

Figure CN114463736B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of three-dimensional target detection and two-dimensional target detection, and in particular to a multi-target detection method and device based on multi-modal information fusion. Background Art
[0002] In recent years, with the development of LiDAR (Light Detection and Ranging) technology, the acquisition speed and accuracy of point cloud data have been greatly improved. How to achieve efficient and accurate point cloud target detection is an important issue in the fields of intelligent driving, remote sensing, augmented reality, virtual reality, etc. Compared with traditional two-dimensional target detection, three-dimensional target detection requires more output parameters to determine the bounding box of the target. Due to the data characteristics of LiDAR point clouds, target detection tasks often face problems such as low input data resolution, missing texture and color information, and high computational overhead, making them more challenging.
[0003] In the face of these problems, the method of multimodal information fusion has become the research focus in this field. At present, multimodal fusion methods are mainly divided into three types: early fusion, late fusion, and deep fusion. The early fusion method performs feature fusion before extracting features from the original sensor data. The representative method is PI-RCNN (Point Cloud-Image Region Convolutional Neural Network). This algorithm directly performs point-by-point continuous convolution on the three-dimensional point cloud, and applies point pooling and attention concentration operations to obtain better fusion performance. Late fusion is the simplest fusion method. It only performs fusion at the decision layer, avoiding problems caused by differences in data from different sensors and reducing the complexity of the algorithm. The deep fusion method interacts at the feature level and makes the most effective use of cross-modal information. The representative method is MV3D (Multi-View 3D). Figure 3 The network consists of two sub-networks: one for generating 3D object candidate regions and the other for multi-view feature fusion.
[0004] The main challenges faced by current multimodal fusion target detection methods include: early fusion and late fusion cannot fully utilize the correlation between multimodal data. Deep fusion often has the disadvantages of being sensitive to data alignment and having a complex network structure. Therefore, it is hoped that there is a technical solution to overcome or at least alleviate at least one of the above defects of the existing technology. Summary of the invention
[0005] The present invention provides a multi-target detection method and device based on multimodal information fusion. The present invention overcomes the limitations of single information and poor robustness in traditional single-modal methods, and can be applied to target recognition tasks in scenarios such as autonomous driving. See the following description for details:
[0006] A multi-target detection method based on multimodal information fusion, the method comprising the following steps:
[0007] Process the LiDAR point cloud data, extract the LiDAR point cloud features, and obtain the three-dimensional feature map;
[0008] Perform two-dimensional image data processing on the RGB image, output the RGB image features through the feature extraction network, and obtain a two-dimensional feature map;
[0009] The 3D and 2D feature maps are forward-transferred through a convolutional neural network with shared parameters, and the convolutional neural network is trained according to the cross entropy to achieve 3D and 2D feature fusion;
[0010] The fused 3D feature map is input into the 3D region generation network, which maps the 3D feature map through two branches and outputs the 3D target detection box position map and probability score map respectively. The fused 2D feature map is input into the 2D region generation network, which outputs the 2D target detection box position map and probability score map. At this time, the position and detection probability score map of the target in the point cloud data and RGB 2D image are obtained respectively. The post-fusion strategy is adopted to fuse the target detection boxes under the two modalities to obtain the final target detection result.
[0011] The LiDAR point cloud data is processed to extract the LiDAR point cloud features and obtain the three-dimensional feature map:
[0012] The point cloud is evenly grouped into several voxels, and the sparse and uneven point cloud is converted into a dense tensor structure. The list of voxel features is obtained by stacking the voxel feature encoding layers. After passing through the convolutional intermediate layer, the voxel features are aggregated in the gradually expanding receptive field to output the LiDAR point cloud features, that is, the three-dimensional feature map.
[0013] Furthermore, the two-dimensional image data processing is performed on the RGB image, and the RGB image features are output through the feature extraction network to obtain the two-dimensional feature map. Specifically,
[0014] Perform uniform grouping operations on the two-dimensional RGB image, and the width and height of each group block are equal to the voxel in LiDAR point cloud data processing;
[0015] Through the feature extraction network composed of convolutional layers, pooling layers and linear rectification layers, the RGB image features, that is, the two-dimensional feature map, are output.
[0016] The method of converting the sparse and uneven point cloud into a dense tensor structure is as follows:
[0017] The voxel coordinates and the coordinates of the voxel midpoint are used as key-value pairs to build a hash table to realize voxel search and initialization. The voxel input features and coordinate buffer are constructed by traversing the point list once.
[0018] Furthermore, the post-fusion strategy is adopted to fuse the target detection frames of the two modalities to obtain the final target detection result as follows:
[0019] The detection results of the two modes are represented by a mixed representation:
[0020]
[0021] Among them, IoU i is the intersection-over-union ratio of the 2D and 3D bounding boxes, which is used to indicate the geometric consistency between the i-th detection result in the image and the j-th detection result in the point cloud; is the confidence score of the i-th detected object in two-dimensional detection; is the confidence score in the point cloud scene; d j Represents the normalized distance from the jth object detected to the ground in the point cloud scene;
[0022] If both the 2D and 3D detection networks correctly detect the same target, the projection of the 3D detection box on the 2D image should have a larger intersection over union ratio with the 2D detection box. If the 2D and 3D detection results are inconsistent and the IoU is 0, then the IoU i , d j All are set to 0;
[0023] The non-empty vector is input into the convolutional neural network and mapped into the target detection box position regression map and the corresponding probability score map through maximum pooling. The final output of the network is the three-dimensional target detection box and its corresponding probability score.
[0024] Wherein, the two-dimensional target detection result is expressed as:
[0025]
[0026] Among them, P 2D is the set of two-dimensional target detection results, k represents the number of targets detected in the current two-dimensional image, P i 2D represents the i-th target in the detection result set, P i 2D The first item in ([x i1 ,y i1 ,x i2 ,y i2 ]) is the two-dimensional object detection bounding box, and the second is the confidence score; x i1,y i1 ,x i2 ,y i2 are the horizontal and vertical coordinates of the upper left corner vertex and the lower right corner vertex of the two-dimensional target bounding box, respectively. is the confidence score of the two-dimensional target bounding box, and K is the number of currently detected targets;
[0027] The three-dimensional target detection result is expressed as:
[0028]
[0029]
[0030] Among them, P 3D is the set of 3D target detection results, n represents the number of targets detected in the current 3D point cloud, P i 3D represents the i-th target in the detection result set, h i ,w i ,l i ,x i ,y i ,θ i They are the height, width, length, vertex abscissa, vertex ordinate, and rotation angle relative to the z-axis of the 3D target bounding box. is the confidence score of the 3D object bounding box.
[0031] In a second aspect, a multi-target detection device based on multimodal information fusion is provided, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to execute any one of the method steps described in the first aspect.
[0032] In a third aspect, a computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the method steps described in the first aspect.
[0033] The beneficial effects of the technical solution provided by the present invention are:
[0034] 1. The present invention forward-transfers the LiDAR point cloud features and the RGB image features through a convolutional neural network with shared parameters, trains the network according to cross entropy, realizes feature fusion, makes up for the limitation of missing single-modal data information, and improves the robustness of the features;
[0035] 2. The present invention introduces a late fusion strategy, which utilizes the geometric consistency between the detection results of different modalities to fuse them at the decision level, thereby improving the detection accuracy;
[0036] 3. The present invention improves the efficiency of point cloud processing by constructing dense tensors and hash tables, making the algorithm more efficient and solving the problem of excessive computational cost in current methods;
[0037] 4. The present invention has confirmed the rationality of the network design and the superiority of its performance through comparative experiments with existing methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 It is a flowchart of a multi-target detection method based on multimodal information fusion;
[0039] Figure 2 Schematic diagram of feature fusion;
[0040] Figure 3 Generate a schematic diagram of the network for a three-dimensional region;
[0041] Figure 4 Schematic diagram of the network generated for a two-dimensional region;
[0042] Figure 5 It is a structural schematic diagram of a multi-target detection device based on multimodal information fusion. DETAILED DESCRIPTION
[0043] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.
[0044] Example 1
[0045] A multi-target detection method based on multimodal information fusion, see Figure 1 , the method comprises the following steps:
[0046] 101: Processing the LiDAR point cloud data and extracting LiDAR point cloud features, i.e., a three-dimensional feature map;
[0047] According to the sparsity of LiDAR data, the embodiment of the present invention adopts a resampling method to increase the sampling points, which can improve the density of data to a certain extent, thereby improving the effect of the three-dimensional feature map and the effectiveness of detection.
[0048] 102: Perform two-dimensional image data processing on the RGB image, and output RGB image features, i.e., a two-dimensional feature map, through a feature extraction network;
[0049] Since two-dimensional images naturally lack three-dimensional information, in the detection stage after feature extraction, it is necessary to associate the three-dimensional information based on spatial position and pixel information to ensure the effectiveness of subsequent fusion.
[0050] 103: The 3D and 2D feature maps are forward-transferred through a convolutional neural network with shared parameters, and the network parameters are trained according to the feedback mechanism of the cross entropy loss function to achieve effective fusion of 3D and 2D features;
[0051] 104: Generate target detection frames through three-dimensional and two-dimensional region generation networks respectively using the fused feature map to obtain actual position information of the template to be detected;
[0052] 105: Adopt a post-fusion strategy to fuse the detection frames of the two modalities to obtain the final target detection result.
[0053] In summary, the embodiments of the present invention overcome the limitations of single information and poor robustness in traditional single-modal methods through the above steps, and can be applied to target recognition tasks in scenarios such as autonomous driving.
[0054] Example 2
[0055] The scheme in Example 1 is further introduced below in combination with specific examples and calculation formulas, as described below for details:
[0056] 201: Processing the LiDAR point cloud data and outputting the LiDAR point cloud features, i.e., a three-dimensional feature map;
[0057] Specifically, the point cloud is evenly grouped into a number of voxels, the sparse and uneven point cloud is converted into a dense tensor structure, and a list of voxel features is obtained by stacking voxel feature encoding layers. After passing through the convolutional intermediate layer, the voxel features are aggregated in the gradually expanding receptive field to output the LiDAR point cloud features, that is, the three-dimensional feature map.
[0058] 202: Perform two-dimensional image data processing on the RGB image, and output RGB image features, i.e., a two-dimensional feature map, through a feature extraction network;
[0059] Specifically, a uniform grouping operation is performed on the two-dimensional RGB image, and the width and height of each group block are equal to the voxel in the LiDAR point cloud data processing; through a feature extraction network composed of a convolutional layer, a pooling layer, and a linear rectification layer, the RGB image features, i.e., a two-dimensional feature map, are output.
[0060] 203: The 3D and 2D feature maps are forward-transferred through a convolutional neural network with shared parameters, and the convolutional neural network is trained to achieve 3D and 2D feature fusion (i.e., cross entropy), so that the features of the same target in the 3D and 2D feature maps are more similar. This part of the schematic diagram is shown in FIG. Figure 2 As shown;
[0061] 204: Generate target detection frames respectively through three-dimensional and two-dimensional region generation networks using the fused feature maps;
[0062] Specifically, the fused 3D feature map is input into the 3D region generation network, which maps the 3D feature map through two branches, and outputs a 3D target detection box regression map and a probability score map respectively; similarly, the fused 2D feature map is input into the 2D region generation network, and outputs a 2D target detection box regression map and a probability score map. Then, the target category and position of the candidate region in the point cloud data and RGB image are obtained.
[0063] 205: Adopt a post-fusion strategy to fuse the target detection frames of the two modalities to obtain the final target detection result.
[0064] The operation of uniformly grouping the point cloud into a number of voxels in step 201 is specifically as follows:
[0065] According to the position transformation matrix between the laser radar and the camera, the point cloud is converted to the camera's coordinate system. Assume that the lengths of the point cloud in the Z, Y, and X axis ranges are H, D, and W respectively. Define voxels with width and height as vH and vW respectively, then the number of voxels in the height and width dimensions is:
[0066] H'=H / v H , W'=W / v W ,
[0067] For simplicity, let H, W be v H , v W The points in the point cloud are grouped according to the voxels they are in.
[0068] The uniform grouping operation performed on the two-dimensional RGB image in step 202 is specifically as follows:
[0069] The width and height of the input image are H and W respectively. In order to better focus on local features and facilitate alignment with three-dimensional data and later fusion, the images are grouped and each size is defined as n H , n W The blocks, where n H , n W The size of v in the three-dimensional feature extraction network H , v W The number of blocks in width and height dimensions is:
[0070] H'=H / n H , W'=W / n W ,
[0071] Let H, W be n H , n W An integer multiple of .
[0072] The operation of converting the sparse and uneven point cloud into a dense tensor structure in step 201 is specifically as follows:
[0073] First, initialize a K×T×7-dimensional tensor structure to store the voxel input feature buffer, where K represents the maximum number of non-empty voxels, T represents the maximum number of points per voxel, and the input encoding size of each point is 7. These points are randomized before processing. For each point in the point cloud, check whether the corresponding voxel already exists. Using the voxel coordinates and the coordinates of the point in the voxel as key-value pairs to construct a hash table can efficiently realize voxel search and initialization. The voxel input feature and coordinate buffer can be constructed by traversing the point list once, so its complexity is O(n).
[0074] To further improve storage and computational efficiency, only a limited number of voxels can be stored, while voxels containing very few points can be ignored.
[0075] The operations of generating target detection frames through the three-dimensional and two-dimensional region generation networks respectively using the fused feature maps in step 204 are specifically as follows:
[0076] The fused 3D feature map is input into the 3D region generation network. The network diagram is shown in Figure 3 As shown in Figure 1. The network has three blocks of fully convolutional layers. The first layer of each block downsamples the feature map by half through a convolution with a stride of 2, followed by a sequence of convolutions with a stride of 1; BN (batch normalization) and ReLU (rectified linear function) operations are applied after each convolution layer. The output of each block is upsampled to a fixed size with a height and width of half the height and width of the original block, and the upsampled results of each block are spliced into a high-resolution feature map. With each point in the feature map as the center, 9 anchor boxes of different sizes are set as the initial detection box. The feature maps are mapped separately through two branches. The first branch classifies each anchor box and outputs a probability score map; the second branch is used to calculate the regression offset of the anchor box relative to the target true value box and output the translation and scaling parameters. The foreground anchor box and the bounding box regression offset are input into the candidate layer (proposal) together.
[0077] Similarly, the fused two-dimensional feature map is input into the two-dimensional region generation network. The processing flow of the two-dimensional region generation network is as follows: Figure 4As shown in the figure. The two-dimensional region generation network consists of two branches. With each point in the input feature map as the center, 9 anchor boxes of different sizes are set as the initial detection box. The left branch obtains the positive and negative classification of the detection anchor box through the normalized exponential function (softmax), and the right branch is used to calculate the regression offset of the anchor box relative to the true value box. The final region generation (Proposal) layer is responsible for synthesizing the positive anchor box and the corresponding regression offset, and outputting the probability score map and the anchor box regression map. Finally, the target category and position of the candidate region in the point cloud data and RGB image are obtained.
[0078] The post-fusion strategy adopted in step 205 is used to fuse the detection frames of the two modalities to obtain the final target detection result. Specifically, the operation is as follows:
[0079] If the same target is correctly detected in the output of the 3D and 2D region generation networks, the projection of the 3D target bounding box on the 2D image should have a high geometric consistency with the 2D target bounding box, which can be used as a link between the detection results of different modalities. The 2D target detection result can be expressed as:
[0080]
[0081] Among them, P 2D is the set of two-dimensional target detection results, k represents the number of targets detected in the current two-dimensional image, P i 2D represents the i-th target (1≤i≤k) in the detection result set, P i 2D The first item in ([x i1 ,y i1 ,x i2 ,y i2 ]) is the two-dimensional object detection bounding box, and the second is the confidence score; x i1 ,y i1 ,x i2 ,y i2 are the horizontal and vertical coordinates of the upper left corner vertex and the lower right corner vertex of the two-dimensional target bounding box, respectively. is the confidence score of the two-dimensional target bounding box, K is the number of currently detected targets, and is a fixed value in specific detection. For example, if 6 targets are detected in this picture, then k = 6. refers to the sixth target detected, and i is a variable, P i 2D Refers to the i-th target in the detection result, where i can be any value between 1 and k.
[0082] Similarly, the 3D object detection result can be expressed as:
[0083]
[0084]
[0085] Among them, P 3D is the set of 3D target detection results, n represents the number of targets detected in the current 3D point cloud, P i 3D Represents the i-th target in the detection result set (1≤i≤n).
[0086] h i ,w i ,l i ,x i ,y i ,θ i They are the height, width, length, vertex abscissa, vertex ordinate, and rotation angle relative to the z-axis of the 3D target bounding box. is the confidence score of the 3D object bounding box.
[0087] The detection results of the two modes are represented by a mixed representation:
[0088]
[0089] Among them, IoU i It is the intersection of two-dimensional and three-dimensional bounding boxes, which is used to represent the geometric consistency between the i-th detection result in the image and the j-th detection result in the point cloud. i express; is the confidence score of the i-th detected object in two-dimensional detection; is the confidence score in the point cloud scene; d j Represents the normalized distance from the jth object detected to the ground in the point cloud scene.
[0090] Among them, if the 2D and 3D detection networks both correctly detect the same target, the projection of the 3D detection box on the 2D image should have a larger intersection over union ratio with the 2D detection box; if the 2D and 3D detection results are inconsistent and the IoU is 0, then the IoU i , d j All are set to 0. The non-empty vector is input into the convolutional neural network and mapped into the target detection box position regression map and the corresponding probability score map through maxpooling. The final output of the network is the 3D target detection box and its corresponding probability score.
[0091] In summary, the embodiment of the present invention extracts the feature information of the three-dimensional point cloud and the two-dimensional image through the above steps, and through feature fusion, the feature vector describes the data more comprehensively, avoiding the loss of information; by constructing dense tensors and hash tables, the efficiency of point cloud processing is improved, making the algorithm more efficient, and more accurate and scientific in multi-target detection.
[0092] Example 3
[0093] The feasibility of the schemes in Embodiments 1 and 2 is verified by combining specific examples, as described below:
[0094] The KITTI dataset is used to evaluate the performance of the algorithm. The KITTI dataset is currently the largest algorithm evaluation dataset in the world for autonomous driving scenarios. It contains 7481 point clouds and images for training and 7518 point clouds and images for testing, including three categories: cars, pedestrians, and cyclists. For each category, the detection results are evaluated according to three difficulty levels: simple, medium, and difficult. The three difficulty levels are determined according to the target size, occlusion status, and truncation level. The algorithm is fully evaluated, and the training data is subdivided into a training set and a validation set, resulting in 3712 data samples for training and 3769 data samples for validation. After segmentation, samples of the same sequence will not be included in both the training and validation sets.
[0095] At the same time, in order to verify the effect of the grouping operation on the two-dimensional object detection performance in the algorithm, the two-dimensional image object detection method in the method of the present invention is evaluated on the VOC2007 dataset. The dataset contains 9963 images in 20 categories, of which 5011 are used for training and 4952 are used for testing.
[0096] The algorithm and various control algorithms were evaluated on the KITTI validation set. According to the official evaluation protocol, target detection requires the simultaneous realization of two tasks: target location and target recognition. The correctness of target location is determined by comparing the intersection over union (IoU) of the predicted bounding box and the true value box and the size of the threshold; the correctness of target recognition is determined by comparing the confidence score and the threshold. The above two steps comprehensively determine whether the target detection is correct, and finally convert the detection problem of multi-category targets into a binary classification problem of "correct detection of a certain type of object and wrong detection", so that the confusion matrix can be constructed and the model accuracy can be evaluated using a series of indicators for target classification. In the experiment, the IoU threshold of the car class is set to 0.7, and the IoU threshold of the pedestrian and cyclist class is set to 0.5. The experiment uses the average precision (AP) indicator, that is, the average of the precision under different recall rates, to compare the algorithms. For this method, the LiDAR data and RGB image data provided by KITTI are used for training from scratch, and the network weight parameters are randomly initialized.
[0097] In order to analyze the impact of the random sampling threshold of the point cloud on the performance of the algorithm, a control experiment was conducted on the algorithm of the present invention with different sampling thresholds (T) at three difficulty levels of the car category of the KITTI validation set. The experiment used the average precision index to measure the accuracy of the algorithm and recorded the average time (milliseconds) consumed by the algorithm for each detection. The experimental results are shown in Table 1. As the random sampling threshold increases, the time cost and accuracy of the algorithm are improved. Analysis of the data shows that after the sampling threshold is greater than 30, the average accuracy of the algorithm increases very little, while the time cost still increases linearly. Considering the detection speed and accuracy comprehensively, the random sampling threshold of the point cloud is determined to be 30 in subsequent experiments.
[0098] Table 1 Performance comparison of different point cloud sampling thresholds on the KITTI validation set
[0099] Table 1 Performance comparison of different pointcloud samplingthresholds on the KITTI validation set.
[0100]
[0101] In order to analyze the impact of the grouping method on the detection effect of the algorithm, three different block division methods are set:
[0102] (1)W'=W / n W =200, H' = H / n H =150,
[0103] (2)W'=W / n W =400, H' = H / n H =300,
[0104] (3)W'=W / n W =800, H' = H / n H =600,
[0105] The algorithms using the three grouping methods are compared in experiments at all difficulty levels of the three categories on the KITTI dataset. The experimental results are shown in Table 2.
[0106] Table 2 Comparison of average target detection accuracy of different grouping methods on the KITTI validation set
[0107] Table 2 Performance comparison in 3D detection: average precision onKITTI validation set.
[0108]
[0109]
[0110] From the data analysis, we can see that the second partitioning method achieves the best results. The experimental results show that a partitioning method that is too sparse will ignore the local information of the input data, while a partitioning method that is too dense will ignore the connection between features due to excessive focus on the local information. In subsequent experiments, this method will use the second partitioning method.
[0111] In order to analyze the importance of the fusion method in this method, two control examples are set up: the first control example (denoted as Deep fusion) only performs feature fusion processing on the 3D feature map and the 2D feature map output by the data processing module of the two modalities, without post-fusion of the detection box, and directly uses the detection box and target category output by the 3D region generation network as the final detection result; the second control example (denoted as late fusion) performs target detection on the data of the two modalities independently, and only post-fusion of the 3D and 2D detection results is performed in the decision stage. The control example uses the same grouping method, network structure and parameters as this method, and conducts comparative experiments on the KITTI dataset. The detection performance is evaluated using the average precision (AP). The experimental results are shown in Table 3. The detection performance of this method is superior to the control algorithm that only uses feature fusion or post-fusion, which proves that feature fusion and post-fusion have an improvement effect on target detection performance.
[0112] Table 3 Performance comparison of different fusion methods on the KITTI validation set
[0113] Table 3 Performance comparison of different fusion methods on the KITTI validation set.
[0114]
[0115] In order to analyze the role of grouping methods in target detection tasks, the two-dimensional image target detection algorithm in this method is compared with Faster R-CNN on the VOC2007 dataset. This method uses the same training method. In order to compare the detection performance, the performance of the YOLOv3 algorithm on the same dataset is added for comparison. The detection performance is evaluated by the mean average accuracy (mAP) of all classes, and the test results are shown in Table 4. The test results show that grouping processing has a significant improvement on the performance of two-dimensional target detection.
[0116] Table 4 Comparison of the average accuracy of different 2D object detection algorithms on the VOC2007 dataset
[0117] Table 4 Performance comparison in 2D detection: mean average precision on VOC2007.
[0118]
[0119]
[0120] For the car category, this method is compared with several best performing algorithms, including: image-based methods: Mono3D and 3DOP; LiDAR-based methods: VeloFCN and MV3D. Mono3D, 3DOP, and MV3D are initialized using pre-trained models. The experimental data for the above algorithms are from the experimental part of the Voxelnet paper, see the literature. According to the experimental settings in the literature, other existing algorithms are initialized with pre-trained models and then trained on the KITTI dataset. For this method, the LiDAR data and RGB image data provided by KITTI are used for training from scratch, and the network weight parameters are randomly initialized.
[0121] In order to analyze the importance of multimodal information fusion, two single-modal comparison algorithms were set up in the experiment. The network structure of the comparison algorithms is the same as the LiDAR point cloud data processing module of the present invention. Only point cloud data is used for target detection. The first comparison algorithm does not undergo grouping operations, while the second comparison algorithm undergoes the same grouping operations as this method. The comparison algorithm is trained using the LiDAR data provided by KITTI.
[0122] The comparison results with the existing methods are shown in Table 5. For the car class, the AP index of our method is significantly better than all other methods at all difficulty levels. Specifically, the performance of our method is significantly better than the representative LiDAR+RGB based method MV, exceeding it by 11.66%, 4.80% and 7.66% at the easy, medium and difficult levels respectively.
[0123] The comparison results with the single-modal control algorithms are shown in Table 6. The method is compared with the two single-modal control algorithms in the detection of three-dimensional cars, pedestrians and cyclists. Due to the large changes in the height of three-dimensional posture and shape, the detection of pedestrians and cyclists requires better three-dimensional shape representation. As shown in Table 6, the average accuracy of the method in the experiments of three difficulties in all categories is higher than that of the two single-modal control algorithms. It can be seen that the fusion of information in two-dimensional images has an improvement in the performance of three-dimensional target detection. At the same time, compared with the single-modal control algorithm without grouping operation, the detection accuracy of the control algorithm with grouping operation is improved in all three difficulties of all categories, which proves the effectiveness of the grouping operation's focus on local information in three-dimensional target detection.
[0124] Table 5 Performance comparison of different methods on the KITTI validation set
[0125] Table 5 Performance comparison of different methods on the KITTIvalidation set.
[0126]
[0127]
[0128] Table 6 Performance comparison with single-modal control method on KITTI validation set
[0129] Table 6 Performance comparison with the monomodal comparison methods on the KITTI validation set.
[0130]
[0131] In order to verify the efficiency of this method, a control algorithm was set up, whose network structure and parameters are the same as those of the method of the present invention, but the method of constructing dense tensor structures and hash tables is not used when processing point cloud data. The rate test of this method and the control algorithm was carried out on TitanX GPU and 1.7Ghz CPU. The results showed that the total inference time consumed by this method was 225ms, of which the grouping operation took 5ms. The control algorithm took up to 345ms, of which the grouping operation took 120ms, which is much slower than this method. The test results show that constructing dense tensor structures and hash tables for point clouds has a significant effect on improving data processing efficiency.
[0132] Example 4
[0133] A multi-target detection device based on multimodal information fusion, see Figure 5 The device includes a processor 1 and a memory 2, wherein the memory 2 stores program instructions, and the processor 1 calls the program instructions stored in the memory 2 to enable the device to execute the following method steps in Example 1:
[0134] Process the LiDAR point cloud data, extract the LiDAR point cloud features, and obtain the three-dimensional feature map;
[0135] Perform two-dimensional image data processing on the RGB image, output the RGB image features through the feature extraction network, and obtain a two-dimensional feature map;
[0136] The 3D and 2D feature maps are forward-transferred through a convolutional neural network with shared parameters, and the convolutional neural network is trained according to the cross entropy to achieve 3D and 2D feature fusion;
[0137] The fused 3D feature map is input into the 3D region generation network, which maps the 3D feature map through two branches and outputs the 3D target detection box position map and probability score map respectively. The fused 2D feature map is input into the 2D region generation network, which outputs the 2D target detection box position map and probability score map. At this time, the position and detection probability score map of the target in the point cloud data and RGB 2D image are obtained respectively. The post-fusion strategy is adopted to fuse the target detection boxes under the two modalities to obtain the final target detection result.
[0138] Among them, the LiDAR point cloud data is processed, the LiDAR point cloud features are extracted, and the three-dimensional feature map is obtained as follows:
[0139] The point cloud is evenly grouped into several voxels, and the sparse and uneven point cloud is converted into a dense tensor structure. The list of voxel features is obtained by stacking the voxel feature encoding layers. After passing through the convolutional intermediate layer, the voxel features are aggregated in the gradually expanding receptive field to output the LiDAR point cloud features, that is, the three-dimensional feature map.
[0140] Furthermore, the RGB image is processed into two-dimensional image data, and the RGB image features are output through the feature extraction network to obtain the two-dimensional feature map:
[0141] Perform uniform grouping operations on the two-dimensional RGB image, and the width and height of each group block are equal to the voxel in LiDAR point cloud data processing;
[0142] Through the feature extraction network composed of convolutional layers, pooling layers and linear rectification layers, the RGB image features, that is, the two-dimensional feature map, are output.
[0143] The specific method of converting the sparse and uneven point cloud into a dense tensor structure is as follows:
[0144] The voxel coordinates and the coordinates of the voxel midpoint are used as key-value pairs to build a hash table to realize voxel search and initialization. The voxel input features and coordinate buffer are constructed by traversing the point list once.
[0145] Furthermore, a post-fusion strategy is adopted to fuse the target detection frames of the two modalities to obtain the final target detection result as follows:
[0146] The detection results of the two modes are represented by a mixed representation:
[0147]
[0148] Among them, IoU i is the intersection-over-union ratio of the 2D and 3D bounding boxes, which is used to indicate the geometric consistency between the i-th detection result in the image and the j-th detection result in the point cloud; is the confidence score of the i-th detected object in two-dimensional detection; is the confidence score in the point cloud scene; d j Represents the normalized distance from the jth object detected to the ground in the point cloud scene;
[0149] If both the 2D and 3D detection networks correctly detect the same target, the projection of the 3D detection box on the 2D image should have a larger intersection over union ratio with the 2D detection box. If the 2D and 3D detection results are inconsistent and the IoU is 0, then the IoU i , d j All are set to 0;
[0150] The non-empty vector is input into the convolutional neural network and mapped into the target detection box position regression map and the corresponding probability score map through maximum pooling. The final output of the network is the three-dimensional target detection box and its corresponding probability score.
[0151] Among them, the two-dimensional target detection result is expressed as:
[0152]
[0153] Among them, P 2D is the set of two-dimensional target detection results, k represents the number of targets detected in the current two-dimensional image, P i 2D represents the i-th target in the detection result set, P i 2D The first item in ([x i1 ,y i1 ,x i2 ,y i2 ]) is the two-dimensional object detection bounding box, and the second is the confidence score; x i1 ,y i1 ,x i2 ,y i2 are the horizontal and vertical coordinates of the upper left corner vertex and the lower right corner vertex of the two-dimensional target bounding box, respectively. is the confidence score of the two-dimensional target bounding box, and K is the number of currently detected targets;
[0154] The three-dimensional target detection result is expressed as:
[0155]
[0156]
[0157] Among them, P 3D is the set of 3D target detection results, n represents the number of targets detected in the current 3D point cloud, P i 3D represents the i-th target in the detection result set, h i ,w i ,l i ,x i ,y i ,θ i They are the height, width, length, vertex abscissa, vertex ordinate, and rotation angle relative to the z-axis of the 3D target bounding box. is the confidence score of the 3D object bounding box.
[0158] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.
[0159] The execution subjects of the above-mentioned processor 1 and memory 2 can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and are selected according to the needs of actual applications.
[0160] The data signal is transmitted between the memory 2 and the processor 1 via the bus 3, which will not be described in detail in the embodiment of the present invention.
[0161] Example 5
[0162] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.
[0163] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.
[0164] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.
[0165] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.
[0166] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.
[0167] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.
[0168] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A multi-target detection method based on multimodal information fusion, characterized in that: The method comprises the following steps: Process the LiDAR point cloud data, extract the LiDAR point cloud features, and obtain the three-dimensional feature map; Perform two-dimensional image data processing on the RGB image, output the RGB image features through the feature extraction network, and obtain a two-dimensional feature map; The 3D and 2D feature maps are forward-transferred through a convolutional neural network with shared parameters, and the convolutional neural network is trained according to the cross entropy to achieve 3D and 2D feature fusion; The fused 3D feature map is input into the 3D region generation network, which maps the 3D feature map through two branches and outputs the 3D target detection box position map and probability score map respectively. The fused 2D feature map is input into the 2D region generation network, which outputs the 2D target detection box position map and probability score map. At this time, the position and detection probability score map of the target in the point cloud data and RGB 2D image are obtained respectively. The post-fusion strategy is adopted to fuse the target detection boxes under the two modalities to obtain the final target detection result.
2. The multi-target detection method based on multimodal information fusion according to claim 1, characterized in that: The LiDAR point cloud data is processed, the LiDAR point cloud features are extracted, and the three-dimensional feature map is obtained as follows: The point cloud is evenly grouped into several voxels, and the sparse and uneven point cloud is converted into a dense tensor structure. The list of voxel features is obtained by stacking the voxel feature encoding layers. After passing through the convolutional intermediate layer, the voxel features are aggregated in the gradually expanding receptive field to output the LiDAR point cloud features, that is, the three-dimensional feature map.
3. The multi-target detection method based on multimodal information fusion according to claim 1, characterized in that: The two-dimensional image data processing is performed on the RGB image, and the RGB image features are output through the feature extraction network to obtain the two-dimensional feature map. Specifically, Perform uniform grouping operations on the two-dimensional RGB image, and the width and height of each group block are equal to the voxel in LiDAR point cloud data processing; Through the feature extraction network composed of convolutional layers, pooling layers and linear rectification layers, the RGB image features, that is, the two-dimensional feature map, are output.
4. The multi-target detection method based on multimodal information fusion according to claim 2, characterized in that: The conversion of the sparse and uneven point cloud into a dense tensor structure is specifically as follows: The voxel coordinates and the coordinates of the voxel midpoint are used as key-value pairs to build a hash table to realize voxel search and initialization. The voxel input features and coordinate buffer are constructed by traversing the point list once.
5. The multi-target detection method based on multimodal information fusion according to claim 2, characterized in that: The post-fusion strategy is adopted to fuse the target detection frames of the two modalities to obtain the final target detection result as follows: The detection results of the two modes are represented by a mixed representation: Among them, IoU i is the intersection-over-union ratio of the 2D and 3D bounding boxes, which is used to indicate the geometric consistency between the i-th detection result in the image and the j-th detection result in the point cloud; is the confidence score of the i-th detected object in two-dimensional detection; is the confidence score in the point cloud scene; d j Represents the normalized distance from the jth object detected to the ground in the point cloud scene; If both the 2D and 3D detection networks correctly detect the same target, the projection of the 3D detection box on the 2D image should have a larger intersection over union ratio with the 2D detection box. If the 2D and 3D detection results are inconsistent and the IoU is 0, then the IoU i , d j All are set to 0; The non-empty vector is input into the convolutional neural network and mapped into the target detection box position regression map and the corresponding probability score map through maximum pooling. The final output of the network is the three-dimensional target detection box and its corresponding probability score.
6. The multi-target detection method based on multimodal information fusion according to claim 5, characterized in that: The two-dimensional target detection result is expressed as: Among them, P 2D is the set of two-dimensional target detection results, k represents the number of detected targets in the current two-dimensional image, P i 2D represents the i-th target in the detection result set, P i 2D The first item in ([x i1 ,y i1 ,x i2 ,y i2 ]) is the two-dimensional object detection bounding box, and the second is the confidence score; x i1 ,y i1 ,x i2 ,y i2 are the horizontal and vertical coordinates of the upper left corner vertex and the lower right corner vertex of the two-dimensional target bounding box, respectively. is the confidence score of the two-dimensional target bounding box, and K is the number of currently detected targets; The three-dimensional target detection result is expressed as: Among them, P 3D is the set of 3D target detection results, n represents the number of targets detected in the current 3D point cloud, P i 3D represents the i-th target in the detection result set, h i ,w i ,l i ,x i ,y i ,θ i They are the height, width, length, vertex abscissa, vertex ordinate, and rotation angle relative to the z-axis of the 3D target bounding box. is the confidence score of the 3D object bounding box.
7. A multi-target detection device based on multimodal information fusion, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method steps described in any one of claims 1-6.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method steps described in any one of claims 1 to 6.
Citation Information
Patent Citations
Multi-target detection and recognition method based on deep learning
CN112668469A
Multi-mode 3D target detection method and system thereof, terminal and medium
CN112990229A
Point cloud target detection method fusing original point cloud and voxel division
CN113378854A
Voxel-based feature learning network
US10970518B1
