A target detection method based on multi-source information multi-scale fusion

Abstract

The application discloses a target detection method based on multi-source information multi-scale fusion, corresponding data processing is performed on original image data and original point cloud data, multi-scale fusion image feature maps of the original image data are extracted, and point cloud features of the original point cloud data are extracted; a PPFVNet model is constructed; the image features and the point cloud features are modified to the same dimension through dimension matching, then point-to-point splicing is performed to complete fusion operation of the image features and the point cloud features, and features of multi-source information fusion are constructed as input of the PPFVNet model; the fused features are input into the PPFVNet model for training, a classification probability map and a position regression map of a target are obtained, a loss function of a predicted value and an actual value is calculated, and a gradient correction and parameter back propagation are performed, and finally, a target detection result is output. The application has good performance in target recognition accuracy and robustness, and has certain effectiveness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly relates to a multi-source information multi-scale fusion target detection method. BACKGROUND

[0002] In recent years, target detection technology based on image information has developed rapidly and plays a very key role in various fields. However, there are still many problems. On the one hand, using only image data for target detection is easily affected by target deformation, being hidden, diversity, lighting conditions and background factors, resulting in a decrease in target detection accuracy. On the other hand, using only a single scale of features for target detection cannot simultaneously consider the overall information of large targets and the detailed information of small targets, so the target detection model cannot completely learn the target features, resulting in a decrease in target detection performance. Therefore, scholars at home and abroad have conducted in-depth analysis and research on target detection and have made related breakthroughs. Among the most famous and effective target detection methods in existing literature, the main ones include: 1. Faster R-CNN: Real-time object detection based on regional convolutional neural network: In 2015, Ren S, He K, Girshick R, et al. Faster r-cnn: Towards real-time object detection with region proposal networks [J]. Advances in neural information processing systems, 2015, 28. proposed using a region proposal network to extract features using a convolutional neural network on candidate regions and output a set of rectangular proposal boxes to complete target detection, which greatly improved the accuracy and speed of target detection and achieved real-time target detection. 2. Voxel processing network: End-to-end learning for 3D object detection based on point cloud: In 2018, Zhou Y, Tuzel O. Voxelnet: End-to-end learning for point cloud based 3d object detection [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 4490-4499. proposed dividing the point cloud into voxels, then performing random sampling and normalization processing, and encoding the voxel features of each non-empty voxel to obtain the corresponding features. After passing through the 3D convolution layer and the region proposal network, the position and category of the target are finally confirmed.3. Joint 3D Proposal Generation and Object Detection from View Aggregation: Ku J, Mozifian M, Lee J, et al. Joint 3d proposal generation and object detection from view aggregation [C] / / 2018 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2018: 1-8. An encoder-decoder structure in feature pyramid is introduced to extract features of high-resolution point cloud and image, which can simultaneously fuse high-level semantic information and low-level detailed information; and eight corner points needed to describe three-dimensional candidate boxes are reduced to four corner point coordinates for description. The model significantly improves the detection quality of small targets and effectively improves the target detection speed.

[0003] The multi-scale feature fusion network is a feature processing network similar to a capital tower structure, mainly through feature extraction of images of different sizes to obtain feature maps of different sizes, using multi-scale feature maps to obtain different feature information, training the network by fusing features of different scales, so that the semantic information of high-level features and the geometric information of low-level features can be learned. As a method to improve the feature learning ability of neural networks, the multi-scale feature fusion network is widely used in target detection tasks that are difficult to balance large and small targets. The most famous self-supervised learning method in the existing literature mainly includes: 1. Feature pyramid network method for target detection research: Lin T Y, Dollar P, Girshick R, et al. Feature pyramid networks for object detection [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 2117-2125. It is proposed to extract features from the original image to obtain feature maps of different sizes, and to expand the size by upsampling the feature maps, to make the two be fused by side linking and added, and then to perform a prediction task. 2. Path aggregation network method for instance segmentation research: Liu S, Qi L, Qin H, et al. Path aggregation network for instance segmentation [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 8759-8768. It is proposed to add a bottom-up feature fusion layer to enhance the representation ability of the network, to add an adaptive feature pooling layer to fuse the features of each layer to avoid hard matching of feature map size and network layer number, and to add a fully connected fusion layer to increase the mask quality of the network prediction. In this way, the performance of the network model is further improved. SUMMARY

[0004] The application aims to provide a target detection method based on multi-source information multi-scale fusion, which can better improve the target detection precision and robustness.

[0005] The application aims to provide a target detection method based on multi-source information multi-scale fusion, which can better improve the target detection precision and robustness.

[0006] A target detection method based on multi-source information multi-scale fusion, comprising the following steps:

[0007] Step (1) corresponding data processing is performed on the original image data and the original point cloud data, multi-scale fused image features of the original image data are extracted, and point cloud features of the original point cloud data are extracted;

[0008] Step (2) constructing a PPFVNet model;

[0009] Step (3) image features and point cloud features are modified to the same dimension through dimension matching, and then are spliced point by point to complete fusion of the image features and the point cloud features, and multi-source information fused features are constructed as input of the PPFVNet model;

[0010] Step (4) the fused features are input into the PPFVNet model for training, a classification probability map and a position regression map of a target are obtained, a loss function of a predicted value and an actual value is calculated, and gradient correction and parameter back propagation are performed, and finally a target detection result is output.

[0011] In step (1), the extraction process of the image features is as follows:

[0012] Step (1-1) after the original image data is scaled, initial feature maps are obtained through a feature extraction network, and different sizes of the initial feature maps are extracted and fused through a path aggregation network; wherein the feature extraction network is composed of a convolution layer, a ReLU layer and a pooling layer;

[0013] Step (1-2) each three-dimensional point is projected onto the initial feature map using a calibration matrix, and a feature corresponding to the index of the projected position is added to each three-dimensional point, and then corresponding image features are obtained.

[0014] In step (1), the extraction process of the point cloud features is as follows: the point cloud data distributed in a certain space is divided into voxels, the points in the point cloud are grouped according to the voxels, then the voxels containing more than a certain number of points are randomly sampled, and voxel feature coding is performed to obtain point cloud features.

[0015] In step (2), the PPFVNet model includes a voxel feature coding chain layer, a three-dimensional convolution intermediate layer and a three-dimensional region proposal network, the voxel feature coding chain layer combines high-dimensional features and local aggregation features to complete feature enhancement, and the three-dimensional region proposal network corrects the candidate proposal frame and constructs a high-resolution feature map.

[0016] The voxel feature coding chain layer is stacked by a plurality of voxel feature coding layers, for a given non-empty voxel (x i ,y i ,z i ) represents the XYZ coordinates of the i-th point, r iThe reflectivity received by the i-th point, t represents the number of points in the non-empty voxel, T is a random sampling of T points from the voxel containing more than T points, and then the local average value is calculated as the centroid (v x , y , z ) of V, and then a centroid relative offset w, r, t is added to each point to obtain the input feature set

[0017] Each is converted into a feature space by a fully connected layer for gathering information from the point cloud features to encode the surface information contained in the voxel; wherein the fully connected layer consists of a linear layer, a batch normalization layer and a ReLU layer; after obtaining the point-to-point features, the point cloud features f i associated with all non-empty voxels V are subjected to max-pooling to obtain the local aggregation features of the non-empty voxels V Finally, the local aggregation features are used to expand each point cloud feature to form a point-to-point splicing feature:

[0018]

[0019] At this point, the output feature set is also obtained. All non-empty voxels are encoded in the above manner and share parameters in the fully connected layer.

[0020] The three-dimensional region candidate network is composed of three fully convolutional layer blocks, the first layer of each convolutional layer block is a convolution with a step size of 2, followed by a certain number of convolution sequences with a step size of 1, and batch normalization and ReLU activation function operations are performed after each convolutional layer block to downsample the feature map; then the output of each layer block is upsampled to a fixed size and subjected to concave processing to construct a high-resolution feature map; finally, the feature map is mapped to obtain a probability classification and a position regression map.

[0021] In step (3), the image features and the point cloud features are modified to the same dimension through a set of fully connected layers.

[0022] In step (4), the loss function is:

[0023]

[0024] Wherein, α, β are coefficients, N pos is the number of positive class boxes, N neg is the number of negative anchor boxes, i, j are the IDs of the boxes, L cls is the classification loss function, and L reg is the regression loss function. represents the classification loss of positive class boxes with 1, represents the classification loss of negative class anchor boxes with 0, represents the regression loss. i is the predicted regression map, is the real regression map.

[0025] The present application has the following beneficial effects:

[0026] a. In actual application scenarios, the traditional single-source information target detection method based on image information is easily affected by factors such as target deformation, being hidden and covered, self diversity, light and weather conditions, and textured environment, which reduces the detection accuracy of the target detection model. By adding information such as point cloud as supplementary information to the target detection model to form multi-source information input, the advantages of multi-source information are taken and the influence of external environmental factors is reduced. According to the defects of the single-source information target detection model and the characteristics and theoretical advantages of multi-source information fusion, the present application extracts a target detection model based on multi-source information fusion;

[0027] b. In order to solve the sparsity and complexity of point cloud information itself, direct processing of point cloud information will increase the calculation cost and affect the target detection quality. Therefore, the present application selects a voxelization method to process the original point cloud data. By defining the three-dimensional size of each voxel, the point cloud is divided into groups, and then the voxels containing more than a certain number of points are randomly sampled. In this way, not only the calculation amount can be reduced and the running efficiency can be improved, but also the sampling deviation caused by the imbalance of the number of points between voxels can be reduced, and the diversity of training can be ensured;

[0028] c. For the problem of multi-source information fusion, the present application uses a calibration matrix to project each point in the point cloud data onto the image, and through feature indexing, the features of the corresponding projection position in the image are attached to the corresponding points. At this point, whether there is a detection target at each position of the image and the shape information of the detection target are associated with the points in the point cloud. Then, after modifying the point cloud features to the correct dimension through a fully connected layer, the features are sent to a point-to-point splicing module for feature splicing to obtain fused features;

[0029] d.In order to further improve the model target detection performance, the application introduces a path aggregation network in the image information processing process to perform multi-scale feature extraction and fusion, so that the model can simultaneously focus on the overall information and the detail information of the target, and in the subsequent voxel feature encoding chain layer, each point in the voxel can interact with the local aggregation feature, so that the final output feature can describe the shape information contained in the voxel. Subsequently, in the three-dimensional region proposal network, high-resolution feature maps are constructed by performing multiple downsampling and deconvolution operations on the features, which are used for the final target detection. Finally, the category and position information of the corresponding detection target are output to complete the target detection task.

[0030] In summary, the target detection method based on multi-source information multi-scale fusion proposed by the application has good performance in target detection accuracy and robustness, and has certain effectiveness. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a flowchart of the method of the application;

[0032] Figure 2 is a randomly selected image and point cloud information instance diagram in the data set, wherein (a) is an image information instance diagram, (b) is a point cloud information instance diagram, and (c) is a projected fusion information instance diagram;

[0033] Figure 3 is a path aggregation network structure diagram;

[0034] Figure 4 is a PPFVNet model structure diagram proposed by the application;

[0035] Figure 5 is a first layer structure diagram of the voxel feature encoding chain layer;

[0036] Figure 6 is a comparison diagram of the target detection performance of the multi-source information fusion target detection model PFVNet and the target detection model based on image information or point cloud information or fusion information proposed by the application, wherein (a) is a comparison diagram of the target detection accuracy of the KITTI data set on each model, (b) is a comparison diagram of the target detection accuracy of the nuScenes data set on each model, (c) is a comparison diagram of the loss value of the KITTI data set on each model, and (d) is a comparison diagram of the loss value of the nuScenes data set on each model;

[0037] Figure 7Figure is a comparison chart of target detection performance of the multi-source information multi-scale fusion target detection model PPFVNet proposed in the present application and PFVNet and PFVNet model introducing a feature pyramid, wherein (a) is a comparison chart of target detection precision of each model on the KITTI dataset, (b) is a comparison chart of target detection precision of each model on the nuScenes dataset, (c) is a comparison chart of loss values of each model on the KITTI dataset, and (d) is a comparison chart of loss values of each model on the nuScenes dataset. DETAILED DESCRIPTION

[0038] The present application will be further described in detail below in combination with the drawings and examples.

[0039] The present embodiment is illustrated by two datasets, one of which is the KITTI dataset jointly established by the Karlsruhe Institute of Technology in Germany and the Toyota Technical Institute in the United States, which aims to evaluate the performance of computer vision technologies such as optical flow, visual odometry, three-dimensional object detection and three-dimensional target tracking in a vehicle environment. The KITTI dataset contains real data of various scenes, and each image contains multiple types of targets with different degrees of occlusion and truncation. The KITTI dataset contains a total of 389 pairs of stereo images and optical flow images, 39.2 km of visual odometry sequences, and more than 200k images of three-dimensional labeled objects. The other is the nuScenes large-scale dataset for autonomous driving developed by the Motional (formerly nuTonomy) team, which is derived from 1000 driving scenes collected in Boston and Singapore, each scene selects a 20-second long video, a total of about 15 hours of driving data. When selecting scenes, various driving operations, traffic conditions and unexpected situations are fully considered, such as different locations, weather conditions, vehicle types, vegetation, road signs and driving rules, etc.

[0040] Referring to Figure 1 The multi-source information multi-scale fusion target detection method shown in the present embodiment comprises the following specific steps:

[0041] Step (1) performs corresponding data processing on the original image data and the original point cloud data, extracts the multi-scale fusion image feature map of the original image data, and extracts the point cloud feature of the original point cloud data.

[0042] The process of obtaining the image feature is as follows:

[0043] Step (1-1) obtains the initial feature map through the feature extraction network after scaling the original image data as shown in (a), and then obtains the multi-scale fusion image feature map through the multi-scale fusion network as shown in (b). Figure 2 Figure 3 ​The path aggregation network shown extracts and fuses image feature maps of different sizes. The feature extraction network consists of 13 convolutional layers, 13 ReLU layers, and 4 pooling layers. The convolutional layers have a kernel size of 3, padding, and stride of 1. The pooling layers have a kernel size of 2, padding of 0, and stride of 2.

[0044] The formulas for calculating convolution and pooling are:

[0045]

[0046] Where height is the image height, width is the width, the subscripts out and in represent the input and output respectively, kernel represents the convolution kernel size, padding is the padding amount, and stride is the stride.

[0047] According to the formula and the parameters of the convolutional and pooling layers, the size of the feature map does not change after each convolutional layer. After the pooling layer, the width and height of the feature map become half of the original. That is, given an input image of size M×N, after passing through the feature extraction network, an initial feature map of size (M / 16)×(N / 16) will be obtained.

[0048] Steps (1-2) use a calibration matrix to project each 3D point onto the initial feature map, and add features corresponding to the projection position index for each 3D point, thereby obtaining the corresponding image features;

[0049] The point cloud feature extraction process is as follows: for example... Figure 2 The point cloud data distributed in a certain space shown in (b) is divided into spatial voxels, and then the points in the point cloud are grouped according to the voxels. Then, voxels containing more than a certain number of points are randomly sampled, and then voxel feature encoding is performed to obtain point cloud features.

[0050] In the spatial voxel partitioning section, the main task is to divide the point cloud into equally spaced voxels. Given a point cloud distributed in a 3D space with distances of D, H, and W along the Z, Y, and X axes respectively, we define the dimensions of each voxel as vD, vH, and vW. Therefore, the final size of the voxel mesh is:

[0051] D′=D / vD

[0052] H′=H / vH

[0053] W′=W / vW

[0054] For ease of subsequent calculations, we assume that D, H, and W are integer multiples of vD, vH, and vW.

[0055] Step (2) Construct the PPFVNet model.

[0056] Because single-source target detection models based on image data are easily affected by various factors such as device quality, external environment, and the diversity of the target itself, leading to false positives or false negatives and thus reducing the accuracy of target detection, the PPFVNet model is proposed. (See also...) Figure 4 As shown, the PPFVNet model includes a voxel feature encoding chain layer, a 3D convolutional intermediate layer, and a 3D region candidate network. The voxel feature encoding chain layer combines high-dimensional features with local aggregated features to achieve feature enhancement. The 3D region candidate network corrects candidate proposal boxes and constructs high-resolution feature maps.

[0057] ① Voxel feature encoding chain layer

[0058] Voxel feature encoding chain layers are composed of stacked voxel feature encoding layers. For a given non-empty voxel... This represents a non-empty voxel containing t ≤ T points, where t represents the number of points in the non-empty voxel, and T is the maximum value set when randomly sampling T points from voxels containing more than T points to reduce computation. In other words, voxels with more than T points are downsampled to T points, while those with fewer than T points remain unchanged. (x i ,y i ,z i ) represents the XYZ coordinates of the i-th point, r i Let R represent the reflectance received at the i-th point, and let R denote the dimensional space, where the superscript of R indicates the specific number of dimensions. Then, the local average value is calculated as the centroid of V (vi). x ,v y ,v z Then give each point p i Add a relative centroid offset w,r,t to obtain the input feature set. Then, each is connected through a fully connected layer. Transform into a feature space where features from the point cloud can be aggregated. The information is encoded to provide surface information contained within voxels. The fully connected layer consists of a linear layer, a batch normalization layer, and a ReLU layer. Its function is to map the input data from a high dimension to a low dimension and output the result. After obtaining the point-to-point features, all point cloud features f associated with non-empty voxels V are... i Local aggregation characteristics of non-empty voxels V were obtained after max pooling. Finally, local clustering features are used to augment each point cloud feature, forming a point-to-point stitched feature:

[0059]

[0060] Thus, the output feature set has been obtained. All non-empty voxels are encoded in the above manner, and parameters can be shared in the fully connected layer. Because the output features combine point-to-point features and local aggregation features, the voxel feature encoding chain layer can enable point clouds and voxels to interactively encode, and the final output representation can describe the corresponding shape information. Figure 5 is a first layer structure diagram of the voxel feature encoding chain layer.

[0061] ②Three-dimensional region proposal network

[0062] The three-dimensional region candidate network is composed of three fully convolutional layer blocks. The first layer of each convolutional layer block is a convolution with a step size of 2, followed by a certain number of convolution sequences with a step size of 1, and batch normalization and ReLU activation function operations are performed after each convolutional layer block to downsample the feature map. Then the output of each layer block is upsampled to a fixed size and processed with a concave surface, so as to construct a high-resolution feature map. Finally, the feature map is mapped to obtain a probability classification and a position regression map.

[0063] ③Three-dimensional convolution intermediate layer

[0064] The three-dimensional convolution intermediate layer is three three-dimensional convolutions.

[0065] Step (3) modifies the image features and the point cloud features to the same dimension through dimension matching, and then performs point-to-point splicing to complete the fusion of the image features and the point cloud features, and constructs a multi-source information fusion feature as the input of the PPFVNet model.

[0066] Because the dimensions of the image features and the point cloud features obtained in step (1) are different, in this embodiment, a group of fully connected layers is used to reduce the dimensions of the input image features and the point cloud features to 16, and then they are spliced point-to-point to construct a multi-source information fusion feature.

[0067] By combining the semantic information of the image features with the shape information of the point cloud features, a fusion feature with good target detection characteristics and robustness is learned, and a path aggregation network is used to optimize the image features to contain more context information, while taking into account the overall information and detailed information of the target.

[0068] Step (4) inputs the fused feature into the PPFVNet model for training to obtain a classification probability map and a position regression map of the target, then calculates the loss function of the predicted value and the true value and performs gradient correction and parameter back propagation, and finally outputs the target detection result.

[0069] The pre-trained convolution filter is used to calculate the image feature map. The three-dimensional point cloud is projected to the image using the calibration information, and the corresponding image features are attached to the three-dimensional point cloud, the aggregated data is processed by the voxel feature encoding chain layer and the three-dimensional region candidate network to generate the three-dimensional detection. The voxel feature encoding chain layer enables the point cloud to interact with the voxel, and finally outputs the feature that can describe the corresponding shape information. The output feature of the voxel feature encoding chain layer is sent as input into the region proposal network. Convolution, batch normalization and ReLU activation function operations are performed to downsample the feature map.

[0070] The feature map output by the region proposal network is upsampled and concave processed to construct a high-resolution feature map, which is mapped to the desired learning target to obtain a probability classification map and a position regression map, and finally the target detection task is completed.

[0071] Loss function of target detection model PPFVNet based on multi-source information and multi-scale fusion:

[0072] Set is the set of positive anchor boxes N pos , is the set of negative anchor boxes N neg , and the GT box is parameterized as where represents the coordinates of the GT box center in the three-dimensional coordinate system, l g ,w g ,h g represents the length, width and height of the GT box, and θ g represents the yaw angle of the GT box around the z-axis. The parameterization is The loss function calculation formula of the box parameterized as is as follows: where d a is the length of the bottom diagonal of the anchor box: In addition to the loss function of position regression, the loss function of classification prediction needs to be added to the target detection, so the final total loss function is as follows:

[0073] where α and β are coefficients, N pos is the number of positive boxes, N neg is the number of negative anchor boxes, i and j are the IDs of the boxes, L cls is the classification loss function, L reg is the regression loss function, represents the classification loss of the positive box and 1, represents the classification loss of the negative anchor box and 0, represents the regression loss. u i is the predicted regression map, is the true regression map.

[0074] In order to verify the effectiveness of the PPFVNet model, the recognition precision (Precision) and average precision (Average Precision, AP) are used as evaluation indexes. There are four situations when the algorithm performs target detection: TP (True Positive) is that the detector gives a positive sample result, which is actually a positive sample, that is, the target is correctly detected; TN (True Negative) is that the detector gives a negative sample result, which is actually a negative sample, that is, the non-target is correctly detected; FP (False Positive) is that the detector gives a positive sample result, which is actually a negative sample, that is, the detection fails; and FN (False Negative) is that the detector gives a negative sample structure, which is actually a positive sample, that is, the missed detection occurs. The calculation formula of the precision (Precision) is:

[0075]

[0076] The calculation formula of the recall rate is:

[0077]

[0078] The recall rate is taken as the abscissa, and the precision is taken as the ordinate, that is, the precision-recall curve is obtained, and the AP is the area under the curve, which is generally estimated by the 11-point interpolation method or the all-point interpolation method. That is:

[0079]

[0080] Or:

[0081]

[0082] In order to verify the effectiveness of the target detection method based on multi-source information multi-scale fusion proposed in the application, the precision and average precision of the KITTI dataset and the nuScenes dataset are given. Figure 6 is a comparison chart of target detection performance of the multi-source information fusion target detection model PFVNet proposed in the application and the target detection model based on image information or point cloud information or fusion information, wherein Figure 6 (a) is a comparison chart of target detection precision of the KITTI dataset on each model, Figure 6 (b) is a comparison chart of target detection precision of the nuScenes dataset on each model, Figure 6 (c) is a comparison chart of loss value of the KITTI dataset on each model, Figure 6 (d) is a comparison chart of loss value of the nuScenes dataset on each model.

[0083] Table 1. Recognition accuracy of each model on the KITTI dataset

[0084] Model Information source Precision (%) YOLO Image 82.72 Faster-RCNN Image 75.65 VoxelNet Point cloud 83.26 MV3D Image + point cloud 83.67 PFVNet Image + point cloud 86.58

[0085] Table 2. Recognition accuracy of each model on the nuScenes dataset

[0086] Model Information source Precision (%) YOLO Image 81.54 Faster-RCNN Image 77.63 VoxelNet Point cloud 83.98 MV3D Image + point cloud 84.57 PFVNet Image + point cloud 87.12

[0087] From the analysis of the experimental data shown in Tables 1 and 2, it can be seen that the PFVNet model proposed in this paper achieves the best recognition accuracy on both datasets, proving that the model has certain effectiveness in improving target detection accuracy. On the KITTI dataset, the recognition accuracy of PFVNet is 86.58%, which is 3.86% higher than that of the YOLO model, 10.93% higher than that of the Faster-RCNN model, 2.32% higher than that of the VoxelNet model, and 1.91% higher than that of the MV3D model. On the nuScenes dataset, the recognition accuracy of PFVNet is the best among the five experimental models, reaching an accuracy of 87.12%, which is 5.58% higher than that of YOLO, 9.49% higher than that of Faster-RCNN, 2.14% higher than that of VoxelNet, and 1.55% higher than that of MV3D.

[0088] On both datasets, PFVNet has a faster convergence speed and higher accuracy than other models. The better experimental results of PFVNet compared to MV3D prove that the proposed early fusion of image data and point cloud data has certain advantages in target detection through the network structure of voxel feature encoding chain layer and region candidate network. Compared to YOLO and Faster-RCNN models, PFVNet has better experimental results, indicating that mixing point cloud information and image information for target detection tasks has higher target detection accuracy than using only image information for target detection tasks. By exploring the corresponding relationship between the two information sources, the advantages of each information are enhanced, and the disadvantages are made up, proving that multi-source information fusion for target detection is helpful for target detection tasks. Compared to the VoxelNet model, the experimental results are also improved but not as high as single-image information target detection models, which still shows that multi-source information fusion for target detection is better than single-source information performance, and using image geometric information can enhance the feature representation ability of point cloud information, achieving the complementary advantages of multi-source information.

[0089] Table 3. Average precision (IoU = 0.7) of each model on the KITTI dataset

[0090] Model Modality Easy target Medium target Difficult target YOLO Image 86.24 77.55 74.54 Faster-RCNN Image 87.93 79.13 70.98 VoxelNet Point cloud 89.68 84.89 78.63 MV3D Image + point cloud 86.23 77.35 76.37 PFVNet Image + point cloud 89.55 84.98 79.03

[0091] Table 4 Average precision (IoU = 0.7) of each model on the experimental data set under the nuScenes data set

[0092]

[0093]

[0094] On the two experimental data sets, the target detection performance of PFVNet is good for simple, medium and difficult target detection when the intersection over union is set to 0.7. PFVNet still maintains relatively high precision for difficult targets, which shows that the positioning ability of the model can be improved after fusing image information and point cloud information, thereby obtaining better target detection performance. The target detection model using point cloud as input performs better than the target detection model using image as input for difficult targets, which shows that point cloud contains more shape geometric information, and the image target detection model performs better for simple targets, which shows that image contains more semantic information. Therefore, the fusion of the two kinds of information has certain advantages compared with single information.

[0095] In order to verify the effectiveness of the target detection method based on multi-source information multi-scale fusion proposed in the application, the PFVNet, the target detection model introducing the traditional feature pyramid FPN structure and the PPFVNet introducing the path aggregation network are compared and tested on the KITTI data set and the nuScenes data set, Figure 7 Fig. 1 is a comparison chart of target detection performance of the target detection model PPFVNet based on multi-source information multi-scale fusion proposed in the application and the PFVNet and the PFVNet model introducing the feature pyramid, wherein Figure 7 (a) is a comparison chart of target detection precision of each model on the KITTI data set, Figure 7 (b) is a comparison chart of target detection precision of each model on the nuScenes data set, Figure 7 (c) is a comparison chart of loss value of each model on the KITTI data set, Figure 7 (d) is a comparison chart of loss value of each model on the nuScenes data set.

[0096] Table 5 Target recognition precision of each model on the KITTI data set

[0097] Experimental model Precision (%) PFVNet 86.58 PFVNet + FPN 87.81 PPFVNet 89.13

[0098] Table 6 Target recognition precision of each model on the nuScenes data set

[0099]

[0100]

[0101] The experimental data of Table 5 and Table 6 are analyzed. The PFVNet model achieves a target detection accuracy of 86.58% and 87.12% on the KITTI dataset and the nuScenes dataset, respectively, before introducing the multi-scale feature fusion network. By introducing the feature pyramid, the PFVNet+FPN model achieves a target detection accuracy of 87.41% and 87.74% on the two datasets, respectively, which is improved by 1.23% and 0.62%. The PPFVNet model with the introduction of the path aggregation network achieves a target detection accuracy of 89.13% and 88.69% on the two datasets, which is improved by 2.55% and 1.57% compared with the PFVNet model, and is improved by 1.32% and 0.95% compared with the PFVNet+FPN model.

[0102] After introducing the feature pyramid into the PFVNet, the model target detection accuracy and the model convergence speed are improved, which proves that through the multi-scale feature fusion network, the network can learn high-level semantic features and low-level geometric features at the same time, and the context information contained in the rich features makes the network pay attention to the overall image and the edge features of small targets at the same time, improving the model target detection performance. After introducing the PANet as the multi-scale feature fusion network, the model performance is improved compared with the PFVNet model and the PFVNet+FPN model, which proves that the PANet uses one more bottom-up feature fusion layer and adaptive feature pooling layer, which further fuses high-level features and low-level features, so that the features have more rich context information, thus further improving the model target detection accuracy and the model convergence speed.

[0103] Table 5 Average precision (IoU=0.7) of each model in the experimental data set under the KITTI dataset

[0104] Model Easy target Medium target Difficult target PFVNet 89.55 84.98 79.03 PFVNet + FPN 90.64 86.26 80.14 PPFVNet 91.81 87.79 80.92

[0105] Table 6 Average precision (IoU=0.7) of each model in the experimental data set under the nuScenes dataset

[0106] Model Easy target Medium target Difficult target PFVNet 88.58 84.67 78.94 PFVNet + FPN 89.62 86.04 80.06 PPFVNet 90.97 87.22 80.73

[0107] When the intersection over union is set to 0.7, the average precision of the PPFVNet model on the KITTI dataset for simple targets, medium targets and difficult targets is 91.81%, 87.79% and 80.92%, which is respectively increased by 2.26%, 2.81% and 1.89% compared with the PFVNet model, and is respectively increased by 1.17%, 1.53% and 0.78% compared with the PFVNet+FPN model; the average precision of the PPFVNet on the nuScenes dataset for simple targets, medium targets and difficult targets is 90.97%, 87.22% and 80.73%, which is respectively increased by 2.39%, 2.55% and 1.79% compared with the PFVNet model, and is respectively increased by 1.35%, 1.18% and 0.67% compared with the PFVNet+FPN model. It can be proved that by introducing the multi-scale feature fusion network, the context information contained in the features can be enriched, so that the model can pay attention to the overall target and the smaller target at the same time, and the average precision of detection for various difficult detection targets is improved.

[0108] Through the analysis of the above experimental results, it is proved that by introducing the multi-scale feature fusion network into the target detection model, the ability of the model to detect various targets is enhanced. By introducing the multi-scale feature fusion network, the information contained in the rich features is enriched, and the information sharing between the features is strengthened, so that the model can pay attention to the overall target and the small target at the same time, and the target detection performance of the model is improved. The target detection method based on multi-source information multi-scale fusion proposed in the present application has good performance in target detection accuracy and robustness, and has certain effectiveness.

[0109] It can be understood that for those skilled in the art, equivalent replacements or changes can be made according to the technical solutions and inventive concepts of the present application, and all these changes or replacements shall belong to the protection scope of the claims appended to the present application.

Claims

1. A target detection method based on multi-source information multi-scale fusion, characterized in that The method comprises the following steps: Step (1) corresponding data processing is performed on original image data and original point cloud data, multi-scale fused image features of the original image data are extracted, and point cloud features of the original point cloud data are extracted; Step (2) a PPFVNet model is constructed; the PPFVNet model comprises a voxel feature coding chain layer, a three-dimensional convolution intermediate layer and a three-dimensional region candidate network, so as to complete feature enhancement by combining the voxel feature coding chain layer with high-dimensional features and local aggregation features; the three-dimensional region candidate network is used to correct a candidate proposal box and construct a high-resolution feature map; The voxel feature encoding chain layer is stacked by a plurality of voxel feature encoding layers, for a given non-empty voxel (x i ,y i ,z i ) represents the XYZ coordinates of the i-th point, r i is the reflectivity received by the i-th point, t represents the number of points in the non-empty voxel, T is to randomly sample T points from the voxel containing more than T points, and then calculate the local average value as the centroid (v x ,v y ,v z ) of V, and then add a centroid relative offset w, r, t to each point, so as to obtain the input feature set Each of the point cloud features f is converted into a feature space for aggregating information from the point cloud features f to encode the surface information contained in the voxels; wherein the fully connected layer consists of a linear layer, a batch normalization layer and a ReLU layer; after obtaining the point-to-point features, the point cloud features f i associated with the non-empty voxels V are max-pooled to obtain the local aggregated features for the non-empty voxels V Finally, the local aggregated features are used to augment each point cloud feature f to form the point-to-point concatenation features At this point, the output feature set is also obtained All non-empty voxels are encoded in the above manner and share parameters within the fully connected layer; The three-dimensional region candidate network is composed of three fully convolutional layer blocks, the first layer of each convolutional layer block is a convolution with a step of 2, followed by a certain number of convolution sequences with a step of 1, and batch normalization and ReLU activation function operations are performed after each convolutional layer block to perform down-sampling processing on the feature map; then the output of each layer block is up-sampled to a fixed size and subjected to concave processing, so as to construct a high-resolution feature map; finally, the feature map is mapped to obtain a probability classification and a position regression map; Step (3) the image features and the point cloud features are modified to the same dimension through dimension matching, then are spliced point by point to complete fusion of the image features and the point cloud features, and multi-source information fused features are constructed as inputs of the PPFVNet model; Step (4) the fused features are input into the PPFVNet model for training, a classification probability map and a position regression map of a target are obtained, then a loss function of a predicted value and an actual value is calculated, and gradient correction and parameter back propagation are performed, and finally a target detection result is output.

2. The target detection method based on multi-source information multi-scale fusion according to claim 1, characterized in that In step (1), the extraction process of the image features is as follows: Step (1-1) after the original image data is scaled, initial feature maps are obtained through a feature extraction network, and different sizes of the initial feature maps are extracted and fused through a path aggregation network; wherein the feature extraction network is composed of a convolutional layer, a ReLU layer and a pooling layer; Step (1-2) each three-dimensional point is projected onto the initial feature map using a calibration matrix, and features corresponding to the position index of the projection are added to each three-dimensional point, and then corresponding image features are obtained. 3.The target detection method based on multi-source information multi-scale fusion according to claim 1, characterized in that In step (1), the extraction process of the point cloud features is as follows: the point cloud data distributed in a certain space is subjected to spatial voxel division, then the points in the point cloud are grouped according to the voxels, then voxels containing more than a certain number of points are randomly sampled, and finally point cloud features are obtained through voxel feature coding.

4. The target detection method based on multi-source information multi-scale fusion according to claim 1, characterized in that In step (3), the image features and the point cloud features are modified to the same dimension through dimension matching by a group of fully connected layers.

5. The target detection method based on multi-source information multi-scale fusion according to claim 1, characterized in that In step (4), the loss function is: wherein a, b are coefficients, N pos is the number of positive class boxes, N neg is the number of negative class anchor boxes, i, j are the IDs of the boxes, L cls is the classification loss function, L reg is the regression loss function, represents the classification loss of the positive class box with 1, represents the classification loss of the negative class anchor box with 0, represents the regression loss, u i is the predicted regression map, is the real regression map.

Images