A 3D target detection method based on multi-modal data fusion

Abstract

The present invention discloses a three-dimensional target detection method of multi-modal data fusion, which makes full use of image information and mainly solves the problems of point cloud sparsity and difficulty in detecting distant small objects. First, the method adopts voxel feature extraction to encode the points inside the voxel and obtain voxel-level features. Secondly, multi-layer semantic feature extraction and point cloud coloring of the image are performed, the bottom-level semantic features provide precise position information, the high-level semantic features provide semantic information, and the point cloud and image information are fused. Then, voxel feature enhancement is performed, and the voxel neighborhood point cloud features make up for the defect of point cloud information loss in the voxel discretization process. Finally, a dual attention mechanism is introduced, and the channel attention mechanism and the voxel attention mechanism enhance useful feature information and suppress useless feature information. The method of the present invention can solve the problems of point cloud sparsity and difficulty in detecting distant small objects, can enhance useful feature information, suppress useless feature information, and improve target detection performance.

Description

Technical Field

[0001] The invention relates to a three-dimensional target detection method of multi-mode data fusion, belonging to the field of pattern recognition and intelligent information processing. Background Art

[0002] As an important step in visual perception systems, 3D object detection has been widely used in autonomous driving, robotics, virtual reality, and augmented reality. LiDAR sensors are widely used in autonomous driving and robotics to capture 3D scene information due to their ability to directly obtain 3D structural information and accurate depth information of spatial objects. However, due to the sparsity and inhomogeneity of LiDAR point cloud data, it performs poorly in accurate object detection and detection of distant and small objects.

[0003] Most 3D detection methods can be divided into voxel-based methods, multi-view-based methods, and point cloud-based methods according to the different representation methods of 3D point clouds. For voxel methods, point clouds are converted into regularly spaced 3D voxel grids, where each voxel unit contains features such as point cloud coordinates. 3D convolutional networks are used to extract high-order representations of voxels from voxel grids. VoxelNet (Zhou Y, Tuzel O. Voxelnet: End-to-end learning for pointcloud based 3D object detection [C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 4490-4499.) proposed an end-to-end deep learning framework, using the feature extractor VFE layer to learn voxel features. Since point clouds are inherently sparse and uneven, sparse voxel grids bring a lot of redundant calculations. At the same time, VoxelNet stipulates the maximum number of points in each voxel during the voxelization process, which leads to the loss of point feature information when the number of points in the voxel exceeds the limit. Second (Yan Y, Mao Y, Li B. Second: Sparsely embedded convolutional detection [J]. Sensors, 2018, 18 (10): 3337.) proposed sparse convolution to replace the 3D convolution in VoxelNet to extract features, effectively reducing the amount of calculation and improving the reasoning performance. For multi-view methods, Pixor (Yang B, Luo W, Urtasun R. Pixor: Real-time 3d object detection from point clouds [C]. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018: 7652-7660.) projects the point cloud to a bird's-eye view to obtain a dense and compact representation similar to a picture, and then uses a two-dimensional convolutional network to extract point cloud features. However, information loss will occur during the projection discretization process, making it difficult to detect distant objects. For point cloud methods, features are extracted directly from the original point cloud data without causing information loss due to voxelization.PointNet (Qi CR, Su H, Mo K, et al. Pointnet: Deep learning on point sets for 3d classification and segmentation [C]. Proceedings of the IEEE conference on computer vision and pattern recognition, 2017: 652-660.) is an end-to-end deep neural network that directly learns the global features of point clouds from the original point clouds. This method has good results in 3D target recognition, instance segmentation, and semantic segmentation. PointNet++ improves PointNet and can learn local features of point clouds at different scales. However, because it is also based on point cloud data, it also has the problem of poor performance in detecting distant objects due to the sparsity of point clouds. Therefore, the detection performance of the above three methods will deteriorate when detecting distant objects due to the sparsity of point clouds. Summary of the invention

[0004] In order to solve the problem that the existing detection methods are difficult to detect distant and small objects due to the sparsity of point clouds in real scenes, the present application proposes a three-dimensional target detection method with multi-modal data fusion. After obtaining the picture of the real scene and the corresponding point cloud data, the method first uses the feature extractor VFE to convert the voxel features of the point cloud data into a high-order representation; then, the pre-trained faster RCNN network is used to extract the multi-layer semantic features of the picture, and then the point cloud is projected onto the multi-layer semantic features of the picture according to the projection relationship, and the multi-layer semantic features of the picture corresponding to the point cloud are obtained respectively through bilinear interpolation; then, the voxel neighborhood point cloud features are aggregated to make up for the feature information lost in the voxel process; the channel and voxel dual attention mechanism enhances useful features and suppresses useless features; finally, the voxel features are sent to the convolutional network and the region proposal network to realize target detection.

[0005] Optionally, using a KITTI target detection labeling dataset as a picture of a real scene and corresponding point cloud data, the method includes:

[0006] (1) The point cloud data on the KITTI target detection labeling dataset is divided into a spatially uniformly distributed three-dimensional voxel grid. Along the x, y, and z directions, the length, width, and depth ranges of the point cloud data are d, w, and h, respectively; the length, width, and depth of each voxel are defined as v d 、v w 、v h , then the length, width and depth of the final three-dimensional voxel grid are

[0007] (2) constructing a feature extractor, wherein the feature extractor VFE is composed of a fully connected network and a maximum pooling operation, and the superimposed feature extractor encodes each point in the voxel at the voxel level and outputs the VFE feature; the fully connected network is composed of a linear layer, a batch normalization operation, and a ReLU layer;

[0008] (3) Train a faster RCNN network based on ResNet; use the layers 1 to 4 modules of ResNet as multi-layer semantic feature extractors for the image and output the image feature map;

[0009] (4) Using the mapping relationship between point cloud and image, the point cloud is mapped to the image feature map, and the multi-layer semantic feature information of the image corresponding to the point cloud is obtained through interpolation. The image features are converted using a fully connected network, and then the multi-layer semantic features of each point are connected;

[0010] (6) Channel attention explores the importance of different modal features. The input of channel attention is image features, VFE features, and voxel neighborhood enhancement features. Channel attention consists of two linear layers and a sigmoid operation.

[0011] (7) Voxel attention explores the contribution of different voxel features to 3D object detection, enhances useful features, and suppresses useless features. Voxel attention consists of a linear layer and a sigmoid operation.

[0012] (8) The voxel features enhanced by the dual attention mechanism are sent to a set of convolutional intermediate layers to output the voxel features.

[0013] (9) The region proposal network compresses the output features of the convolutional intermediate layer in the H dimension, obtains the feature map through downsampling and upsampling operations, connects the upsampled features, and outputs the final feature map

[0014] (10) The final feature map is sent to the classifier and regressor to obtain the type and position of the target to be detected, the size and direction of the target respectively.

[0015] Optionally, the feature extractor is composed of two superimposed feature extractors VFE1 and VFE2, and the (2) constructing the feature extractor includes:

[0016] 2.1 Definition is a non-empty voxel, which contains 5 points, where x i ,y i ,z i For point P i The three-dimensional coordinates, r i For point P iReflection value, the features of a point are the three-dimensional coordinates and reflection value;

[0017] 2.2 Calculate the coordinate mean of all points in the non-empty voxel V as the cluster center of voxel V (c x ,c y ,c z ), and the coordinates of the voxel in the three-dimensional voxel grid are used as the coordinates V of the voxel V center (v x ,v y ,v z ), and then calculate the relative position of each point in voxel V to the cluster center and voxel coordinate V center The coordinates of the points are offset and the features of the points are connected to obtain the feature extractor VFE input V in :

[0018]

[0019] 2.2 Input Features V in The feature dimension is increased and converted to the feature space through the fully connected network At this time, the feature dimension of all points in the non-empty voxel V is increased to m-dimensional, m = 64; then by aggregating V feature The feature information encodes the surface shape of the three-dimensional object represented by each voxel, that is, feature Perform maximum pooling to obtain the local aggregation features of voxel V

[0020] 2.3 The local aggregation feature V of the non-empty voxel V obtained in 2.2 local With V feature Connect to enhance the features of all points in V; get the output V of the feature extractor VFE out :

[0021]

[0022] T represents the number of points contained in the voxel.

[0023] Optionally, the layer 1 to layer 4 modules of ResNet are used as a multi-layer semantic feature extractor of the image to output a feature map of the image, including:

[0024] The input image dimensions include the width, height and channels of the image. Resnet layers 1 to 4 increase the image channel dimensions to 256, 512, 1024 and 2048 respectively, reducing the image size by 2 times. Feature pyramid network 1 to feature pyramid network 5 unify different channel dimensions to the same dimension and output multi-layer semantic features of the image.

[0025] Optionally, the mapping relationship between the point cloud and the image is used to map the point cloud to the image feature map, the multi-layer semantic feature information of the image corresponding to the point cloud is obtained by interpolation, the image features are converted using a fully connected network, and then the multi-layer semantic features of each point are connected, including:

[0026] For a point P (x, y, z, r) in the point cloud, point P is mapped to the image and the corresponding multi-layer semantic features are obtained as follows:

[0027] Use a fully connected network to transform the image features, output dimension 128, then connect the multi-layer semantic features of each point, further transform the connected features, and finally output dimension 128. The image feature of the final point P is

[0028] For a voxel, by coloring the point cloud in the voxel, the image features corresponding to T points are 128-dimensional, and the image features of T points in the voxel are aggregated to obtain the voxel-level features. At this time, the voxel image features are (D, W, H, 128);

[0029] For voxel V, coordinate V center , the local features of K points in its neighborhood are obtained in the point cloud to make up for the feature information lost in the voxel process.

[0030] Optionally, the voxel V coordinate V center , obtain the local features of K points in its neighborhood in the point cloud to make up for the feature information lost in the voxel process, including:

[0031] Given a point cloud {P t =(x t ,y t ,z t ,r t )|t=1,…,N}, where (x t ,y t ,z t ) is point P t The world coordinates, r t For point P t Reflectance value feature; for voxel coordinates Find at most K points in a neighborhood with a radius no greater than r, and get the voxel coordinates Neighborhood set of

[0032]

[0033] in, for World coordinates, coord j For point Pj World coordinates, coordinate offset With point cloud features j Connecting is done to indicate the local relative positions of point cloud features;

[0034] For voxel V i , the neighborhood radius is r, and PointNet is used to generate its local features

[0035]

[0036] Among them, Q is the neighborhood set The random sampling of is used to keep at most K neighborhood points, M is a superimposed full convolutional network that encodes the features and local positions of the points, and max is the maximum pooling operation to ensure that the feature dimensions of all voxels are consistent;

[0037] The stacked fully convolutional network consists of two fully convolutional networks, each of which consists of a two-dimensional convolution, a batch normalization, and a ReLU operation. The convolution kernel size of the two-dimensional convolution is 1 and the output dimension is 32.

[0038] Set two different radii r and j, and The feature dimensions of both are 32, and the dimension of the voxel feature obtained by connecting the two features is 64;

[0039] At this time, after all voxel features are enhanced, their features are (D, W, H, 64).

[0040] Optionally, the two linear layers in the channel attention are Linear1(256,32) and Linear2(32,256). Linear1(256,32) means the input is (D×W×H, 256) and the output is (D×W×H, 32). The input is 256-dimensional and the output is 32-dimensional. The same is true for Linear2(32,256).

[0041] Optionally, the input and output of the linear layer in the voxel attention are 259 and 1 respectively.

[0042] Optionally, the region proposal network compresses the output features of the convolutional intermediate layer in the H dimension, obtains a feature map through downsampling and upsampling operations, connects the upsampled features, and outputs the final feature map include:

[0043] The convolutional middle layer consists of 5 convolutional modules. The first convolutional module is Block1 (SubMConv3d(), SubMConv3d()): It consists of two sub-population convolutional modules, where the sub-population convolutional module consists of a sub-population convolution, a batch normalization and a ReLU operation, and the sub-population convolution input dimension is 256 and the output dimension is 16; the second to fourth convolutional modules are composed of a sparse convolutional module and two sub-population convolutional modules, where the sparse convolutional module consists of a sparse convolution, a batch normalization and a ReLU operation; the output dimensions of the second to fourth convolutional modules are 32, 64, 64 respectively; the fifth convolutional module consists of a sparse convolutional module, the sparse convolution step size is (1, 1, 2), and the output dimension is 128; the voxel features after the convolutional middle layer become

[0044] The region proposal network consists of two downsampling modules and two upsampling modules; each downsampling module consists of six full convolutional networks Conv2d. Except for the first full convolutional network in the second downsampling module, the step size of the remaining full convolutional networks is 1. The input and output dimensions of the last five full convolutional networks of each downsampling module are consistent. The output dimensions of the two downsampling modules are 128 and 256 respectively; each full convolutional network contains a two-dimensional convolution, a batch normalization and a ReLU operation; the upsampling module increases the output of the two downsampling modules to 256 to maintain the high resolution of the feature map; the output dimensions of the two upsampling modules are connected to obtain the final feature map

[0045] Optionally, the final feature map is fed into a classifier and a regressor to obtain the type and position of the target to be detected, the size and direction of the target, respectively, including:

[0046] The classifier is a two-dimensional convolution conv2d(512, 18), with an input of 512 dimensions and an output of 18 dimensions, and the output is the type of the target; the regressor is a two-dimensional convolution conv2d(512, 42), with an input of 512 dimensions and an output of 42 dimensions, and the output is the three-dimensional world coordinates, length, width, height, and yaw angle relative to the z-axis of the target box.

[0047] The beneficial effects of the present invention are:

[0048] When performing three-dimensional target detection, the present application first uses a feature extractor VFE to convert voxel features into high-order representations; then, uses a pre-trained faster RCNN network to extract image semantic features, and then projects the three-dimensional point cloud onto a multi-layer semantic feature map of the image based on the projection relationship, and obtains the features corresponding to the point cloud through bilinear interpolation; then, aggregates the voxel neighborhood point cloud features to make up for the feature information lost in the voxel process; at the same time, uses a channel and voxel dual attention mechanism to enhance useful features and suppress useless features, thereby solving the problem of poor detection performance for distant objects due to the sparsity of point clouds in the existing three-dimensional target detection process, and effectively improving the detection performance for distant, small and difficult objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is an implementation flow chart of Example 1.

[0051] Figure 2 Schematic diagram of the main structure of the image feature residual network in the method of the present invention in Example 1.

[0052] Figure 3 This is a schematic diagram of the main structure of obtaining image features through point cloud mapping in the method of the present invention in Example 1.

[0053] Figure 4 Schematic diagram of the main structure of channel attention in the method of the present invention in Example 1.

[0054] Figure 5 Schematic diagram of the main structure of voxel attention in the method of the present invention in Example 1.

[0055] Figure 6 Schematic diagram of the main structure of the convolutional intermediate layer in the method of the present invention in Example 1.

[0056] Figure 7 Schematic diagram of the main structure of the region proposal network in the method of the present invention in Example 1.

[0057] Figure 8 This is a qualitative analysis comparison diagram of the method of the present invention in Example 1 and the VoxelNet comparison method. DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0059] To facilitate understanding of the technical solution of the present invention, the basic theory of the technical solution of the present invention is first introduced as follows:

[0060] 1. VoxelNet network

[0061] The main components of the VoxelNet network are (1) feature extractor, (2) convolutional intermediate layer and (3) region proposal network. The main idea is to connect the points in each voxel with the local aggregated features through the feature extractor, obtain the interaction between the points inside the voxel, and then obtain the voxel-level features. The superimposed feature extractor VFE enables each voxel to learn the complex features of local three-dimensional information. Then the convolutional intermediate layer further aggregates the local voxel features, expands its receptive field, and converts the point cloud to a high-dimensional quantitative representation. Finally, the region proposal network uses the voxel features to generate detection results.

[0062] Since the output features of the feature extractor include point-level features and voxel-level local aggregation features, the superimposed feature extractor can further encode the relationship between points within the voxel, so that the final feature representation can learn the surface structure information of the three-dimensional object. The output of the superimposed feature extractor is fed into a fully connected network, and the voxel-level features are obtained using maximum pooling.

[0063] 2. Loss Function

[0064] The loss function is used to evaluate the difference between the model's predicted value and the true value, and is also the objective function optimized in the neural network. The process of neural network training or optimization is the process of minimizing the loss function. The smaller the loss function, the closer the model's predicted value is to the true value, and the better the model's accuracy. The closer the model's predicted output for the sample is to the true sample label 0, the smaller the loss value L; the closer the prediction is to 1, the larger L.

[0065] Embodiment 1:

[0066] This embodiment provides a three-dimensional target detection method for multi-mode data fusion, see Figure 1 , the method comprising:

[0067] (1) The point cloud data on the KITTI target detection labeling dataset is divided into a spatially uniformly distributed three-dimensional voxel grid. Along the x, y, and z directions, the length, width, and depth ranges of the point cloud data are d, w, and h, respectively. The length, width, and depth of each voxel are defined as v d 、v w 、v h , then the length, width and depth of the final three-dimensional voxel grid are

[0068] (2) Build a feature extractor and define is a non-empty voxel containing 5 points, where x i ,y i ,z i For point P i The three-dimensional coordinates, r i For point P i Reflection value, the features of a point are the three-dimensional coordinates and the reflection value.

[0069] First, the coordinate mean of all points in voxel V is calculated as the cluster center of voxel V (c x ,c y ,c z ), and the coordinates of the voxel in the three-dimensional voxel grid are used as the coordinates V of the voxel V center (v x ,v y ,v z ), and then calculate the relative position of each point in voxel V to the cluster center and voxel coordinate V center The coordinate offset is calculated and the features of the points are connected to obtain the feature extractor VFE input:

[0070]

[0071] Input feature V in The feature dimension is increased and converted to the feature space through the fully connected network At this time, the feature dimension of all points in voxel V is increased to m dimension, m = 64. Then, by aggregating V feature The feature information encodes the surface shape of the three-dimensional object represented by each voxel, that is, feature Perform maximum pooling to obtain the local aggregation features of voxel V

[0072] Finally, the local aggregate feature V of voxel V is local With V feature Connect to enhance the features of all points in V. So far, the output of the feature extractor VFE is

[0073] This application uses two superimposed feature extractors VFE1 and VFE2 to extract features from voxels, and the output of VFE1 is used as the input of VFE2. The two superimposed feature extractors are VFE1(10,128) and VFE2(128,64), that is, the input of VFE1 is (T,10), and the output is (T,128), T is the default value 5, representing the number of points inside the voxel V, the input feature dimension is 10, and the output feature dimension is 128, and the same is true for VFE2(128,64);

[0074] (3) Train a resnet-based Faster RCNN network, which is first trained on the COCO dataset and then fine-tuned on the KITTI dataset. Figure 2 As shown in the figure, the input image dimensions are width, height and channel. Resnet layers 1 to 4 increase the image channel dimensions to 256, 512, 1024 and 2048 respectively, reducing the image size by 2 times. Feature pyramid network 1 to feature pyramid network 5 unify different channel dimensions to the same dimension and output multi-layer semantic features of the image.

[0075] (4) Based on Figure 3 As shown in the figure, the mapping relationship between point cloud and image is used to map the point cloud to the image feature map, and the multi-layer semantic feature information of the image corresponding to the point cloud is obtained by interpolation. For a point P (x, y, z, r) in the point cloud, point P is mapped to the image, and the corresponding multi-layer semantic feature is obtained as follows:

[0076] Use a fully connected network to transform the image features, output dimension 128, then connect the multi-layer semantic features of each point, further transform the connected features, and finally output dimension 128. The image feature of the final point P is For a voxel, by coloring the point cloud in the voxel, the image features corresponding to T points are 128-dimensional, and the image features of T points in the voxel are aggregated to obtain the voxel-level features. At this time, the voxel image features are (D, W, H, 128);

[0077] (5) For voxel V coordinate V center , obtain the local features of K points in its neighborhood in the point cloud to make up for the feature information lost in the voxel process. The specific steps are:

[0078] Given a point cloud {P t =(x t ,y t ,z t ,r t )|t=1,…,N}, where (x t ,y t ,z t ) is point Pt The world coordinates, r t For point P t Reflectance value feature. For voxel coordinates Find at most K points in a neighborhood with a radius no greater than r, and get the voxel coordinates Neighborhood set of

[0079]

[0080] in, for World coordinates, coord j For point P j World coordinates, coordinate offset With point cloud features j Connections are made to indicate the local relative positions of point cloud features.

[0081] For voxel V i , the neighborhood radius is r, and PointNet is used to generate its local features

[0082]

[0083] Among them, Q is the neighborhood set SV centeri The random sampling of is used to keep at most K neighborhood points. M is a stacked full convolutional network that encodes the features and local positions of the points. max is the maximum pooling operation to ensure that the dimensions of all voxel features are consistent. The stacked full convolutional network consists of two full convolutional networks, each of which consists of a two-dimensional convolution (the convolution kernel size is 1 and the output dimension is 32), a batch normalization, and a ReLU operation, that is, The feature dimension of is 32. At the same time, multi-scale neighborhood information is used to aggregate local features of different receptive fields, so as to obtain contextual information of different sizes with richer voxels. Two different radii r and j are set, and The feature dimensions of are all 32, and the dimension of the voxel feature obtained by connecting the two features is 64. At this time, after all voxel features are enhanced, their features are (D, W, H, 64).

[0084] (6) Channel attention explores the importance of different modal features;

[0085] according to Figure 4As shown in the figure, the channel attention input is the image feature, VFE feature and voxel neighborhood enhancement feature. The two linear layers are Linear1(256,32) and Linear2(32,256). Linear1(256,32) means the input is (D×W×H, 256) and the output is (D×W×H, 32). The input is 256-dimensional and the output is 32-dimensional. Similarly, Linear2(32,256) has '+' as the concatenation operation and '×' as the multiplication of the corresponding elements.

[0086] (7) Voxel attention explores the contribution of different voxel features to 3D target detection, enhances useful features, and suppresses useless features. Before being sent to the voxel attention, the voxel features are connected to the voxel 3D coordinates to provide accurate 3D object location information. At this time, the input of the voxel attention is (D, W, H, 259). Figure 5 As shown, the voxel attention consists of a linear layer and a sigmoid operation, where the input and output of the linear layer are 259 and 1 respectively, '+' is a connection operation, and '×' is the multiplication of corresponding elements;

[0087] (8) The voxel features (D, W, H, 256) enhanced by the dual attention mechanism are sent to a set of convolutional intermediate layers. The three-dimensional convolution further expands the receptive field of each voxel feature and obtains semantic information to improve the target detection performance.

[0088] according to Figure 6 As shown in the figure, the convolutional middle layer consists of 5 convolutional modules. The first convolutional module is Block1 (SubMConv3d(), SubMConv3d()): It consists of two sub-population convolutional modules, where the sub-population convolutional module consists of a sub-population convolution, a batch normalization and a ReLU operation. The sub-population convolution input dimension is 256 and the output dimension is 16. The second to fourth convolutional modules are composed of a sparse convolutional module and two sub-population convolutional modules, where the sparse convolutional module consists of a sparse convolution (with a step size of 2), a batch normalization and a ReLU operation. For example, the second convolutional module is composed as follows: Block2 (SparseConv3d(), SubMConv3d(), SubMConv3d()). The output dimensions of the second to fourth convolutional modules are 32, 64, and 64 respectively. The fifth convolutional module consists of a sparse convolutional module, the sparse convolution step size is (1, 1, 2), and the output dimension is 128. The output dimension of each module is represented by an orange number. Unless otherwise specified, the stride of the popular convolution module is 1. The voxel features after the convolution intermediate layer become

[0089] (9) The region proposal network compresses the output features of the convolutional intermediate layer in the H dimension. At this time, the voxel feature is And obtain the feature map through downsampling and upsampling operations. Figure 7 As shown in the figure, the region proposal network consists of two downsampling modules and two upsampling modules. Each downsampling module consists of six full convolution networks Conv2d. Except for the first full convolution network in the second downsampling module, the step size of the remaining full convolution networks is 1. The input and output dimensions of the last five full convolution networks of each downsampling module are consistent. The output dimensions of the two downsampling modules are 128 and 256 respectively. Each full convolution network contains a two-dimensional convolution, a batch normalization and a ReLU operation. The upsampling module increases the output of the two downsampling modules to 256 to maintain the high resolution of the feature map. The output dimensions of the two upsampling modules are connected to obtain the final feature map:

[0090] (10) The feature map is fed into the classifier and regressor to generate the type and position of the target, the size of the target, and the direction of the target, respectively. The classifier is a two-dimensional convolution conv2d(512, 18), with an input of 512 dimensions and an output of 18 dimensions, that is, the output is the target type. The regressor is a two-dimensional convolution conv2d(512, 42), with an input of 512 dimensions and an output of 42 dimensions, that is, the output is the three-dimensional world coordinates, length, width, height, and yaw angle relative to the z-axis of the target box.

[0091] In order to verify the performance of the proposed method in accurate object detection and distant small object detection, the following simulation experiments are conducted: 1.

[0092] Simulation conditions and parameters

[0093] For the kitti dataset, the point cloud distance ranges in the x, y, and z directions are [0m, 70.4m], [-40m, 40m], and [-3m, 1m], respectively. The voxel size is set to (0.05m, 0.05m, 0.1m), and the point cloud voxel structure consists of 1600×1408×40 voxels. Two anchor boxes with different angles along the x direction are set, which are 0° and 90° respectively. For cars, pedestrians, and cyclists, the anchor box sizes are set to (3.9m, 1.6m, 1.56m), (0.8m, 0.6m, 1.73m), and (1.76m, 0.6m, 1.73m), respectively. The network optimizer uses the ADAM optimizer with a one_cycle strategy, with a maximum learning rate of 0.003, a weight decay of 0.01, a momentum of 0.9, a batch size of 1, and 100 iterations.

[0094] The experimental environment is RTX2080Ti graphics card, the deep learning framework is Pytorch, and the training time is about 44 hours. For data enhancement, since point cloud and image multimodal data are used at the same time, the data enhancement of point cloud needs to be consistent with the image data enhancement. Therefore, this experimental data enhancement uses x-axis random flipping, global rotation and scaling, and does not use target true value sampling data enhancement.

[0095] 2. Simulation content and result analysis

[0096] In order to verify the detection performance of the method described in Example 1 when detecting distant and small objects, this application conducts comparative experiments on the Kitti public dataset. The comparative algorithms are:

[0097] MV3D, please refer to Chen

[0098] PIXOR, please refer to Chen

[0099] F-PointNet, please refer to Qi CR, Liu W, Wu C, et al. Frustum pointnets for 3dobject detection from rgb-d data[C]. Proceedings of the IEEE conference on computer vision and pattern recognition, 2018:918-927.

[0100] VoxelNet, please refer to Zhou Y, Tuzel O. Voxelnet: End-to-end learning for pointcloud based 3d object detection [C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 4490-4499.

[0101] For MVX-Net and MVX-Net, please refer to Sindagi VA, Zhou Y, Tuzel O. MVX-Net: Multimodalvoxelnet for3D object detection[C]. 2019 International Conference on Robotics and Automation (ICRA), 2019: 7276-7282. and other commonly used target detection algorithms.

[0102] The comparison process uses qualitative analysis and quantitative analysis. Qualitative analysis directly observes the pros and cons of the results through visual comparison of the target detection result graph; quantitative analysis judges the performance of the algorithm by calculating evaluation indicators. The indicators used in the experiment are mAP.

[0103] Tables 1 and 2 show the mAP scores of the proposed method and the benchmark method VoxelNet in the three-dimensional pattern index and the bird's-eye view index of the kitti validation set, respectively. The higher the mAP score, the better the detection performance, and the lower the mAP score, the worse the detection performance.

[0104] In the three-dimensional mode, compared with the baseline method VoxelNet, the proposed method improves the difficult detection indicators of cars, pedestrians and cyclists by 5.49%, 2.53% and 1.31% respectively, and improves the medium detection indicators by 8.78%, 3.55% and 3.85% respectively.

[0105] It is worth noting that the performance of the proposed method is greatly improved in both the bird's-eye view and the three-dimensional mode difficulty index. Similarly, the performance of the proposed method is better than the recent well-performing methods. Tables 3 and 4 show the experimental results of the proposed method in the automotive category compared with various single-modal and multi-modal methods. Compared with the multi-modal method MV3D, the proposed method is improved by 13.8%, 11.41% and 13.96% on the three-dimensional mode validation set, respectively. Compared with F-PointNet, the proposed method is improved by 3.34% and 4.64% on the medium and difficult indicators. Compared with MVX-Net, the proposed method is improved by 0.94% on both the medium and difficult indicators. In the results of the three-dimensional mode validation set and the bird's-eye view validation set, the performance of the proposed method is greatly improved compared with various single-modal and multi-modal detection methods. The experimental results prove the effectiveness of the proposed method, which greatly improves the deficiency of information loss in the voxel process of the benchmark method VoxelNet and improves the detection performance of distant and small difficult objects.

[0106] Table 1 Results of the 3D model validation set

[0107]

[0108] Table 2 Results of the bird's-eye view mode validation set

[0109]

[0110] Table 3 Results of the automobile 3D model verification set

[0111]

[0112] Table 4. Results of the validation set of the bird's-eye view model for cars

[0113]

[0114] Figure 8 The first row represents the label data of the scene, including six cars nearby, three cars in the distance, and a severely occluded car (marked by the box in the upper right corner). The second row represents the detection results of the benchmark method VoxelNet. The six cars in the vicinity are unobstructed and have rich point cloud information, so VoxelNet successfully detected them. The two farthest cars are partially occluded and have sparse point clouds, so VoxelNet missed the two cars (marked by the box in the upper right corner). The third row is the detection result of the method of the present invention. The method of the present invention successfully detected all the successful detection results of VoxelNet, including the six cars in the vicinity and a car in the distance. It is worth noting that the method of the present invention successfully detected the two partially occluded cars in the distance that were missed by the VoxelNet method.

[0115] From the above analysis, it can be seen that the method of the present invention can effectively detect the target object in the complex environment of long-distance objects and small objects. The detection performance is better than all the comparison algorithms, and it can complete the three-dimensional target detection task in the complex environment, and the comprehensive performance is significantly improved.

[0116] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A three-dimensional target detection method based on multi-modal data fusion, characterized in that: After obtaining the image of the real scene and the corresponding point cloud data, the method first uses the feature extractor VFE to convert the voxel features of the point cloud data into a high-order representation; then, the pre-trained Faster RCNN network is used to extract the multi-layer semantic features of the image, and then the point cloud is projected onto the multi-layer semantic features of the image according to the projection relationship, and the multi-layer semantic features of the image corresponding to the point cloud are obtained respectively through bilinear interpolation; then, the feature information lost in the voxel process is compensated by aggregating the voxel neighborhood point cloud features; the channel and voxel dual attention mechanism enhances useful features and suppresses useless features; finally, the voxel features are sent to the convolutional network and the region proposal network to realize target detection; The method of aggregating voxel neighborhood point cloud features to compensate for feature information lost in the voxel process specifically includes: given a point cloud {P t =(x t ,y t ,z t ,r t )|t=1,…,N}, where (x t ,y t ,z t ) is point P t The world coordinates, r t Point P t Reflectance value feature; for voxel coordinates Find at most K points in a neighborhood with a radius no greater than r, and get the voxel coordinates Neighborhood set of in, for World coordinates, coord j Point P j World coordinates, coordinate offset With point cloud features j Connecting is done to indicate the local relative positions of point cloud features; For voxel V i , the neighborhood radius is r, and PointNet is used to generate its local features Among them, Q is the neighborhood set The random sampling of is used to keep at most K neighborhood points, M is a superimposed full convolutional network that encodes the features and local positions of the points, and max is the maximum pooling operation to ensure that the feature dimensions of all voxels are consistent; The stacked fully convolutional network consists of two fully convolutional networks, each of which consists of a two-dimensional convolution, a batch normalization, and a ReLU operation. The convolution kernel size of the two-dimensional convolution is 1 and the output dimension is 32. Set two different radii r and j, and The feature dimensions of both are 32, and the dimension of the voxel feature obtained by connecting the two features is 64; At this time, after all voxel features are enhanced, their features are (D, W, H, 64); The channel and voxel dual attention mechanism enhances useful features and suppresses useless features. The specific steps include: Channel attention explores the importance of different modal features. The input of channel attention is image features, VFE features and voxel neighborhood enhancement features. Channel attention consists of two linear layers and a sigmoid operation. Voxel attention explores the contribution of different voxel features to 3D object detection, enhances useful features, and suppresses useless features. Voxel attention consists of a linear layer and a sigmoid operation.

2. The method according to claim 1, characterized in that Using the KITTI target detection label data set as a picture of a real scene and corresponding point cloud data, the method includes: (1) The point cloud data on the KITTI target detection labeling dataset is divided into a spatially uniformly distributed three-dimensional voxel grid. Along the x, y, and z directions, the length, width, and depth ranges of the point cloud data are d, w, and h, respectively; the length, width, and depth of each voxel are defined as v d 、v w 、v h , then the length, width and depth of the final three-dimensional voxel grid are (2) constructing a feature extractor, wherein the feature extractor VFE is composed of a fully connected network and a maximum pooling operation, and the superimposed feature extractor encodes each point in the voxel at the voxel level and outputs the VFE feature; the fully connected network is composed of a linear layer, a batch normalization operation, and a ReLU layer; (3) Train a faster RCNN network based on ResNet; use the layers 1 to 4 modules of ResNet as multi-layer semantic feature extractors for the image and output the image feature map; (4) Using the mapping relationship between point cloud and image, the point cloud is mapped to the image feature map, and the multi-layer semantic feature information of the image corresponding to the point cloud is obtained through interpolation. The image features are converted using a fully connected network, and then the multi-layer semantic features of each point are connected; (5) For voxel V coordinate V center , obtain the local features of K points in its neighborhood in the point cloud to make up for the feature information lost in the voxel process; (6) Channel attention explores the importance of different modal features. The input of channel attention is image features, VFE features, and voxel neighborhood enhancement features. Channel attention consists of two linear layers and a sigmoid operation. (7) Voxel attention explores the contribution of different voxel features to 3D object detection, enhances useful features, and suppresses useless features. Voxel attention consists of a linear layer and a sigmoid operation. (8) The voxel features enhanced by the dual attention mechanism are sent to a set of convolutional intermediate layers to output the voxel features. (9) The region proposal network compresses the output features of the convolutional intermediate layer in the H dimension, obtains the feature map through downsampling and upsampling operations, connects the upsampled features, and outputs the final feature map (10) The final feature map is sent to the classifier and regressor to obtain the type and position of the target to be detected, the size and direction of the target respectively.

3. The method according to claim 2, characterized in that The feature extractor is composed of two superimposed feature extractors VFE1 and VFE2. The (2) constructing the feature extractor includes: 2.1 Definition is a non-empty voxel, which contains 5 points, where x i ,y i ,z i Point P i The three-dimensional coordinates, r i Point P i Reflection value, the features of a point are the three-dimensional coordinates and reflection value; 2.2 Calculate the coordinate mean of all points in the non-empty voxel V as the cluster center of voxel V (c x ,c y ,c z ), and the coordinates of the voxel in the three-dimensional voxel grid are used as the coordinates V of the voxel V center (v x ,v y ,v z ), and then calculate the relative position of each point in voxel V to the cluster center and voxel coordinate V center The coordinates of the points are offset and the features of the points are connected to obtain the feature extractor VFE input V in : 2.2 Input Features V in The feature dimension is increased and converted to the feature space through the fully connected network At this time, the feature dimension of all points in the non-empty voxel V is increased to m-dimensional, m = 64; then by aggregating V feature The feature information encodes the surface shape of the three-dimensional object represented by each voxel, that is, feature Perform maximum pooling to obtain the local aggregation features of voxel V 2.3 The local aggregation feature V of the non-empty voxel V obtained in 2.2 local With V feature Connect to enhance the features of all points in V; get the output V of the feature extractor VFE out : T represents the number of points contained in the voxel.

4. The method according to claim 3, characterized in that The layer 1 to layer 4 modules of ResNet are used as multi-layer semantic feature extractors of the image to output the image feature map, including: The input image dimensions include the width, height and channels of the image. Resnet layers 1 to 4 increase the image channel dimensions to 256, 512, 1024 and 2048 respectively, reducing the image size by 2 times. Feature pyramid network 1 to feature pyramid network 5 unify different channel dimensions to the same dimension and output multi-layer semantic features of the image.

5. The method according to claim 4, characterized in that The method utilizes the mapping relationship between the point cloud and the image to map the point cloud to the image feature map, obtains the multi-layer semantic feature information of the image corresponding to the point cloud by interpolation, converts the image features using a fully connected network, and then connects the multi-layer semantic features of each point, including: For a point P (x, y, z, r) in the point cloud, point P is mapped to the image and the corresponding multi-layer semantic features are obtained as follows: Use a fully connected network to transform the image features, output dimension 128, then connect the multi-layer semantic features of each point, further transform the connected features, and finally output dimension 128. The image feature of the final point P is For a voxel, by coloring the point cloud in the voxel, the image features corresponding to T points are 128-dimensional, and the image features of T points in the voxel are aggregated to obtain the voxel-level features. At this time, the voxel image features are (D, W, H, 128); For voxel V, coordinate V center , the local features of K points in its neighborhood are obtained in the point cloud to make up for the feature information lost in the voxel process.

6. The method according to claim 5, characterized in that The two linear layers in the channel attention are Linear1(256,32) and Linear2(32,256). Linear1(256,32) means the input is (D×W×H, 256) and the output is (D×W×H, 32). The input is 256-dimensional and the output is 32-dimensional. The same is true for Linear2(32,256).

7. The method according to claim 6, characterized in that The input and output of the linear layer in the voxel attention are 259 and 1 respectively.

8. The method according to claim 7, characterized in that The region proposal network compresses the output features of the convolutional intermediate layer in the H dimension, obtains the feature map through downsampling and upsampling operations, connects the upsampled features, and outputs the final feature map include: The convolutional middle layer consists of 5 convolutional modules. The first convolutional module is Block1 (SubMConv3d(), SubMConv3d()): It consists of two sub-population convolutional modules, where the sub-population convolutional module consists of a sub-population convolution, a batch normalization and a ReLU operation, and the sub-population convolution input dimension is 256 and the output dimension is 16; the second to fourth convolutional modules are composed of a sparse convolutional module and two sub-population convolutional modules, where the sparse convolutional module consists of a sparse convolution, a batch normalization and a ReLU operation; the output dimensions of the second to fourth convolutional modules are 32, 64, 64 respectively; the fifth convolutional module consists of a sparse convolutional module, the sparse convolution step size is (1, 1, 2), and the output dimension is 128; the voxel features after the convolutional middle layer become The region proposal network consists of two downsampling modules and two upsampling modules; each downsampling module consists of six full convolutional networks Conv2d. Except for the first full convolutional network in the second downsampling module, the step size of the remaining full convolutional networks is 1. The input and output dimensions of the last five full convolutional networks of each downsampling module are consistent. The output dimensions of the two downsampling modules are 128 and 256 respectively; each full convolutional network contains a two-dimensional convolution, a batch normalization and a ReLU operation; the upsampling module increases the output of the two downsampling modules to 256 to maintain the high resolution of the feature map; the output dimensions of the two upsampling modules are connected to obtain the final feature map 9. The method according to claim 8, characterized in that The final feature map is sent to the classifier and the regressor to obtain the type and position of the target to be detected, the size and direction of the target, including: The classifier is a two-dimensional convolution conv2d(512, 18), with an input of 512 dimensions and an output of 18 dimensions, and the output is the type of the target; the regressor is a two-dimensional convolution conv2d(512, 42), with an input of 512 dimensions and an output of 42 dimensions, and the output is the three-dimensional world coordinates, length, width, height, and yaw angle relative to the z-axis of the target box.

Images