A point cloud target detection method based on an improved second network

Abstract

The application belongs to the technical field of three-dimensional target detection, and discloses a point cloud target detection method based on an improved SECOND network. The method first trims and divides original point cloud data, and divides the whole point cloud graph region into voxels with the same size. Then, a voxel feature encoder considering the spatial position relationship inside the voxel is used to encode the features of all points in the voxel into a tensor as the feature of the voxel. Subsequently, a backbone network with enhanced feature extraction capability is realized through sparse convolution and the use of a ResNeXt module to extract features. Finally, the detection network is fed into a multi-detection head to obtain a predicted detection result. The road target detection method based on the improved SECOND network designed by the application can solve the problem of poor effect of difficult example detection in point cloud targets, greatly improve the detection accuracy, and provide technical support for the development of the field of three-dimensional point cloud target detection based on laser radar.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional target detection, and particularly relates to a road target detection method based on an improved SECOND network. BACKGROUND

[0002] With the rapid development of laser radar-based 3D acquisition technology, three-dimensional data is increasingly widely applied, including the fields of automatic driving, robot technology, remote sensing and medical treatment. Because it can provide rich geometric, shape and scale information, as a supplement to two-dimensional images, three-dimensional data provides more information for environment perception. Because it retains the original geometric information in three-dimensional space, it can accurately identify the position, size, direction and category of objects, and therefore three-dimensional target detection based on three-dimensional point cloud data is crucial for a laser radar system. Meanwhile, with the rapid improvement of storage technology and computer computing power, deep learning methods have gradually become the mainstream choice in the field of target detection and other fields due to their high accuracy and high robustness.

[0003] At present, there are many methods for three-dimensional target detection based on deep learning, which can be mainly divided into point-based methods, view-based methods and voxel-based methods. The point-based method identifies targets according to the features of each point, and the classic method is to learn point-by-point features through a PointNet structure, i.e. a multi-layer perception machine, and extract global shape features through maximum pooling. The advantage is that the original spatial information of the point is retained, but the disadvantage is that the number of points in the point cloud is huge, resulting in large computational consumption of this method. The view-based method converts the three-dimensional problem into a two-dimensional problem by compressing the three-dimensional image to two dimensions in a certain dimension, which is simple and easy to calculate, but loses the three-dimensional spatial information. The voxel-based method divides the point cloud image into regular voxels, and then directly processes the voxels through three-dimensional convolution, but consumes a large amount of computing resources, limiting the resolution of the voxels.

[0004] It can be seen that although the three-dimensional target detection method based on deep learning has made significant progress, it still has difficulties in handling difficult examples and learning discriminative features due to the large search space, irregularity and sparsity of point cloud data. SUMMARY

[0005] To solve the above problems, the application provides a point cloud target detection method based on an improved SECOND network, which introduces a voxel feature encoder that introduces the internal spatial position information of voxels, thereby increasing the spatial semantic information contained in each voxel feature. Further, a residual multi-path convolution (ResNeXt) module is introduced into the backbone network to strengthen the feature extraction capability of the network, thereby improving the recognition capability of the network for point cloud target detection and the difficult example detection accuracy.

[0006] The technical scheme of the present application:

[0007] A point cloud target detection method based on an improved SECOND network, the steps being as follows:

[0008] Step 1: obtaining three-dimensional point cloud data of a road target to be detected;

[0009] Step 2: preprocessing the three-dimensional point cloud data obtained in step 1, cropping and dividing the point cloud range in the three-dimensional point cloud data obtained in step 1 according to the same size, obtaining a new three-dimensional point cloud graph, the three-dimensional point cloud graph maintaining the original relative spatial positions of the points, the difference being that the points are respectively divided into different voxels;

[0010] Step 3: introducing the spatial position information of each point in each voxel, encoding it into the corresponding point feature information, and constructing a voxel feature encoder; encoding the points in each voxel in step 2 and using the existing layer feature integration to obtain a feature tensor representing each voxel and obtain a three-dimensional feature graph;

[0011] Step 3.1: calculating the geometric spatial center point coordinates of each voxel in the three-dimensional point cloud graph to obtain the three-dimensional coordinates C voxel of each voxel corresponding to the voxel;

[0012] Step 3.2: taking the voxel as a unit, calculating the coordinate difference C offset between each point in the voxel and the spatial position center C voxel of the corresponding voxel;

[0013] Step 3.3: to obtain the relative spatial information of the point and the voxel, calculating the Euclidean distance between the coordinates of each point and the spatial center point coordinates of the corresponding voxel, obtaining the distance C euclidean of each point to the center point, and the feature dimension is 1 dimension;

[0014] Step 3.4: using the new features obtained in the above steps: the three-dimensional coordinates C voxel of each voxel, the coordinate difference C offset between each point and the spatial position center of the corresponding voxel, and the distance C euclidean of each point to the center point, expanding the feature dimension of each point in the voxel; in the three-dimensional point cloud data obtained in step 1, the feature F of each original point contains four dimensions of three-dimensional coordinates and reflection intensity, which is now expanded, and each feature is spliced in the feature dimension according to the order of C euclidean , C voxel , F, C offset , and the new tensor F' is obtained as the new feature of the point;

[0015] Step 3.5: The extended features F' are integrated through a linear layer, a batch normalization layer, and a linear rectifier function, and the calculation results are maximum-pooled in the point dimension of the voxel tensor to obtain the maximum value of each point in each voxel to represent the high-dimensional features of each voxel for subsequent network feature extraction operations;

[0016] Step 4: To adapt to the data sparsity of the point cloud, improve the efficiency and speed of the calculation, and construct a sparse convolutional network using sparse convolution, perform three-dimensional convolution operations on the three-dimensional feature map composed of voxels obtained in step 3 to extract spatial features, and use Z-axis direction compression to compress the obtained sparse feature map to a two-dimensional overhead feature map;

[0017] Step 5: To effectively extract features on the two-dimensional overhead feature map, introduce a residual multi-path convolution module to construct a two-dimensional convolutional neural network, and perform feature extraction on the overhead feature map obtained in step 4 to obtain a two-dimensional feature map containing different scale information;

[0018] Step 5.1: Perform dimension reduction convolution on the two-dimensional overhead feature map obtained in step 4 to reduce the calculation amount, and obtain a feature map f1 with the original size and half the original dimension;

[0019] Step 5.2: Introduce a residual multi-path convolution network ResNeXt structure, replace the ordinary convolution with a residual multi-path convolution, and construct a residual multi-path convolution module to perform multiple feature extractions on the feature map f1 to obtain a feature map f2 with the same size and dimension as the input feature map f1;

[0020] Step 5.3: To obtain a feature map with lower resolution and higher dimension, perform two times convolution upsampling on the feature map f2, and again use the same method as in step 5.2 to construct a residual multi-path convolution module to perform multiple feature extractions, obtaining a down-sampled feature map f3;

[0021] Step 5.4: Use transpose convolution to perform two times upsampling on the feature map f3 to obtain a feature map with the same size and dimension as the input feature map f3, and concatenate it with the feature map f2 in the channel dimension to obtain a feature map f4. The two-dimensional feature map obtained by performing batch normalization and linear rectifier function processing on the feature map f4 is used for subsequent detection head regression operations;

[0022] Step 6: Based on the two-dimensional feature map obtained in step 5, use the RPN network anchor detection head to perform classification and regression to obtain the prediction box, direction, and category.

[0023] The beneficial effects of the present application: the three-dimensional target detection method based on improved SECOND designed by the present application can solve the problem of large difference in accuracy of different categories and poor difficult example detection effect caused by the lack of voxel space information in point cloud three-dimensional target detection, and significantly improve the difficult example detection accuracy. The present application is helpful for the application of three-dimensional target detection algorithm in laser radar system, and further provides technical support for three-dimensional point cloud target detection system based on laser radar for automatic driving and the like. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The method flowchart of the present application.

[0025] Figure 2 A typical three-dimensional point cloud image sample.

[0026] Figure 3 The network structure diagram of the RPN backbone network based on ResNeXt structure realized by the present application. DETAILED DESCRIPTION

[0027] The specific embodiments of the present application will be further described in detail below in combination with the drawings and technical solutions.

[0028] Taking the road three-dimensional point cloud image sample represented by the above as an example, three main road targets are detected and identified using deep learning technology, including bounding box regression and category detection. The three categories are car (Car), bicycle (Cyclist), and pedestrian (Pedestrian). Figure 2

[0029] Step 1: Use KITTI, the largest computer vision algorithm evaluation data set in the world at present, jointly established by Karlsruhe Institute of Technology in Germany and Toyota Technical Research Institute in the United States, to obtain three-dimensional point cloud data of road targets. The data categories include training set and test set, and the data types include calib file, label file in txt format, image file in png format, and point cloud data file in bin format;

[0030] Step 2: Preprocess the three-dimensional point cloud image data described in step 1.

[0031] Step 2.1: Crop the range of the point cloud image, and crop the irregular three-dimensional space into a space with a width of 70.4 meters, a length of 80 meters, and a height of 4 meters.

[0032] Step 2.2: Perform several feature enhancement operations on the cropped regular point cloud data, including: random global flipping with x-axis as axis, random global rotation with angle [-π / 4, π / 4] as range, and random global scaling with [0.95, 1.05] times as range.​

[0033] Step 2.3: Voxelization processing is performed on the point cloud image data that has been pre-processed as above. The three-dimensional point cloud image processed as described above is divided into voxels of the same and fixed size: 0.05 meters in length and width, and 0.1 meters in height, to obtain a voxelized three-dimensional point cloud composed of voxels of the same size stacked together while maintaining the original relative spatial positions;

[0034] Step 3: A voxel feature encoder is constructed to encode the point information in each voxel in Step 2 to obtain a feature tensor representing each voxel and obtain a three-dimensional feature map. The following steps will describe the specific operations of the encoder:

[0035] Step 3.1: First, the geometric spatial center point coordinates of each voxel are calculated based on the index value of each voxel and the length, width, and height of the voxel to obtain its three-dimensional coordinates C voxel relative to the coordinate system;

[0036] Step 3.2: For each point contained in each voxel, the deviation value of each dimension in the three coordinate axis directions is calculated based on the three-dimensional coordinates of each point and the spatial center point coordinates C voxel of the corresponding voxel. The coordinate value of the point is subtracted from the coordinate value of the corresponding voxel center point to obtain the calculation result as C offset ;

[0037] Step 3.3: The Euclidean distance of each point in the voxel from the spatial center point of the corresponding voxel is calculated. The C offset calculated in the above step is squared and added together and then the square root is taken to obtain the distance C euclidean of each point from the center point of the corresponding voxel;

[0038] Step 3.4: The feature dimensions of each point in the voxel are expanded. The original point feature F contains four dimensions, including three coordinates corresponding to the three coordinate axes and the reflectance intensity feature. Now, in the order of C euclidean , C voxel , F, C offset , each feature is spliced in the feature dimension to obtain a new feature tensor F 11 with expanded dimensions as the new feature of the point;

[0039] Step 3.5: The expanded and enriched feature F 11 of each point is processed through a linear layer, a batch normalization layer, and a linear rectifier function to integrate the features within each voxel and perform maximum pooling on the feature dimensions within the voxel to obtain a high-dimensional feature representing each voxel as the feature vector of the voxel for subsequent network operations such as feature extraction.

[0040] Step 4.1: using sparse convolution and submanifold convolution, construct a three-dimensional sparse convolution intermediate extraction layer to perform three-dimensional feature extraction on the three-dimensional feature map obtained in step 3.

[0041] Step 4.2: convert the extracted point cloud sparse features to a two-dimensional bird's eye view perspective. First, convert the sparse features to a voxel feature format, then use height compression in the Z-axis direction to compress the three-dimensional feature map and convert it to a two-dimensional overhead feature map format.

[0042] Step 5: construct a two-dimensional convolution backbone network to extract features from the overhead feature map described in step 4 and obtain two-dimensional feature maps containing different scale information for subsequent classification and regression. Figure 3 Taking the network structure represented by the network structure diagram as an example, the following will be described in detail:

[0043] Step 5.1: first, perform dimension reduction convolution on the obtained two-dimensional overhead feature map to obtain a two-dimensional feature map f1 with the same size as the original size and half the original dimension.

[0044] Step 5.2: introduce ResNeXt structure to construct residual multi-path convolution module. The module design follows the segmentation- transformation-merging mode, and the specific steps are as follows: first, the input of the module is a two-dimensional feature map with shape (batch, channel, width, height), first perform a dimension reduction convolution, set the output dimension of the convolution to half of the input dimension, the convolution kernel and the step are 1, and follow the batch normalization processing after the convolution. Then, perform another convolution on the output, which is a grouped multi-path convolution. The input feature map is divided into multiple branches, each branch simultaneously and independently performs the same convolution. Each branch uses the same topology to extract features multiple times, and the results are connected in the channel dimension to obtain an integrated two-dimensional feature map, which is then batch normalized. Finally, perform another convolution calculation on the feature map, which increases the dimension of the intermediate feature map in the channel dimension to restore it to the input dimension of the module, and performs batch normalization to obtain a two-dimensional feature map with the same size and dimension as the input feature map.

[0045] Step 5.3: Using the residual multi-path convolution module described above, extract features from the feature map f1 multiple times to obtain the original two-dimensional feature map f2. Then, perform double convolution on the feature map f2 to upsample it and obtain a feature map with lower resolution and higher dimension. Then, use the residual multi-path convolution module again to extract features from it multiple times to obtain the downsampled feature map f3.

[0046] Step 5.4: Using transposed convolution, perform a 2x convolution upsampling on feature map f3 after feature extraction by the residual multi-path convolution module, while keeping the output dimension unchanged, to obtain a feature map with the same size and dimension as the original input. At the same time, perform transposed convolution upsampling on feature map f2 to obtain a feature map with the same size and dimension as the original input. Concatenate the two feature maps obtained in this step along the channel dimension to obtain feature map f4, and perform batch normalization, linear rectification function, and other processing on feature map f4 to obtain the final output of this backbone network, which is used for target detection and regression operations in subsequent steps.

[0047] Step 6: Based on the two-dimensional feature map obtained in Step 5 after feature extraction via the backbone network, classification and regression are performed using an anchor head to obtain the prediction results. For example... Figure 1 As shown in the DetectHead section, the predictions include the regression of the predicted bounding box, the corresponding category, and the orientation of the box. The two-dimensional feature map obtained in step 5 is first subjected to a dimensionality reduction convolution, from 512 dimensions to 64 dimensions. Then, three different convolution operations with 1x1 kernels are performed on this convolution result to obtain the candidate bounding box, category, and orientation as three different prediction targets.

[0048] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1.A point cloud object detection method based on an improved SECOND network, characterized in that, The steps are as follows: Step 1: Obtain three-dimensional point cloud data of the road target to be detected; Step 2: Preprocess the three-dimensional point cloud data obtained in step 1, crop and divide the point cloud range in the three-dimensional point cloud data obtained in step 1 according to the same size, and obtain a new three-dimensional point cloud graph. The three-dimensional point cloud graph maintains the original relative spatial positions of each point, except that each point is divided into different voxels; Step 3: By introducing the spatial position information of each point inside each voxel, encode it into the corresponding point feature information, and construct a voxel feature encoder; Encode the points in each voxel in step 2 and use the existing layer feature integration to obtain a feature tensor representing each voxel and obtain a three-dimensional feature map; Step 3.1: Calculate the geometric center point coordinates of each voxel in the three-dimensional point cloud map to obtain the three-dimensional coordinates C corresponding to each voxel voxel ; Step 3.2: In voxel units, calculate the coordinate difference C voxel of each point within the voxel from the spatial position center C offset of its corresponding voxel. Step 3.3: To obtain the relative spatial information of points and voxels, the Euclidean distance between each point in the voxel and the corresponding spatial center point coordinate of the voxel is calculated to obtain the distance C of each point to the center point euclidean The characteristic dimension is 1 dimension; Step 3.4: using the new features obtained in the above steps: the three-dimensional coordinates C corresponding to each voxel voxel , the coordinate difference C of each point and the spatial position center of the corresponding voxel offset , the distance C of each point to the center point euclidean , the feature dimension of each point in the voxel is expanded; the feature F of each original point in the three-dimensional point cloud data obtained in step 1 contains four dimensions of three-dimensional coordinates and reflection intensity, which is now expanded, and the features are spliced in the feature dimension according to the order of C euclidean , C voxel , F, C offset , and the new tensor F' is obtained as the new feature of the point; Step 3.5: Integrate the expanded features F' through a linear layer, a batch normalization layer, and a linear rectifier function, and perform maximum pooling on the point dimension of the voxel tensor to obtain the maximum value of each point in each voxel to represent the high-dimensional features of each voxel for subsequent network feature extraction operations; Step 4: To adapt to the data sparsity of the point cloud and improve the efficiency and speed of the calculation, a sparse convolution network is constructed using sparse convolution to perform three-dimensional convolution operations on the three-dimensional feature map composed of voxels obtained in step 3 to extract spatial features, and the sparse feature map obtained by the operation is compressed in the Z-axis direction to a two-dimensional overhead feature map; Step 5: To effectively extract features on the two-dimensional overhead feature map, introduce a residual multi-path convolution module to construct a two-dimensional convolutional neural network to extract features from the overhead feature map obtained in step 4 and obtain a two-dimensional feature map containing different scale information; Step 5.1: Perform dimension reduction convolution on the two-dimensional overhead feature map obtained in step 4 to reduce the calculation amount and obtain a feature map f1 with the original size and half the original dimension; Step 5.2: Introduce a residual multi-path convolution network structure, replace the ordinary convolution with a residual multi-path convolution, and construct a residual multi-path convolution module to extract features from the feature map f1 multiple times to obtain a feature map f2 with the same size and dimension as the input feature map f1; Step 5.3: To obtain a feature map with lower resolution and higher dimension, perform two times convolution upsampling on the feature map f2, and again use the same method in step 5.2 to construct a residual multi-path convolution module to extract features multiple times to obtain a down-sampled feature map f3; Step 5.4: Use transpose convolution to perform two times upsampling on the feature map f3 to obtain a feature map with the same size and dimension as the input feature map f3, and concatenate it with the feature map f2 in the channel dimension to obtain a feature map f4. The two-dimensional feature map obtained by batch normalization and linear rectifier function processing of the feature map f4 is used for subsequent detection head regression operations; Step 6: Based on the two-dimensional feature map obtained in step 5, use the RPN network anchor detection head to classify and regress to obtain the prediction box, direction, and category.

Images