A 3D object detection method in spherical coordinates based on special-shaped 3D convolution

Abstract

A method for 3D object detection in spherical coordinates based on heterogeneous 3D convolutions comprises the following steps: Step 1: Acquire raw point cloud data; Step 2: Convert the received raw point cloud data and the coordinates of the target bounding box from a 3D Cartesian coordinate system to a spherical coordinate system; Step 3: Divide the space into voxels according to spherical coordinates, converting the sparse point cloud data into a dense point cloud data; Step 4: Encode voxel features for each voxel to obtain a voxel feature map; Step 5: Extract spatial features from the voxel feature map using heterogeneous 3D convolutions to obtain a dense feature map matrix; Step 6: Construct a 3D object detection network S based on heterogeneous 3D convolutions; Step 7: Train the 3D object detection network S based on heterogeneous 3D convolutions. The present invention utilizes heterogeneous 3D convolution modules to replace conventional symmetric convolution layers to extract voxel spatial features, thereby obtaining high-dimensional spatial features for the voxels. This method can effectively reduce the computational complexity of 3D convolution layers.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a 3D target detection method in spherical coordinates based on special-shaped three-dimensional convolution. Background Art

[0002] In the field of autonomous driving, the accuracy of 3D object detection directly affects the safety performance of autonomous vehicles. The autonomous driving system needs to use the results of 3D object detection to perceive the current environment, and then plan the travel path, perform obstacle avoidance operations, etc.

[0003] The backbone network of 3D target detection can be divided into point-based methods, projection-based methods and voxel-based methods. The point-based method regards points as a collection of unordered points and directly calculates the overall features, but it cannot effectively extract the spatial context information of the point cloud; the projection-based method converts point cloud data into a two-dimensional image through perspective projection, but loses the structure of the point cloud data; the voxel-based method divides the point cloud data into 3D voxel blocks and uses 3D sparse convolution for processing, which can effectively extract spatial context information and occupies a large proportion in today's outdoor large-scale point cloud scene processing.

[0004] However, since most 3D sensors, such as LiDAR and structured light cameras, generate 3D point clouds that only cover the surface of objects, using a symmetrical 3D convolutional structure within the network to extract features from the input can be understood as extracting surface features. The input point cloud is like a patch in space, and only a subset of elements in the symmetrical 3D convolution kernel intersect with the point cloud. Consequently, a small percentage of these elements actually contribute to feature extraction, leading to wasted memory and a high computational load. Therefore, there is still room for improvement in voxel-based 3D object detection.

[0005] The application, published as CN115937850A, is titled "A LiDAR Point Cloud Semantic Segmentation Method Based on Asymmetric Convolution." The proposed method utilizes an asymmetric convolutional backbone network and contextual feature enhancement module to effectively enhance point cloud feature extraction and improve semantic segmentation accuracy. The asymmetric convolutional backbone network consists of four downsampling asymmetric convolutional modules and four upsampling asymmetric modules. Three skip connections are used to concatenate the upsampling results (low-level features) with the corresponding downsampling results, effectively fusing low-level and high-level features and improving the model's ability to learn detailed information.

[0006] However, this method has the following shortcomings: 1. The spatial coordinate system uses a rectangular coordinate system, and the implementation principle of lidar results in uneven density in the raw data output in this rectangular coordinate system, resulting in poor performance of the model in small-scale target detection. 2. The point cloud data is encoded using a projection method, which sacrifices its structure. 3. It does not effectively utilize temporal information.

[0007] In 3D object detection, 3D scanners use one or more sensors to capture spatial point cloud data in the form of continuous scanning and accumulation. The output point cloud data is the result of multiple frames of scanning at different locations. The common feature of these point cloud data sources is that they only contain point cloud information on the surface of the object. The vast majority of the scene is holes, without point cloud data. They cannot be tightly arranged according to the spatial structure like 2D images. Instead, they are mostly stored in a sequence and are mostly disordered in space. Therefore, point cloud-based 3D object detection needs to be redesigned according to the characteristics of point cloud data. It mainly includes three forms: voxelization, point input, and point-voxel combination methods.

[0008] The traditional symmetric convolution architecture used for voxel feature extraction is computationally intensive because the computational complexity of three-dimensional convolution is proportional to the cube of the kernel's side length. This results in a low percentage of elements actually contributing to feature extraction, resulting in large memory usage and low efficiency. Summary of the Invention

[0009] To overcome the above-mentioned problems in the existing technology, the present invention aims to provide a 3D object detection method in spherical coordinates based on special-shaped 3D convolution. This method uses a special-shaped 3D convolution module to replace the commonly used symmetric convolution layer to extract voxel spatial features, thereby obtaining high-dimensional spatial features of the voxels. This method can effectively reduce the computational complexity of the 3D convolution layer.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is:

[0011] A 3D target detection method in spherical coordinates based on special-shaped 3D convolution includes the following steps:

[0012] Step 1: Obtain raw point cloud data through the lidar sensor;

[0013] Step 2: Convert the received original point cloud data and the coordinate representation of the target bounding box from the 3D rectangular coordinate system to the spherical coordinate system;

[0014] Step 3: Divide the space into voxels according to spherical coordinates, convert the sparse point cloud data into dense point cloud data, and achieve a balanced number of point clouds in each voxel to improve the utilization of point cloud spatial features and the efficiency of feature extraction;

[0015] Step 4: Encode the voxel features of each voxel to obtain a voxel feature map F with a scale of i×j×k×4 in ;

[0016] Step 5: Get the voxel feature map F in Use the special-shaped three-dimensional convolution to further extract spatial features and obtain the dense feature map matrix F t ;

[0017] Step 6: Construct a 3D object detection network S based on heterogeneous 3D convolution and transform the dense feature map matrix F t Enter the network S;

[0018] Step 7: Train the 3D object detection network S based on special-shaped 3D convolution to obtain the performance indicators of the network under the KITTI dataset.

[0019] The step 1 is specifically as follows:

[0020] KITTI data is collected using the HDL-64E lidar sensor. During operation, the entire laser module is driven by a direct-drive motor and rotates at a set speed. The sensor has a horizontal field of view of up to 360°±0.09° and a vertical field of view of up to [+2°, -24.33°]. The raw data output by the sensor can be expressed as the distance d(θ, φ) from the object to the lidar and the object's reflection intensity I(θ, φ), where θ is the azimuth angle and φ is the pitch angle.

[0021] The KITTI dataset is used, which contains training samples and test samples. The samples include lidar point clouds and images, and are labeled into three categories: vehicles (Car), bicycles (Cyclist), and pedestrians (Pedestrian).

[0022] The point cloud data refers to the original point cloud data received by the lidar sensor.

[0023] The step 2 is specifically as follows:

[0024] Step 2.1: Convert the point cloud data into coordinates. Point cloud data is usually in (x i ,y i ,z i ,I i ), where x i ,y i ,z i Represents the coordinates of the i-th point in the rectangular coordinate system, I i Represents the reflection intensity of the point;

[0025] For each point (x i ,y i ,zi ,I i ),calculate:

[0026]

[0027] Get (d i ,θ i ,φ i ,I i ), where d i Represents the straight-line distance from the i-th point to the origin (lidar), θ i 、φ i is the azimuth and elevation angle of the i-th point, I i is the reflection intensity at that point;

[0028] Step 2.2, coordinate transformation of the target bounding box;

[0029] In the three-dimensional rectangular coordinate system, the bounding box includes the position of the bottom center point of the target (x, y, z), the target size (w, h, l) (w, h, l refer to the length, height, and width of the target size, respectively), and the rotation angle r. The spherical coordinates of the bottom center point (x, y, z) are converted to (d, θ, φ) according to the coordinate conversion formula described in 2.1, while (w, h, l, r) remain unchanged. The dataset annotation is converted to the spherical coordinate system bounding box (d, θ, φ, w, h, l, r).

[0030] The step 3 is specifically as follows:

[0031] Step 3.1: The voxelization module takes the point cloud data in the spherical coordinate system converted in step 2.1 as input (the voxelization module is a module that converts the geometric representation of an object into the voxel representation that is closest to the object and generates volume data). d ,v θ ,v φ , where v d ,v θ ,v φ are the length, width, and height of a single voxel, and the range of the three dimensions [d min ,d max ][θ min ,θ max ][φ min ,φ max ], d min ,d max is the minimum and maximum value of the voxel coordinate in the distance dimension; θ min ,θ max is the minimum and maximum value of the voxel coordinate in the azimuth dimension; φ min ,φ maxis the minimum and maximum value of the voxel coordinate in the pitch angle dimension, and the space is divided into voxel grids. The number of grids in the three dimensions is:

[0032]

[0033] Step 3.2: In the spherical coordinate system, divide the spherical coordinate system into d′×θ′×φ′ with side length (dv d ,dv θ ,dsinθv φ ) as a voxel, dv d ,dv θ ,dsinθv φ They refer to the length, width, and height of the voxel respectively. The lengths of the three sides of the voxel are proportional to d. When d changes, the proportions of the three sides of the voxel remain unchanged, that is, the shape of the voxel is constant.

[0034] The step 4 is specifically as follows:

[0035] Step 4.1, define the voxel coordinates and voxel center points in the spherical coordinate system:

[0036]

[0037] Among them, the (i,j,k)th voxel is a v d ,v θ ,v φ The center of the voxel is:

[0038]

[0039] Step 4.2: Extract features from each voxel and select a more lightweight voxel feature extractor. In order to decouple the voxel features from the absolute position of the voxel itself in space, the voxel feature extractor uses a manually designed feature calculate:

[0040]

[0041] Among them, d c ,θ c ,φ c is the voxel center, is the average value of the points within the voxel, that is, the relative distance from the average value of the points within the voxel to the center point of the voxel is used as the voxel feature. is the average value of the reflection intensity within the voxel;

[0042] Step 4.3: Store the features of non-empty voxels (voxels containing data) to generate a 4D tensor with a size of C′×d′×θ′×φ′, where C′ is the feature dimension size. Calculate the feature encoding of all voxels to obtain a voxel feature map F with a scale of i×j×k×4. in .

[0043] The step 5 is specifically as follows:

[0044] Since the number of original points represented in a single voxel is very small, the voxel feature itself has very few spatial features. Therefore, after obtaining the voxel feature, it is necessary to use special-shaped three-dimensional convolution to further extract its spatial features. This can greatly reduce the number of model parameters and the memory space occupied, thereby greatly improving efficiency.

[0045] The input layer of the special-shaped three-dimensional convolution module uses two SubMConv3D convolution layer structures to perform preliminary extraction of the voxel feature space, and then uses three three-dimensional special-shaped convolution kernels to extract local spatial features.

[0046] The special-shaped three-dimensional convolution module is specifically:

[0047] In step 5.1, eight different shaped 3D convolution kernels are used for convolution, which are optimized for vertical and oblique surfaces based on the characteristics of radar scanning.

[0048] Step 5.2: Encode the designed convolution kernels of different shapes using a mask. The length of the mask is the cube of the maximum side length of the convolution kernel. The special-shaped three-dimensional convolution kernel is designed. Each mask bit represents a spatial position and is encoded from left to right and from top to bottom.

[0049] Step 5.3: To meet the two-dimensional input requirements of the region proposal network in step 6, it is necessary to apply the index matrix to reorganize the output into a feature matrix. At the output of the module, a SubMConv3D convolutional layer is used to convert the spatial dimension from three dimensions to the d-θ plane, and the sparse features obtained are converted into a dense feature map matrix output F. t .

[0050] The step 6 is specifically as follows:

[0051] The 3D object detection network S based on special-shaped 3D convolution consists of four parts: FPN feature extraction module S1, bottom-up path enhancement module S2, adaptive feature pooling module S3, and region proposal module S4;

[0052] Step 6.1, construct FPN feature extraction module S1, and transform F t Input into S1 and get F c1 、F c2 、Fc3 、F c4 , F c4 Copy to F p4 , for F p4 Upsampling is performed, and convolution is performed on F C3 The number of channels is reduced, and then F p4 With F C3 The corresponding elements are added and convolution is performed to obtain F p3 ; Use convolution to transform F c2 The number of channels is reduced, and F c2 With F p3 Add and perform convolution operation to get F p2 ; Use convolution to transform F c1 The number of channels is reduced, and F c1 With F p2 Add and perform convolution operation to get F p1 ;

[0053] Step 6.2, construct the bottom-up route enhancement module S2; since when the bottom-level features reach F P4 When the network passes through multiple layers, the underlying information is very vague, so a bottom-up route is added to compensate for and strengthen the positioning information; p1 Copy to F N1 After that, F N1 Downsampling, with F p2 Unit addition, convolution operation is performed to fuse the features to obtain F N2 ; for F N2 Downsampling, with F p3 Unit addition, convolution operation is performed to fuse the features to obtain F N3 , for F N3 Downsampling, with F p4 Unit addition, convolution operation is performed to fuse the features to obtain F N4 At the same time, a shortcut across multiple layers is added to the bottom-up and top-down modules respectively.

[0054] Step 6.3: Use the region proposal module S4 to generate anchor frames. The rotation angle r of each anchor frame is 0° and 90°.

[0055] Step 6.4, construct the adaptive feature pooling module S3, and perform classification, position regression and direction regression on the anchor frame, and perform the following steps on (F N1 ~F N4 ) The ROI extracted by the region proposal module S4 is compressed into a one-dimensional feature vector, and then different feature maps are fused by summation.

[0056] The step 7 is specifically as follows:

[0057] The KITTI dataset input method is set to input only the point cloud of ±45° in front of the camera. Only this part of the point cloud contains annotations. The input range is limited to 0≤x≤70.4, -40≤y≤40, -3≤x≤1, and the unit is meter.

[0058] Step 7.1, construct a 3D object detection model (SPHNet) in the spherical coordinate system;

[0059] Step 7.2, in which the intermediate feature extraction module in the SECOND model is replaced with the special-shaped 3D convolution module, and the detection performance of the original structure and the replaced special-shaped 3D convolution structure is tested to verify the impact of the special-shaped 3D convolution module on the detection performance;

[0060] Step 7.3, testing the above two models;

[0061] Step 7.4: Analyze the experimental results. Comparing the detection performance of the control group SPHNet network with the original model structure and the detection performance of the experimental group after the module replacement, it can be seen that the special-shaped three-dimensional convolution module does have a slight impact on the detection performance of the model, but the impact is not large. It can greatly reduce the number of model parameters and reduce the memory space occupied, thereby greatly improving efficiency. Beneficial effects of the present invention:

[0062] The present invention uses a special-shaped three-dimensional convolution module to replace the commonly used symmetric convolution layer to extract voxel spatial features, obtain the high-dimensional spatial features of the voxels, and then realize 3D target detection through a region proposal network.

[0063] The present invention uses a special-shaped three-dimensional convolution structure to replace the 3D convolution layer in the 3D target detection model in the spherical coordinate system, and improves the sparsity of the point cloud by converting the point cloud data into spherical coordinates. The use of the special-shaped three-dimensional convolution structure effectively reduces the computational complexity of the 3D convolution layer while only slightly affecting the model accuracy.

[0064] The spatial coordinate system of the present invention adopts a spherical coordinate system, which improves the sparsity of the point cloud and effectively improves the detection accuracy of small-sized objects. The use of a special-shaped three-dimensional convolution structure for voxel feature extraction can greatly reduce the number of model parameters and memory space occupied, thereby greatly improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Schematic diagram of the correspondence between the three-dimensional rectangular coordinate system and the spherical coordinate system.

[0066] Figure 2 Schematic diagram of the corresponding relationship of the target bounding box transformed from the three-dimensional rectangular coordinate system to the spherical coordinate system.

[0067] Figure 3 Schematic diagram of voxel division in spherical coordinates.

[0068] Figure 4 Schematic diagram of the structure of the special-shaped three-dimensional convolution module.

[0069] Figure 5 Schematic diagram of the 8 types of special-shaped 3D convolution kernel structures used.

[0070] Figure 6 Schematic diagram of 3D object detection network S based on special-shaped 3D convolution.

[0071] Figure 7 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0072] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0073] Step 1: Get point cloud data:

[0074] KITTI data is collected by the HDL-64E lidar, which features 64 fixed-point lasers. During operation, the entire laser module is driven by a direct-drive motor, rotating at a set speed. The sensor has a horizontal field of view of 360° ± 0.09° and a vertical field of view of [+2°, -24.33°]. The raw data output by the sensor can be expressed as the distance d(θ, φ) from the object to the lidar and the object's reflection intensity I(θ, φ), where θ is the azimuth angle and φ is the elevation angle.

[0075] The KITTI dataset is selected, which contains 7481 training samples and 7518 test samples. The samples include lidar point clouds and images, with three annotated categories: vehicles (Car), bicycles (Cyclist) and pedestrians (Pedestrian). The total number of annotated targets reaches 80,256.

[0076] Step 2: Convert the coordinate representation of the point cloud and the target bounding box from the 3D rectangular coordinate system to the spherical coordinate system:

[0077] Step 2.1, transform the point cloud data into coordinates. Point cloud data is usually in (x i ,y i ,z i ,I i ), where x i ,y i ,z i Represents the coordinates of the i-th point in the rectangular coordinate system, I i Represents the reflection intensity of the point.

[0078] For each point (x i ,y i ,zi ,I i ),calculate:

[0079]

[0080] Get (d i ,θ i ,φ i ,I i ), where d i Represents the straight-line distance from the i-th point to the origin (lidar), θ i 、φ i is the azimuth and elevation angle of the i-th point, I i is the reflection intensity at that point.

[0081] The correspondence between the three-dimensional rectangular coordinate system and the spherical coordinate system is as follows Figure 1 As shown:

[0082] Step 2.2 performs coordinate transformation on the target bounding box.

[0083] In a 3D rectangular coordinate system, the bounding box includes the position of the target's bottom center point (x, y, z), the target's dimensions (w, h, l), and the rotation angle r. The spherical coordinates of the bottom center point (x, y, z) are converted to (d, θ, φ) using the coordinate conversion formula described in 2.1. While (w, h, l, r) remain unchanged, the dataset annotations are converted to the spherical coordinate bounding box (d, θ, φ, w, h, l, r).

[0084] The target bounding box is transformed from the 3D rectangular coordinate system to the spherical coordinate system as follows Figure 2 As shown:

[0085] Step 3: Divide the space into voxels according to spherical coordinates:

[0086] Step 3.1: The voxelization module takes the point cloud as input and transforms it into a point cloud according to the preset voxel size v. d ,v θ ,v φ and the range of three dimensions [d min ,d max ][θ min ,θ max ][φ min ,φ max ] Divide the space into voxel grids, and the number of grids in the three dimensions is:

[0087]

[0088] Step 3.2, in the spherical coordinate system, Divide a vector direction into a side length (dv d ,dv θ,dsinθv φ ) as a voxel. The lengths of the three sides of the voxel are proportional to d. When d changes, the proportions of the three sides of the voxel remain basically unchanged, that is, the shape of the voxel is basically constant.

[0089] The schematic diagram of voxel division under spherical coordinates is as follows Figure 3 As shown:

[0090] Step 4: Encode voxel features for each voxel

[0091] Step 4.1, define the voxel coordinates and voxel center points in the spherical coordinate system:

[0092]

[0093] Among them, the (i,j,k)th voxel is a v d ,v θ ,v φ The center of the voxel is:

[0094]

[0095] Step 4.2: Extract features from each voxel and select a more lightweight voxel feature extractor. In order to decouple the voxel features from the absolute position of the voxel itself in space, the voxel feature extractor uses a manually designed feature calculate:

[0096]

[0097] Among them, d c ,θ c ,φ c is the voxel center, is the average value of the points within the voxel, that is, the relative distance from the average value of the points within the voxel to the center point of the voxel is used as the voxel feature.

[0098] Step 4.3: Store the features of the non-empty voxels to create a 4D tensor with a size of C′×d′×θ′×φ′, where C′ is the feature dimension size. Calculate the feature encoding of all voxels to obtain a voxel feature map F with a scale of i×j×k×4. in .

[0099] Step 5: Use special-shaped 3D convolution to extract spatial features:

[0100] Since the number of original points represented in a single voxel is very small, the voxel feature itself has very few spatial features. Therefore, after obtaining the voxel feature, it is necessary to use special-shaped three-dimensional convolution to further extract its spatial features.

[0101] The structure of the special-shaped 3D convolution module is as follows Figure 4 As shown:

[0102] The input layer of the module uses two SubMConv3D convolutional layer structures to perform preliminary extraction of the voxel feature space, and then uses three 3D heterogeneous 3D convolutional blocks to extract local spatial features.

[0103] Step 5.1, use 8 different shaped 3D convolution kernels to perform convolution, and optimize them for vertical and oblique surfaces according to the characteristics of radar scanning. The shaped 3D convolution kernel structure used is as follows: Figure 5 As shown:

[0104] In step 5.2, the designed convolution kernels of different shapes are encoded using a mask. The mask length is the cube of the maximum side length of the convolution kernel. The convolution kernel used in this method is based on a 3×3×3 convolution design with a mask length of 27. Each mask bit represents a spatial position, and the encoding is from left to right and top to bottom. For example, the mask for the fourth convolution kernel in the figure is {[(1,1,1),(1,0,1),(1,1,1)],[(1,0,1),(0,0,0),(1,0,1)],[(1,1,1),(1,0,1),(1,1,1)]}.

[0105] Among them, the masks of the 8 special-shaped 3D convolution kernels are expressed as:

[0106] (1){[(0,0,0),(1,1,1),(0,0,0)],[(0,0,0),(1,1,1),(0,0,0)],[(0,0,0),(1,1,1),(0,0,0)]},

[0107] (2){[(0,0,0),(0,0,0),(0,0,0)],[(1,1,1),(1,1,1),(1,1,1)],[(0,0,0),(0,0,0),(0,0,0)]},

[0108] (3){[(0,1,0),(0,1,0),(0,1,0)],[(0,1,0),(0,1,0),(0,1,0)],[(0,1,0),(0,1,0),(0,1,0)]},

[0109] (4){[(1,1,1),(1,0,1),(1,1,1)],[(1,0,1),(0,0,0),(1,0,1)],[(1,1,1),(1,0,1),(1,1,1)]}、

[0110] (5){[(1,0,0),(0,1,0),(0,0,1)],[(1,0,0),(0,1,0),(0,0,1)],[(1,0,0),(0,1,0),(0,0,1)]},

[0111] (6){[(0,0,1),(0,1,0),(1,0,0)],[(0,0,1),(0,1,0),(1,0,0)],[(0,0,1),(0,1,0),(1,0,0)]},

[0112] (7){[(1,1,1),(0,0,0),(0,0,0)],[(0,0,0),(1,1,1),(0,0,0)],[(0,0,0),(0,0,0),(1,1,1)]}、

[0113] (8){[(0,0,0),(0,0,0),(1,1,1)],[(0,0,0),(1,1,1),(0,0,0)],[(1,1,1),(0,0,0),(0,0,0)]};

[0114] In step 5.3, based on the convolution kernel encoding, the input-output coordinate index matrix and the required calculation rule matrix are generated. Based on the calculation rule matrix, the GEMM operation is applied to the input feature matrix to calculate the intermediate result of the heterogeneous 3D convolution.

[0115] Step 5.4: To meet the two-dimensional input requirements of the region proposal network in step 6, it is necessary to apply the index matrix to reorganize the output into a feature matrix. At the output of the module, a SubMConv3D convolutional layer is used to convert the spatial dimension from three dimensions to the d-θ plane, and the sparse features obtained are converted into a dense feature map matrix output F. t .

[0116] Step 6: Construct a 3D object detection network S based on heterogeneous 3D convolution and transform the dense feature map matrix F t Enter the network S:

[0117] The 3D object detection network S based on special-shaped 3D convolution consists of four parts: FPN feature extraction module S1, bottom-up path enhancement module S2, adaptive feature pooling module S3, and region proposal module S4.

[0118] The network structure is as follows Figure 7 As shown:

[0119] Step 6.1, construct FPN feature extraction module S1, and transform F t Input into S1 and get F c1 、F c2 、Fc3 、F c4 , F c4 Copy to F p4 , for F p4 Upsample and use 1×1 convolution to convert F C3 The number of channels is reduced, and then F p4 With F C3 The corresponding elements are added and a 3×3 convolution operation is performed to obtain F p3 . Use 1×1 convolution to transform F c2 The number of channels is reduced, and F c2 With F p3 Add and perform a 3×3 convolution operation to get F p2 . Use 1×1 convolution to transform F c1 The number of channels is reduced, and F c1 With F p2 Add and perform a 3×3 convolution operation to get F p1 .

[0120] Step 6.2, construct the bottom-up route enhancement module S2. P4 When the network passes through multiple layers, the underlying information becomes very vague, so a bottom-up route is added to compensate for and strengthen the positioning information. p1 Copy to F N1 After that, F N1 Perform downsampling with a step size of 2 and a 3×3 convolution kernel, and F p2 Unit addition, 3×3 convolution operation is performed to fuse the features to obtain F N2 . N2 Perform downsampling with a step size of 2 and a 3×3 convolution kernel, and F p3 Unit addition, 3×3 convolution operation is performed to fuse the features to obtain F N3 . N3 Perform downsampling with a step size of 2 and a 3×3 convolution kernel, and F p4 Unit addition, 3×3 convolution operation is performed to fuse the features to obtain F N4 At the same time, a shortcut across multiple layers is added to the bottom-up and top-down modules respectively.

[0121] In step 6.3, the region proposal module S4 is used to propose the module to generate anchor frames. Three groups of anchor frames with sizes of 3.9m*1.6m*1.65m, 1.76m*0.4m*0.73m, and 0.6m*0.8m*1.73m are set. The rotation angle r of each group of anchor frames is 0° and 90°, for a total of 6 different anchor frames.

[0122] Step 6.4, construct the adaptive feature pooling module S3, and perform classification, position regression and direction regression on the anchor frame, and perform the following steps on (F N1 ~F N4 ) The ROI extracted by the region proposal module S4 is compressed into a one-dimensional feature vector, and then different feature maps are fused by summation.

[0123] Step 7: Train the 3D object detection network S based on special-shaped 3D convolution and obtain the performance indicators of the network under the KITTI dataset:

[0124] In this method, the KITTI dataset input method is set to input only the point cloud of ±45° in front, and only this part of the point cloud contains annotations. The input range is limited to 0≤x≤70.4, -40≤y≤40, -3≤x≤1, and the unit is meter.

[0125] Step 7.1: Construct a 3D target detection model (SPHNet) in a spherical coordinate system. The voxelization range is set to 0≤d≤80m, -45°≤θ≤45°, and -25°≤φ≤25°. The number of voxel grids is set to d′=1408,θ′=1088,φ′=64. The maximum number of training voxels is 20,000, and the maximum number of testing voxels is 40,000. The feature dimensions of the three feature extraction layers of the FPN layer in the RPN network are 128, 256, and 256, respectively. The downsampling intervals are 1, 2, and 2, respectively. The upsampling layer dimensions are 256, 256, and 256, respectively. The feature depth of the FPN layer output feature map is 256×3. The learning rate is set to 0.00045, and the learning rate adjustment policy adopts the cyclic learning rate. The learning rate increase step ratio is set to 0.3. The maximum magnification of the learning rate adjustment target is 10 times, and the minimum magnification is 1×10 -4 , train for 40 generations.

[0126] In step 7.2, the intermediate feature extraction module in the SECOND model is replaced with a special-shaped 3D convolution module. The detection performance of the original architecture and the replaced special-shaped 3D convolution architecture is tested to examine the impact of the special-shaped 3D convolution module on detection performance. The voxelization range is set to 0 ≤ x ≤ 70.4 m, -40 ≤ y ≤ 40 m, and -3 ≤ z ≤ 1 m. The number of voxel grids is set to x′ = 1408, y′ = 1600, and z′ = 41. The maximum number of training voxels is 16,000, and the maximum number of testing voxels is 40,000. In the RPN network, the feature extraction layers of the FPN layer have feature dimensions of 128 and 256, respectively, with downsampling intervals of 1 and 2, and upsampling layer dimensions of 256 and 256, respectively. The feature depth of the FPN layer output feature map is 512. The learning rate is set to 0.0018, and the learning rate adjustment policy adopts a ramp-up interval ratio of 0.45.

[0127] In step 7.3, the two models above are tested. (AC) represents the experimental group using the special-shaped 3D convolution kernel, and the control group is marked without (AC). At the same time, they are compared with the mainstream 3D object detection methods in recent years. The experimental results are shown in the table:

[0128]

[0129]

[0130] In step 7.4, we analyze the experimental results. Comparing the detection performance of the control group (SPHNet and SECOND networks) with the original model structure and the experimental group after replacing the module, we can see that the special-shaped 3D convolution module does have a slight impact on the model's detection performance, but the impact is not significant. However, it can greatly reduce the number of model parameters and memory space occupied, resulting in a significant improvement in efficiency. Furthermore, the detection performance of SPHNet (AC) using the special-shaped 3D convolution module is still significantly superior to that of other 3D object detection methods in the pedestrian category.

Claims

1. A 3D object detection method in spherical coordinates based on special-shaped 3D convolution, characterized in that: The following steps are included: Step 1: Obtain raw point cloud data through the lidar sensor; Step 2: Convert the received original point cloud data and the coordinate representation of the target bounding box from the 3D rectangular coordinate system to the spherical coordinate system; Step 3: Divide the space into voxels according to spherical coordinates, convert the sparse point cloud data into dense point cloud data, and achieve a balanced number of point clouds in each voxel to improve the utilization of point cloud spatial features and the efficiency of feature extraction; Step 4: Encode the voxel features of each voxel to obtain a voxel feature map F with a scale of i×j×k×4 in ; Step 5: Get the voxel feature map F in Use the special-shaped three-dimensional convolution to further extract spatial features and obtain the dense feature map matrix F t ; Step 6: Construct a 3D object detection network S based on heterogeneous 3D convolution and transform the dense feature map matrix F t Enter the network S; Step 7: Train the 3D object detection network S based on special-shaped 3D convolution to obtain the performance indicators of the network under the KITTI dataset; The step 5 is specifically as follows: The input layer of the special-shaped 3D convolution module uses two SubMConv3D convolution layer structures to perform preliminary extraction of voxel feature space, and then uses three 3D special-shaped convolution kernels to extract local spatial features; The special-shaped three-dimensional convolution module is specifically: In step 5.1, eight different shaped 3D convolution kernels are used for convolution, which are optimized for vertical and oblique surfaces based on the characteristics of radar scanning. Step 5.2: Encode the designed convolution kernels of different shapes using a mask. The length of the mask is the cube of the maximum side length of the convolution kernel. The special-shaped three-dimensional convolution kernel is designed. Each mask bit represents a spatial position and is encoded from left to right and from top to bottom. Step 5.3: To meet the two-dimensional input requirements of the region proposal network in step 6, it is necessary to apply the index matrix to reorganize the output into a feature matrix. At the output of the module, a SubMConv3D convolutional layer is used to convert the spatial dimension from three dimensions to the d-θ plane, and the sparse features obtained are converted into a dense feature map matrix output F. t .

2. The 3D target detection method in spherical coordinates based on special-shaped 3D convolution according to claim 1, characterized in that: The step 1 is specifically as follows: KITTI data is collected using the HDL-64E lidar sensor. During operation, the entire laser module is driven by a direct-drive motor and rotates at a set speed. The sensor has a horizontal field of view of 360°±0.09° and a vertical field of view of [+2°, -24.33°]. The raw data output by the sensor is represented as the distance d(θ, φ) from the object to the lidar and the object's reflection intensity I(θ, φ), where θ is the azimuth angle and φ is the pitch angle.

3. The 3D target detection method in spherical coordinates based on special-shaped 3D convolution according to claim 1, characterized in that: The KITTI dataset is used, which contains training samples and test samples. The samples include lidar point clouds and images, and are labeled into three categories: vehicles (Car), bicycles (Cyclist), and pedestrians (Pedestrian). The point cloud data refers to the original point cloud data received by the lidar sensor.

4. The 3D target detection method in spherical coordinates based on special-shaped 3D convolution according to claim 1, characterized in that: The step 2 is specifically as follows: Step 2.1: Convert the point cloud data into coordinates. Point cloud data is usually in (x i ,y i ,z i ,I i ), where x i ,y i ,z i Represents the coordinates of the i-th point in the rectangular coordinate system, I i Represents the reflection intensity of the point; For each point (x i ,y i ,z i ,I i ),calculate: Get (d i ,θ i ,φ i ,I i ), where d i represents the straight-line distance from the i-th point to the origin (lidar), θ i 、φ i is the azimuth and elevation angle of the i-th point, I i is the reflection intensity at that point; Step 2.2, coordinate transformation of the target bounding box; In the three-dimensional rectangular coordinate system, the bounding box includes the position of the bottom center point of the target (x, y, z), the target size (w, h, l) (w, h, l refer to the length, height, and width of the target size, respectively), and the rotation angle r. The spherical coordinates of the bottom center point (x, y, z) are converted to (d, θ, φ) according to the coordinate conversion formula described in 2.1, while (w, h, l, r) remain unchanged. The dataset annotation is converted to the spherical coordinate system bounding box (d, θ, φ, w, h, l, r).

5. The 3D target detection method in spherical coordinates based on special-shaped 3D convolution according to claim 1, characterized in that: The step 3 is specifically as follows: Step 3.1: The voxelization module takes the point cloud data in the spherical coordinate system converted in step 2.1 as input and transforms it into a point cloud according to the preset voxel size v. d ,v θ ,v φ , where v d ,v θ ,v φ are the length, width, and height of a single voxel, and the range of the three dimensions [d min ,d max ][θ min ,θ max ][φ min ,φ max ], d min ,d max is the minimum and maximum value of the voxel coordinate in the distance dimension; θ min ,θ max is the minimum and maximum value of the voxel coordinate in the azimuth dimension; φ min ,φ max is the minimum and maximum value of the voxel coordinate in the pitch angle dimension, and the space is divided into voxel grids. The number of grids in the three dimensions is: Step 3.2: In the spherical coordinate system, divide the spherical coordinate system into d′×θ′×φ′ with side length (dv d ,dv θ ,dsinθv φ ) as a voxel, dv d ,dv θ ,dsinθv φ They refer to the length, width, and height of the voxel respectively. The lengths of the three sides of the voxel are proportional to d. When d changes, the proportions of the three sides of the voxel remain unchanged, that is, the shape of the voxel is constant.

6. The 3D target detection method in spherical coordinates based on special-shaped 3D convolution according to claim 1, characterized in that: The step 4 is specifically as follows: Step 4.1, define the voxel coordinates and voxel center points in the spherical coordinate system: Among them, the (i,j,k)th voxel is a v d ,v θ ,v φ The center of the voxel is: Step 4.2: Extract features from each voxel and select a more lightweight voxel feature extractor. In order to decouple the voxel features from the absolute position of the voxel itself in space, the voxel feature extractor uses a manually designed feature calculate: Among them, d c ,θ c ,φ c is the voxel center, is the average value of the points within the voxel, that is, the relative distance from the average value of the points within the voxel to the center point of the voxel is used as the voxel feature. is the average value of the reflection intensity within the voxel; Step 4.3: Store the features of non-empty voxels (voxels containing data) to generate a 4D tensor with a size of C′×d′×θ′×φ′, where C′ is the feature dimension size. Calculate the feature encoding of all voxels to obtain a voxel feature map F with a scale of i×j×k×4. in .

7. The 3D target detection method in spherical coordinates based on special-shaped 3D convolution according to claim 1, characterized in that: The step 6 is specifically as follows: The 3D object detection network S based on special-shaped 3D convolution consists of four parts: FPN feature extraction module S1, bottom-up path enhancement module S2, adaptive feature pooling module S3, and region proposal module S4; Step 6.1, construct FPN feature extraction module S1, and transform F t Input into S1 and get F c1 、F c2 、F c3 、F c4 , F c4 Copy to F p4 , for F p4 Upsampling is performed, and convolution is performed on F C3 The number of channels is reduced, and then F p4 With F C3 The corresponding elements are added and convolution is performed to obtain F p3 ; Use convolution to transform F c2 The number of channels is reduced, and F c2 With F p3 Add and perform convolution operation to get F p2 ; Use convolution to transform F c1 The number of channels is reduced, and F c1 With F p2 Add and perform convolution operation to get F p1 ; Step 6.2, construct the bottom-up route enhancement module S2; since when the bottom-level features reach F P4 When the network passes through multiple layers, the underlying information is very vague, so a bottom-up route is added to compensate for and strengthen the positioning information; p1 Copy to F N1 After that, F N1 Downsampling, with F p2 Unit addition, convolution operation is performed to fuse the features to obtain F N2 ; for F N2 Downsampling, with F p3 Unit addition, convolution operation is performed to fuse the features to obtain F N3 , for F N3 Downsampling, with F p4 Unit addition, convolution operation is performed to fuse the features to obtain F N4 At the same time, a shortcut across multiple layers is added to the bottom-up and top-down modules respectively. Step 6.3: Use the region proposal module S4 to generate anchor frames. The rotation angle r of each anchor frame is 0° and 90°. Step 6.4, construct the adaptive feature pooling module S3, and perform classification, position regression and direction regression on the anchor frame, and perform the following steps on (F N1 ~F N4 ) The ROI extracted by the region proposal module S4 is compressed into a one-dimensional feature vector, and then different feature maps are fused by summation.

8. The 3D target detection method in spherical coordinates based on special-shaped 3D convolution according to claim 1, characterized in that: The step 7 is specifically as follows: The KITTI dataset input method is set to input only the point cloud of ±45° in front of the camera. Only this part of the point cloud contains annotations. The input range is limited to 0≤x≤70.4, -40≤y≤40, -3≤x≤1, and the unit is meter. Step 7.1, construct a 3D object detection model (SPHNet) in the spherical coordinate system; Step 7.2, in which the intermediate feature extraction module in the SECOND model is replaced with the special-shaped 3D convolution module, and the detection performance of the original structure and the replaced special-shaped 3D convolution structure is tested to verify the impact of the special-shaped 3D convolution module on the detection performance; Step 7.3, testing the above two models; In step 7.4, the experimental results are analyzed. By comparing the detection performance of the control group SPHNet network with the original model structure and the detection performance of the experimental group after the module replacement, it can be seen that the special-shaped three-dimensional convolution module does have a slight impact on the detection performance of the model, but the impact is not large. It can greatly reduce the number of model parameters and reduce the memory space occupied, thereby greatly improving efficiency.

Images