Two-stream 3D object detection learning method and system based on vertical representation

Abstract

The present invention discloses a dual-stream 3D target detection learning method and system based on vertical representation, wherein the method comprises the following steps: performing 3D voxelization and 2D voxelization processing on the original point cloud, respectively, and inputting the processed data into a sparse convolutional network to obtain 3D voxel features and 2D voxel features; adopting sparse pooling and expansion methods to establish a bidirectional relationship between 3D voxels and 2D voxels; using 3D voxel features and 2D voxel features as input, respectively, using a dense fusion module to perform feature fusion at different levels to obtain element-based aggregate features; using a detector based on center point regression and using IoU for score correction to obtain target detection results; the sparse fusion operation amplifies the receptive field of 3D voxels in the vertical direction, enriches the fine-grained semantic information within the 2D voxels, reduces the loss during sparse convolution, improves the accuracy of small targets in the target detection problem, and adopts a dense feature fusion method to fuse the learned 3D voxel and 2D voxel features of different sizes to further improve the detection performance.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and specifically relates to a dual-stream 3D target detection learning method and system based on vertical direction representation. Background Art

[0002] 3D object detection outputs the three-dimensional position, size, yaw angle, and category of objects in the scene based on input information such as images or point clouds. Cameras and lidar are two commonly used sensors for this task. The images acquired by the camera reflect the texture information of the scene, and when performing detection, it is necessary to predict the depth of the target. In contrast, lidar, as a photoelectric ranging sensor, directly converts measurement data into point cloud data that reflects spatial geometric information through the angle of laser emission and time of flight (ToF). With the promotion of lidar, and the characteristics of point clouds that accurately measure spatial information and are insensitive to light changes, 3D object detection based on pure point cloud data has become the current mainstream research direction.

[0003] 3D object detection algorithms can be categorized into two types based on the method used to extract point cloud features: grid-based methods and point-based methods. Grid-based methods are further divided into 3D voxel-based methods and 2D voxel-based methods. 3D voxel-based methods convert point clouds into relatively dense 3D voxels, effectively improving computational efficiency. However, the commonly used 3D sparse convolutions restrict the true diffusion from non-empty 3D voxels to empty 3D voxels, resulting in a limited receptive field in the vertical direction.

[0004] In contrast, the 2D voxelization method discretizes the point cloud into 2D cylindrical voxels, where each 2D voxel encodes all neighboring points in a specific XY coordinate. Although this paradigm obtains the full receptive field in the vertical direction and reduces the amount of computation, which can increase efficiency, it is weak in capturing fine-grained features. Summary of the Invention

[0005] In order to solve the problem of loss of fine-grained semantic information in the vertical direction existing in the prior art, the present invention provides a two-stream 3D object detection learning method based on vertical representation, which enriches the fine-grained semantic information within 2D voxels, reduces the loss during sparse convolution, and improves the accuracy of small targets in the target detection problem. A dense feature fusion method is used to fuse the learned 3D voxel and 2D voxel features of different sizes to further improve the detection performance.

[0006] To achieve the above objectives, the present invention adopts a technical solution: a two-stream 3D object detection learning method based on vertical representation, comprising the following steps:

[0007] The original point cloud is processed into 3D voxel and 2D voxel respectively to obtain the initial features of 3D voxels and 2D voxels;

[0008] The initial features of 3D voxels and 2D voxels are input into the two-stream sparse convolutional network to extract sparse 3D voxel and sparse 2D voxel features;

[0009] Sparse pooling and sparse expansion operations are used to establish a bidirectional relationship between sparse 3D voxel features and sparse 2D voxel features to obtain sparse fusion features;

[0010] The sparse fusion features are converted into dense features from a bird's-eye view and input into a 2D convolutional network for multi-level feature fusion to obtain the final fusion features.

[0011] A center point-based regression detector is used on the final fusion feature to obtain the final 3D object detection result.

[0012] The process of 3D voxelization of point clouds and extracting initial voxel features is as follows: dividing the measurement space into three-dimensional voxels of equal size, projecting the point cloud into the corresponding voxels, constructing the voxel index, dynamically encoding the point cloud in non-empty voxels, and obtaining the initial features of non-empty voxels after mean pooling operation.

[0013] The process of 2D voxelization of point clouds and extracting initial 2D voxel features is as follows: the XY plane of the measurement space is divided into 2D grids of equal size, and the point cloud is projected into the 2D voxels corresponding to the 2D grid. The PointNet network is used and the maximum pooling operation is used to obtain the 2D voxel input features.

[0014] The process of feature extraction for the initial features of 3D voxels and 2D voxels is as follows: using 3D and 2D sparse convolution networks with 4 sparse convolution modules, the 3D voxel features are downsampled by one, two, four and eight times, including regular 3D sparse convolution and substream 3D sparse convolution. Regular 3D sparse convolution completes the spatial downsampling operation, and the substream 3D sparse convolution constrains the output position to be the same as the input position. At the same time, the 2D voxel features are downsampled by one, two, four and eight times, including regular 2D sparse convolution and substream 2D ​​sparse convolution. Regular 2D sparse convolution completes the spatial downsampling operation, and the substream 2D ​​sparse convolution constrains the output position to be the same as the input position.

[0015] The process of establishing a bidirectional relationship between sparse 3D voxel and 2D voxel features is as follows: for the conversion relationship from 3D voxel to 2D voxel, non-empty 3D voxel features are mapped to 2D voxel features in the same vertical direction using sparse pooling operations. The 2D voxel features are spatially consistent with the original sparse 2D voxel features, and then 2D substream convolution is used to convert the feature dimensions; for the conversion relationship from 2D voxel to 3D voxel, non-empty 2D voxel features are first subjected to 2D substream convolution, and then subjected to sparse expansion operations to form 3D voxel features of the corresponding spatial position. The 3D voxel and 2D voxel features are summed element-by-element to obtain sparse fusion features.

[0016] The sparse fusion features are converted into dense features from a bird's-eye view and input into a 2D convolutional network for multi-level feature fusion to obtain the final fusion features. Specifically, the sparse fusion features are converted into dense 3D voxel 2D voxel features from a bird's-eye view, and 2D convolution is used to process the dense 3D voxel features and dense 2D voxel features. First, dense features are extracted using eight-fold and sixteen-fold downsampling convolution blocks for the 3D voxel branch and the 2D voxel branch, respectively. The 3D and 2D voxel features of the same size are fused, and features of different scales are combined through upsampling and splicing operations. Finally, the final fusion features are aggregated through convolution operations.

[0017] When using a center-point-based regression detector to obtain the final 3D object detection result, a center-point-based regression detector is used to perform regression and classification score prediction based on the target, and the classification score is corrected using IoU to obtain the final prediction score.

[0018] Based on the technical concept of the method, the present invention also provides a two-stream 3D target detection learning system based on vertical representation, an initial 3D voxel-2D voxel feature extraction module, a sparse feature extraction module, a sparse fusion module, a dense feature fusion module and a target prediction and regression module;

[0019] The initial feature extraction module is used to take the original point cloud as input, perform 3D voxelization and 2D voxelization on the point cloud according to the set size, and obtain the 3D voxel-2D voxel initial features;

[0020] The sparse feature extraction module is used to input the initial features of 3D voxels and 2D voxels into the two-stream sparse convolutional network to extract sparse 3D voxel and sparse 2D voxel features;

[0021] The sparse fusion module is used to establish a bidirectional relationship between sparse 3D voxel features and sparse 2D voxel features to obtain sparse fusion features;

[0022] The dense feature fusion module is used to convert the sparse fusion features into dense features from a bird's-eye view, and input them into the 2D convolutional network for multi-level feature fusion to obtain the final fusion features;

[0023] The target prediction and regression module is used to use the center point-based regression detector on the final fusion feature and obtain the final 3D target detection result.

[0024] In addition, a computer device is provided, including a processor and a memory, wherein an executable program is stored in the memory. When the processor executes the executable program, the vertical representation-based dual-stream 3D object detection learning method described in the present invention can be executed.

[0025] And a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the dual-stream 3D target detection learning method based on vertical representation described in the present invention.

[0026] Compared with the prior art, the present invention has at least the following beneficial effects: the present invention addresses the problem of loss of fine-grained semantic information in the vertical direction caused by the current 3D target detection gridding method, and proposes a one-stage two-stream 3D target detection framework based on vertical representation; to address the problem that sparse convolution of 3D voxel features limits the diffusion between non-empty 3D voxels to empty 3D voxels, a sparse fusion operation is proposed. The sparse fusion operation amplifies the receptive field of 3D voxels in the vertical direction, enriches the fine-grained semantic information within 2D voxels, reduces the loss during sparse convolution, and can improve the accuracy of small targets in target detection problems; sparse pooling operations and sparse expansion operations are used to obtain the bidirectional relationship between 3D voxel features and 2D voxel features, and the local feature content from the 3D voxel branch improves the 2D voxel representation; to address the feature fusion problem, a dense feature fusion method is adopted to fuse the learned 3D voxel and 2D voxel features of different sizes to further improve the detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A block diagram of a dual-stream 3D object detection method based on vertical representation provided by an embodiment of the present invention.

[0028] Figure 2 Schematic diagram of the 3D voxel-to-2D voxel downsampling process provided for the implementation of the present invention.

[0029] Figure 3 A comparison chart of the present invention and other detection methods in terms of inference time and performance.

[0030] Figure 4 The 3D object detection method provided for the implementation of the present invention is tested on the Waymo Open dataset. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0032] In the description of the present invention, it is to be understood that the terms “include” and “comprise” indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0033] The embodiment of the present invention provides a dual-stream 3D target detection method based on vertical representation, which reduces the loss of fine-grained semantic information in the vertical direction and improves the target detection effect by fusing 3D voxel-2D voxel features at different depths. Figure 1 As shown in the figure, the original point cloud is first processed into 3D voxel and 2D voxel respectively, and then input into the sparse convolutional network. 3D voxel features and 2D voxel features are obtained; sparse pooling and sparse dilation methods are used to establish a bidirectional relationship between 3D voxels and 2D voxels; using 3D voxel features and 2D voxel features as input, a dense fusion module is used to fuse features at different levels to obtain element-based aggregate features; a center point-based regression detector and IoU are used to correct the score to obtain the final target detection result. The specific implementation steps are as follows:

[0034] Step 1: Perform 3D voxelization and 2D voxelization on the original point cloud. Figure 1 As shown, the present invention, as a dual-stream target detection method, includes two branches, namely a 3D voxelization branch and a 2D voxelization branch.

[0035] In the 3D voxelization branch, given a point cloud Where N is the number of point clouds. The present invention divides the measurement space into 3D voxels of equal size with a resolution of L×W×H. Then the point cloud S is quantized by a predefined 3D voxel size to generate a point-to-3D voxel index Subsequently, a mean pooling layer is used to obtain the initial features of non-empty 3D voxels;

[0036] In the 2D voxelization branch, the XY plane of the measurement space is divided into 2D grids of equal size with a resolution of L×W. Then the point cloud S is quantized by the predefined 2D voxel size. Due to the consistency of 3D voxels and 2D voxels in the top-down perspective, the 2D voxel index can be obtained by directly removing the vertical index in the 3D voxel index. Finally, a PointNet network and a max pooling operation are used to generate 2D voxel initial features.

[0037] Step 2: Use a two-stream sparse convolutional backbone network to process the 3D voxel initial features and 2D voxel initial features respectively to extract and generate sparse 3D voxel and sparse 2D voxel features.

[0038] like Figure 1 As shown, the present invention uses a 3D and 2D sparse convolutional network with four sparse convolutional modules, where the four sparse convolutional modules are deployed at 1x, 2x, 4x, and 8x downsampling sizes, respectively, to sequentially generate sparse 3D voxel and sparse 2D voxel features. The 3D voxel branch contains a 3D regular sparse convolutional layer and several 3D substream sparse convolutional layers, and the 2D voxel branch contains a 2D regular sparse convolutional layer and several 2D substream sparse convolutional layers; regular sparse convolution implements spatial downsampling, and substream sparse convolution constrains the output and input feature positions to be the same.

[0039] like Figure 2 As shown in the figure, the kernel size, stride, and padding operation of the 2D and 3D regular sparse convolutions in the horizontal dimension are set to the same to ensure that the 3D voxel features and 2D voxel features with the same spatial semantics are in the same position in the XY plane.

[0040] Step 3: Use the sparse fusion layer (SFL) to establish a bidirectional relationship between the input sparse 3D voxel features and sparse 2D voxel features.

[0041] According to step 1, each 3D voxel index is The corresponding feature is N v is the number of non-empty 3D voxels, D v is the 3D voxel feature dimension. Similarly, each 2D voxel index is The corresponding characteristics are For a specific XY coordinate containing a non-empty 2D voxel, you can find the 3D voxel with the same horizontal index and compare it to get:

[0042] c j ={v i |l i =l j , w i =w j , i=1,…,N v}

[0043] c j The collection of 3D voxels-2D voxel indices Thus, non-empty 3D voxels are projected onto 2D voxels at the same horizontal position, which is called sparse pooling. Conversely, non-empty 2D voxels can also be scattered to matching 3D voxels with reverse indices, which is represented as sparse dilation.

[0044] For sparse 3D voxels as input, the sparse feature fusion layer first calculates the 3D voxel-2D voxel index Sparse features interact with each other through two paths. For the 3D voxel-2D voxel relationship, the 3D voxel features are accumulated in the vertical direction through the sparse pooling operation to obtain 2D voxel class features with a size of (N p ,D v ). Apply 2D substream sparse convolution to transform the 2D voxel-like features that are spatially consistent with the original sparse features into 2D voxel-like features that are consistent with the original feature space and dimension, generating a size of (N p ,D p )’s 2D voxel-like features

[0045] For the input sparse 2D voxel features, we first use 2D substream sparse convolution to perform size conversion, and then use sparse expansion operation to generate a size of (N v ,D v )’s 2D voxel-like features

[0046] Subsequently, element-wise summation is applied to the 3D voxel-like and 2D voxel-like features respectively to obtain sparse fused features:

[0047]

[0048]

[0049] Step 4: Convert the sparse fusion features into the top-down view (BEV) dense features and perform feature fusion.

[0050] The sparse fusion features obtained in the previous stage are converted into dense 3D voxel and 2D voxel features of the top view (BEV) using a densification operation. This feature is input into the dense feature fusion network module (DFN), which makes full use of the dense 3D voxel and 2D voxel features and performs multi-scale feature fusion. Figure 1As shown in the figure, in the dense feature fusion network module, convolutional modules Block(M,D) with 8x and 16x downsampling sizes are used to extract dense features for the 3D voxel and 2D voxel branches, respectively. Block(M,D) represents M convolutional layers and D output dimensions. Next, dense 3D voxel and 2D voxel features of the same scale are fused through element-by-element summation. Features of different scales are then combined through upsampling and concatenation operations. Finally, another convolutional block operation is performed to aggregate the final fused features.

[0051] Step 5: Use the final fusion features as input to train the object detector, using a detector based on center point regression and IoU correction method to combine the regression accuracy and classification score.

[0052] In addition to using the classification prediction head and 3D box regression detection, the intersection-over-union prediction is used to estimate the intersection-over-union ratio between the detection box and the ground-truth box. The final prediction score is given by the following correction function:

[0053]

[0054] Among them S cls is the classification score, IoU pred is the predicted intersection-over-union ratio, and α is a hyperparameter used to balance S cls and IoU pred The IoU is supervised by the L1 loss function and encoded as (2*IoU-0.2)∈[-1, 1].

[0055] In addition, the distance-related IoU loss is added to further optimize the target center regression, specifically:

[0056]

[0057] Where IoU(b, b gt ) represents the 3D prediction box b and the real box b gt The intersection-and-union ratio between them, c represents the predicted box b and the real box b gt The center point offset between the predicted box b and the real box b gt The diagonal distance covered by the minimum enclosing cuboid.

[0058] The overall loss function is defined as:

[0059]

[0060] Where γ is the loss weight, and represents the classification and bounding box regression losses, using focal loss and L1 loss respectively.

[0061] Experiments and Analysis: We evaluate our method on the largest Waymo Open 3D object detection dataset and compare it with the latest methods. We also conduct ablation experiments to analyze the effectiveness of each module.

[0062] Waymo Open Dataset is the largest autonomous driving dataset, collecting data in multiple locations under different weather conditions. The dataset consists of 1,000 annotated sequences, of which 798 sequences (approximately 158,000 point cloud samples) are divided into training sets, and the other 202 sequences (approximately 40,000 point cloud samples) constitute the validation set. The point cloud data is truncated to within 75 meters. In addition to coordinates and reflection intensity, the point cloud data also includes elongation. The evaluation indicators of this dataset are mean average precision (mAP) and orientation-weighted mean accuracy (mAPH), where the IoU thresholds for vehicles and pedestrians are set to 0.7 and 0.5, respectively. The dataset is divided into two difficulty levels, targets with more than 5 points are LEVEL_1, and targets containing at least 1 point are LEVEL_2.

[0063] This paper presents key results and comparisons with state-of-the-art algorithms on the Waymo Open dataset. This paper can be deployed using a variety of neural networks, with four variants at different parameter scales, denoted as VPFusion V1-V4. VPFusion V1 deploys a commonly used 3D voxel CNN for the 3D voxel branch and a VGGNet with sparse 2D convolutions for the 2D voxel branch; VPFusion V2 deploys a 3D voxel CNN and VGGNet with a residual architecture; VPFusion V3 deploys a commonly used 3D voxel CNN and ResNet18; and VPFusion V4 deploys a commonly used 3D voxel CNN and ResNet34. All variants are trained using the same protocol.

[0064] As shown in Table 1, our first-stage detector achieves state-of-the-art results in all comparisons of first-stage and second-stage detectors, significantly outperforming both the 3D voxel-based AFDetV2 and the 2D voxel-based PillatNet. This performance is particularly pronounced for small objects such as pedestrians and cyclists. Our VPFusion V4 performs significantly better than the two-stage network FSD for cars and pedestrians.

[0065] In addition, as shown in Table 2, the present invention also conducts experiments on the WOD test dataset and compares it with all published detection methods that use only lidar data. It is found that the present invention leads all methods and significantly surpasses PV-RCNN++ and other grid-based detection methods, which proves the effectiveness of the detection method of the present invention and the 3D voxel-2D voxel encoding paradigm.

[0066] In addition, as shown in Table 3, the present invention also compares and evaluates the performance of VPFusion-V3 and other advanced detection methods that use only lidar data on the nuScenes test dataset. It is found that the present invention leads previous methods in terms of mAP and NDS indicators and shows considerable improvement, especially in the detection of vertically sensitive objects (people, bicycles).

[0067] At the same time, if Figure 3 As shown in the figure, the performance and inference time of our method are compared with the most advanced methods. The inference times of the four VPFusion variants are 68ms, 72ms, 78ms, and 82ms, respectively. Compared with PillarNet, although the inference speed is slower, the detection accuracy is significantly improved. On the other hand, compared with the advanced FSD, the inference speed of our method is faster while the results are almost the same. This proves that our method achieves a better balance between performance and inference speed. Figure 4 The detection effect diagram of the present invention on the Waymo Open dataset is shown. The method proposed in the present invention can accurately identify targets in scenes with dense target distribution and sparse point clouds ( Figure 4 first two lines). Figure 4 The last two rows show some failure scenarios due to sparse point clouds and confusion between foreground and background point clouds, which can be solved by temporal point cloud fusion and image-point cloud fusion.

[0068] Table 1: Comparison of Waymo Open dataset validation results at different difficulty levels

[0069]

[0070] Table 2: Comparison of Waymo Open dataset test set test results under different difficulty levels

[0071]

[0072] Table 3: Comparison of detection results of the nuScenes dataset test set

[0073]

[0074] In order to verify the effectiveness of the method described in the present invention, the present invention conducted extensive experiments based on the Waymo Open dataset.

[0075] Because VPFusion combines 3D voxel and 2D voxel encoding paradigms, the popular CenterPoint and PillarNet were used as two baselines to verify the effectiveness of 3D voxel-2D voxel encoding fusion. As shown in Table 4, VPFusion achieves a 0.5 mAPH performance improvement at LEVEL_2 difficulty by fusing dense 3D voxel features and 2D voxel features through only the DFN module. If only bidirectional relationships are introduced through SFL, a 1.7 mAPH performance improvement can be achieved. Simultaneously deploying a dense feature fusion network (DFN) and a sparse feature fusion layer (SFL) significantly improves performance compared to the baseline. Furthermore, the experiment used the non-maximum suppression method (NMS) to combine the predictions of CenterPoint and PillarNet. As shown in the third row of Table 3, although this model achieves the same performance as adding the DFN module, it significantly increases latency.

[0076] In order to verify the necessity of establishing a bidirectional relationship between 3D voxels and 2D voxels in the SFL layer, since the SFL module consists of two branches, 3D voxel to 2D voxel and 2D voxel to 3D voxel, the present invention conducts comparative experiments on the performance of unidirectional and bidirectional relationships. As shown in Table 4, each unidirectional relationship can achieve significant performance improvement, and the best effect can be achieved by applying a bidirectional relationship. It is also noted that the path from 3D voxel to 2D voxel contributes more to the module of the present invention. After analysis, this is because the local feature content from the 3D voxel branch improves the 2D voxel representation. In addition, as shown in Table 5, the present invention compares the effects of deploying sparse fusion modules (SFL) at different stages. The experiment shows that the performance of the model improves with the stacking of sparse fusion modules (SFL) at each stage. This is because the sparse fusion module (SFL) amplifies the receptive field of 3D voxels in the vertical direction, enriches the fine-grained semantic information within the 2D voxels, and reduces the loss during sparse convolution.

[0077] Finally, the present invention verifies the effect of the dense feature fusion network (DFN) module. In the framework of the present invention, the dense feature fusion is after the sparse backbone network, but all information can be aggregated in the sparse fusion module (SFL). Therefore, an experiment is designed to study whether similar performance can be achieved by applying the dense feature fusion network to process a single 3D voxel or 2D voxel branch. After the last sparse fusion module (SFL), the single 3D voxel or 2D voxel branch is retained, and the dense feature fusion network is applied to process the separate dense 3D voxel or 2D voxel feature map. As shown in Table 6, the DFN module proposed in the present invention has the best performance.

[0078] Table 4: Comparison of the impact of individual modules

[0079]

[0080] Table 5: Comparison of unidirectional and bidirectional relationships

[0081]

[0082] Table 6: Comparison of Sparse Fusion Module (SFL) deployment at different stages

[0083]

[0084] Table 7: Comparison of the necessity of dense feature fusion network (DFN)

[0085]

[0086] The entire sparse convolutional network of the present invention includes four sparse convolution modules, wherein the 3D sparse convolution takes non-empty 3D voxels and their corresponding 3D voxel features and indices as input for feature extraction, and the 2D sparse convolution takes non-empty 2D voxels and their corresponding 2D voxel features and indices as input for feature extraction; the four sparse convolution modules downsample the 3D voxel features and 2D voxel features by one, two, four, and eight times, respectively, and the obtained 2D voxel features are directly converted into dense top-view features, and the obtained 3D voxel features are accumulated in the Z-axis direction to obtain dense top-view features; each sparse convolution module also includes a sparse feature fusion layer, wherein the 3D voxel features are mapped to the corresponding 2D voxel features through a sparse pooling operation, the 2D voxel features are scattered to 3D voxel features with reverse indices through a sparse broadcasting operation, and the sparse feature fusion layer fuses the sparse 3D voxel features and the sparse 2D voxel features;

[0087] The dense features obtained by processing the sparse backbone network are input into the 2D convolutional neural network for multi-level feature fusion. Specifically, the processed dense features are input into the dense fusion module, and the 3D voxel and 2D voxel branches are processed respectively by eightfold and sixteenth-fold downsampling, and the 3D voxel and 2D voxel features of the same scale are integrated at the element level. Then, upsampling is used to splice features of different scales, and the final fused features are fused through another convolution block. The final fused features are used for target detection and training regression, and a detector based on center point regression and the IoU correction method are used to combine the regression accuracy and classification score.

[0088] Based on the above concept, the present invention provides a dual-stream 3D object detection learning system based on vertical representation, an initial 3D voxel-2D voxel feature extraction module, a sparse feature extraction module, a sparse fusion module, a dense feature fusion module, and a target prediction and regression module;

[0089] The initial feature extraction module is used to take the original point cloud as input, perform 3D voxelization and 2D voxelization on the point cloud according to the set size, and obtain the 3D voxel-2D voxel initial features;

[0090] The sparse feature extraction module is used to input the initial features of 3D voxels and 2D voxels into the two-stream sparse convolutional network to extract sparse 3D voxel and sparse 2D voxel features;

[0091] The sparse fusion module is used to establish a bidirectional relationship between sparse 3D voxel features and sparse 2D voxel features to obtain sparse fusion features;

[0092] The dense feature fusion module is used to convert the sparse fusion features into dense features from a bird's-eye view, and input them into the 2D convolutional network for multi-level feature fusion to obtain the final fusion features;

[0093] The target prediction and regression module is used to use the center point-based regression detector on the final fusion feature and obtain the final 3D target detection result.

[0094] In addition, the present invention can also provide a computer device, including a processor and a memory, the memory is used to store a computer executable program, the processor reads part or all of the computer executable program from the memory and executes it, and when the processor executes part or all of the computer executable program, it can implement the dual-stream 3D target detection learning method based on vertical representation described in the present invention.

[0095] The computer device may be a notebook computer, a vehicle-mounted computer, a desktop computer or a workstation.

[0096] The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or an off-the-shelf field programmable gate array (FPGA).

[0097] At the same time, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the dual-stream 3D target detection learning method based on vertical representation described in the present invention can be implemented.

[0098] The memory of the present invention may be an internal storage unit of a laptop computer, a vehicle-mounted computer, a desktop computer or a workstation, such as a memory or a hard disk; or an external storage unit such as a mobile hard disk or a flash memory card.

[0099] Computer-readable storage media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSD) or optical disks, etc. Among them, random access memory may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

[0100] The above description is only a preferred specific implementation method of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, can make equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A two-stream 3D object detection learning method based on vertical representation, characterized by: The following steps are involved: The original point cloud is processed into 3D voxel and 2D voxel respectively to obtain the initial features of 3D voxels and 2D voxels; The initial features of 3D voxels and 2D voxels are input into the two-stream sparse convolutional network to extract sparse 3D voxel and sparse 2D voxel features; Sparse pooling and sparse expansion operations are used to establish a bidirectional relationship between sparse 3D voxel features and sparse 2D voxel features to obtain sparse fusion features; The sparse fusion features are converted into dense features from a bird's-eye view and input into a 2D convolutional network for multi-level feature fusion to obtain the final fusion features. A center point-based regression detector is used on the final fusion feature to obtain the final 3D object detection result.

2. The vertical representation-based two-stream 3D object detection learning method according to claim 1, characterized in that The process of 3D voxelization of point clouds and extracting initial voxel features is as follows: dividing the measurement space into three-dimensional voxels of equal size, projecting the point cloud into the corresponding voxels, constructing the voxel index, dynamically encoding the point cloud in non-empty voxels, and obtaining the initial features of non-empty voxels after mean pooling operation.

3. The two-stream 3D object detection learning method based on vertical representation according to claim 1 is characterized in that The process of 2D voxelization of point clouds and extracting initial 2D voxel features is as follows: the XY plane of the measurement space is divided into 2D grids of equal size, and the point cloud is projected into the 2D voxels corresponding to the 2D grid. The PointNet network is used and the maximum pooling operation is used to obtain the 2D voxel input features.

4. The two-stream 3D object detection learning method based on vertical representation according to claim 1 is characterized in that The process of feature extraction for the initial features of 3D voxels and 2D voxels is as follows: using 3D and 2D sparse convolution networks with 4 sparse convolution modules, the 3D voxel features are downsampled by one, two, four and eight times, including regular 3D sparse convolution and substream 3D sparse convolution. Regular 3D sparse convolution completes the spatial downsampling operation, and the substream 3D sparse convolution constrains the output position to be the same as the input position. At the same time, the 2D voxel features are downsampled by one, two, four and eight times, including regular 2D sparse convolution and substream 2D ​​sparse convolution. Regular 2D sparse convolution completes the spatial downsampling operation, and the substream 2D ​​sparse convolution constrains the output position to be the same as the input position.

5. The two-stream 3D object detection learning method based on vertical representation according to claim 1 is characterized in that The process of establishing a bidirectional relationship between sparse 3D voxel and 2D voxel features is as follows: for the conversion relationship from 3D voxel to 2D voxel, non-empty 3D voxel features are mapped to 2D voxel features in the same vertical direction using sparse pooling operations. The 2D voxel features are spatially consistent with the original sparse 2D voxel features, and then 2D substream convolution is used to convert the feature dimensions; for the conversion relationship from 2D voxel to 3D voxel, non-empty 2D voxel features are first subjected to 2D substream convolution, and then subjected to sparse expansion operations to form 3D voxel features of the corresponding spatial position. The 3D voxel and 2D voxel features are summed element-by-element to obtain sparse fusion features.

6. The vertical representation-based two-stream 3D object detection learning method according to claim 1, characterized in that The sparse fusion features are converted into dense features from a bird's-eye view and input into a 2D convolutional network for multi-level feature fusion to obtain the final fusion features. Specifically, the sparse fusion features are converted into dense 3D voxel 2D voxel features from a bird's-eye view, and 2D convolution is used to process the dense 3D voxel features and dense 2D voxel features. First, dense features are extracted using eight-fold and sixteen-fold downsampling convolution blocks for the 3D voxel branch and the 2D voxel branch, respectively. The 3D and 2D voxel features of the same size are fused, and features of different scales are combined through upsampling and splicing operations. Finally, the final fusion features are aggregated through convolution operations.

7. The two-stream 3D object detection learning method based on vertical representation according to claim 1 is characterized in that When using a center-point-based regression detector to obtain the final 3D object detection result, a center-point-based regression detector is used to perform regression and classification score prediction based on the target, and the classification score is corrected using IoU to obtain the final prediction score.

8. A two-stream 3D object detection learning system based on vertical representation, characterized by: Initial 3D voxel-2D voxel feature extraction module, sparse feature extraction module, sparse fusion module, dense feature fusion module and target prediction and regression module; The initial feature extraction module is used to take the original point cloud as input, perform 3D voxelization and 2D voxelization on the point cloud according to the set size, and obtain the 3D voxel-2D voxel initial features; The sparse feature extraction module is used to input the initial features of 3D voxels and 2D voxels into the two-stream sparse convolutional network to extract sparse 3D voxel and sparse 2D voxel features; The sparse fusion module is used to establish a bidirectional relationship between sparse 3D voxel features and sparse 2D voxel features to obtain sparse fusion features; The dense feature fusion module is used to convert the sparse fusion features into dense features from a bird's-eye view, and input them into the 2D convolutional network for multi-level feature fusion to obtain the final fusion features; The target prediction and regression module is used to use the center point-based regression detector on the final fusion feature and obtain the final 3D target detection result.

9. A computer device, characterized in that: The invention comprises a processor and a memory, wherein an executable program is stored in the memory, and when the processor executes the executable program, the dual-stream 3D target detection learning method based on vertical representation according to any one of claims 1 to 7 can be executed.

10. A computer-readable storage medium, characterized in that A computer program is stored in a computer-readable storage medium, and when the computer program is executed by a processor, it can implement the vertical representation-based dual-stream 3D object detection learning method described in any one of claims 1 to 7.

Images