A 3D target detection method based on virtual point cloud enhancement

Abstract

This invention relates to a 3D target detection method based on virtual point cloud enhancement, comprising the following steps: acquiring a 2D image and a LiDAR point cloud; obtaining a sparse depth map; converting the sparse depth map into a dense depth map; converting the dense depth map into a dense pseudo-point cloud; supplementing the LiDAR point cloud with the dense pseudo-point cloud to generate candidate boxes; obtaining pseudo-point clouds within the candidate boxes; querying the nearest LiDAR point clouds of each point in the pseudo-point cloud within the candidate boxes in parallel; querying the neighboring pseudo-point clouds for each point in the pseudo-point cloud within the candidate boxes, and obtaining pseudo-point cloud features within the candidate boxes based on the pseudo-point clouds within the candidate boxes, the neighboring pseudo-point clouds, and the nearest LiDAR point clouds; obtaining LiDAR point cloud ROI features; aggregating the pseudo-point cloud ROI features within the candidate boxes; fusing the LiDAR point cloud ROI features and the pseudo-point cloud ROI features within the candidate boxes to optimize the candidate boxes. Compared with existing technologies, this invention has advantages such as improving the accuracy of 3D target detection.

Description

Technical Field

[0001] This invention relates to fields such as autonomous driving and machine vision, and in particular to a 3D target detection method based on virtual point cloud enhancement. Background Technology

[0002] 3D object detection is a crucial component of autonomous driving perception systems. It involves collecting data about the surrounding driving environment using devices such as LiDAR (Light Detection and Ranging), RGB cameras, and millimeter-wave radar. Specific detection algorithms then output the 3D position information of the target, including the center point of the target's bounding box, its length, width, and height, and its rotation angle in the horizontal plane. In the field of autonomous driving, the objects perceived by 3D object detection are primarily dynamic targets, such as vehicles, pedestrians, and cyclists.

[0003] 3D object detection methods based on LiDAR point clouds can be broadly categorized into point-based, voxel-based, and projection-based methods. Point-based methods directly learn the features of unstructured point clouds to generate candidate boxes. However, point-based methods have insufficient learning ability for unstructured point clouds, and their learning efficiency is low when the number of point clouds is large. Voxel-based methods convert irregular point clouds into regular voxels and use 3D sparse convolution to extract features from sparse voxels for generating candidate boxes. To compensate for structural distortion in the preprocessing stage, voxel-based methods require the introduction of complex modules, reducing detection efficiency. Projection-based methods project the point cloud onto a specific plane, discretizing it into a two-dimensional image representation, and then apply two-dimensional convolution for feature extraction and candidate box generation. Two-dimensional projected views preserve metric space and physical priors (depth information, geometry) while providing a more compact data representation, but inevitably lead to information loss; for example, top-view projection loses height information, resulting in slightly compromised accuracy and robustness. Perception schemes based on pure LiDAR point clouds often struggle to provide accurate detection results for occluded or distant targets, because the point clouds on the surfaces of these targets are typically very sparse.

[0004] 3D object detection methods based on multimodal fusion (LiDAR point cloud and RGB image) can overcome the limitations of single sensors, complementing each other's advantages and improving the completeness, accuracy, and reliability of the detection method. Based on the fusion location of the LiDAR point cloud and the image, fusion methods are divided into pre-fusion, mid-fusion, and post-fusion. Pre-fusion refers to aligning and fusing the image and LiDAR point cloud before they are input into the network, i.e., data-level fusion; mid-fusion refers to extracting features from the image and LiDAR point cloud separately and fusing them in the network, i.e., feature-level fusion; post-fusion refers to processing the image and LiDAR point cloud separately by the network to obtain the network output of each modality, and then fusing the network outputs of different modalities, i.e., decision-level fusion. 3D object detection based on multimodal fusion usually requires the design of complex multimodal data augmentation strategies to compensate for insufficient training data. However, due to the different data structures of 2D images and LiDAR point clouds, it is difficult to achieve synchronous data augmentation of LiDAR point clouds and 2D images. Some works use features and semantic information from RGB images to enhance point clouds and employ classic LiDAR point cloud-based 3D object detection networks. However, these works do not consider the sparse point cloud problem on the target surface under conditions such as distance or occlusion. Existing 3D object detection models based on virtual point cloud enhancement fuse features from LiDAR point clouds and pseudo-point clouds during the ROI pooling stage, but do not consider the inaccuracy of depth completion, thus affecting the accuracy of 3D object detection. Summary of the Invention

[0005] The purpose of this invention is to provide a 3D target detection method based on virtual point cloud enhancement to improve the accuracy of 3D target detection.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A 3D object detection method based on virtual point cloud enhancement includes the following steps:

[0008] Acquire 2D images and LiDAR point clouds;

[0009] The LiDAR point cloud is projected onto a 2D plane to obtain a sparse depth map;

[0010] Based on the depth completion method, and with the assistance of the 2D image, the sparse depth map is converted into a dense depth map;

[0011] Convert the dense depth map into a dense pseudo-point cloud;

[0012] The dense pseudo-point cloud is used to supplement the LiDAR point cloud, and candidate boxes are generated based on the supplemented LiDAR point cloud.

[0013] In parallel, determine whether each point in the dense pseudo-point cloud is within the candidate box. If it is, obtain the pseudo-point cloud within the candidate box; otherwise, do not perform any operation.

[0014] Parallel queries are performed on the nearest LiDAR points of each point in the pseudo-point cloud within the candidate box to obtain the nearest LiDAR point cloud.

[0015] The pseudo point cloud in the candidate box is queried by querying the neighboring pseudo point cloud, and the pseudo point cloud features in the candidate box are obtained based on the pseudo point cloud in the candidate box, the neighboring pseudo point cloud, and the nearest LiDAR point cloud.

[0016] Query the voxel features of the nearest LiDAR point cloud and obtain the ROI features of the LiDAR point cloud by aggregation.

[0017] Based on the pseudo-point cloud features within the candidate box, the pseudo-point cloud ROI features within the candidate box are obtained by aggregation.

[0018] The LiDAR point cloud ROI features and the pseudo point cloud ROI features within the candidate boxes are fused together, and the candidate boxes are optimized to obtain 3D target detection boxes.

[0019] Furthermore, the 2D image and LiDAR point cloud are acquired using an RGB camera and a LiDAR sensor, respectively.

[0020] Furthermore, it also includes calibrating the intrinsic and extrinsic parameters of the RGB camera and LiDAR sensor before acquiring the 2D image and LiDAR point cloud.

[0021] Furthermore, the dense depth map is converted into a dense pseudo-point cloud using extrinsic parameters from an RGB camera and a LiDAR sensor.

[0022] Furthermore, the specific steps for supplementing the LiDAR point cloud include:

[0023] Convert the dense depth map corresponding to the dense pseudo-point cloud into a pseudo-point cloud top view;

[0024] Convert the sparse depth map corresponding to the LiDAR point cloud into a top view of the LiDAR point cloud.

[0025] The pseudo point cloud top view and the LiDAR point cloud top view are respectively divided into different sub-images, namely the pseudo point cloud top view sub-image and the LiDAR point cloud top view sub-image;

[0026] The pseudo point cloud top view sub-image and the LiDAR point cloud top view sub-image are divided into the same sub-grid;

[0027] Search for pixels with non-zero values ​​in the sub-grid of the LiDAR point cloud top view sub-image, and calculate the local origin coordinates of the sub-grid in the pseudo-point cloud top view sub-image based on the coordinates in the LiDAR point cloud top view sub-image.

[0028] Based on the local origin coordinates of the subgrid in the top view sub-image of the pseudo-point cloud, a corresponding dense pseudo-point cloud is selected to supplement the LiDAR point cloud.

[0029] Furthermore, a one-stage process using a two-stage 3D object detection model is employed to generate candidate bounding boxes.

[0030] Furthermore, taking each point in the pseudo-point cloud within the candidate box as the center of a sphere, within a preset sphere radius, the LiDAR point cloud within the sphere is taken as the nearest neighbor LiDAR point cloud of the pseudo-point cloud within the candidate box.

[0031] Furthermore, the specific steps for obtaining the pseudo-point cloud features within the candidate bounding box include:

[0032] Extract the homomorphic structure features of the pseudo-point cloud within the candidate box and the neighboring pseudo-point clouds;

[0033] Extract the heteromorphic structural features between the pseudo point cloud and the nearest LiDAR point cloud within the candidate box;

[0034] The homomorphic and heteromorphic structural features are fused together as pseudo-point cloud features within the candidate bounding box.

[0035] Furthermore, a point cloud network module is used for extraction.

[0036] Furthermore, an attention mechanism is employed to fuse the LiDAR point cloud ROI features and the pseudo point cloud ROI features within the candidate bounding boxes.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) This invention alleviates the problem of sparsity of LiDAR point cloud by using pseudo point cloud to enhance LiDAR point cloud, and uses an aggregation method to obtain pseudo point cloud ROI features and LiDAR point cloud ROI features for fusion and optimization of candidate boxes, thereby improving the accuracy of target detection.

[0039] (2) The present invention uses a depth completion method to convert a sparse depth map into a dense depth map, and captures the dense depth map as a dense pseudo point cloud. The dense pseudo point cloud is used to supplement the LiDAR point cloud, thereby alleviating the sparsity problem of the LiDAR point cloud.

[0040] (3) The present invention converts the image into a pseudo point cloud and includes the image information in the pseudo point cloud. The data augmentation operation of the LiDAR point cloud is directly applied to the pseudo point cloud, avoiding the problem that the 2D image and the LiDAR point cloud are difficult to be augmented synchronously during network training, thereby improving the detection accuracy.

[0041] (4) This invention alleviates the uncertainty problem of depth completion by extracting the structural relationship between pseudo point cloud and neighboring LiDAR point cloud. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0043] Figure 2 This is a network structure diagram of the point cloud network module of the present invention;

[0044] Figure 3 This is a network structure diagram of the pseudo-point cloud and LiDAR point cloud homomorphic and heteromorphic feature extraction module of the present invention. Detailed Implementation

[0045] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0046] This embodiment provides a 3D target detection method based on virtual point cloud enhancement, such as... Figure 1 As shown, the method includes the following steps:

[0047] S1. Acquire 2D images and LiDAR point clouds.

[0048] This embodiment focuses on 3D target detection in autonomous vehicles. An RGB camera and a LiDAR sensor are mounted on the vehicle, and their intrinsic and extrinsic parameters are calibrated. The calibrated RGB camera and LiDAR sensor are then used to acquire 2D images. rgb And real 3D point cloud X LiDAR A dataset was created for training a 3D object detection model, containing 2D images I. rgb The resolution is 352×1216, and the LiDAR has 64 lines.

[0049] S2. Project the LiDAR point cloud onto a 2D plane to obtain a sparse depth map.

[0050] Using the extrinsic parameter matrix Tr_velo_to_cam from the LiDAR sensor to the RGB camera, the LiDAR point cloud X LiDAR Project it onto the camera coordinate system, and then use the projection matrix P.rect X LiDAR Projecting the image onto the corresponding RGB camera image yields a sparse depth map D. sparse This enables the association and matching of LiDAR point clouds and RGB images, where D... sparse The calculation formula is:

[0051] D sparse =P rect ×Tr_velo_to_cam×X LiDAR ,

[0052] Sparse depth map D sparse The resolution is 352×1216.

[0053] S3. Based on the depth completion method and with the assistance of a 2D image, the sparse depth map is converted into a dense depth map.

[0054] Input 2D image I rgb and the corresponding sparse depth image D sparse Use depth completion method to process I rgb and D sparse This yields a dense depth image D. dense D dense The calculation formula is D dense =Model(I rgb D sparse The Model is a depth completion model, and the dense depth image D is... dense The resolution is 352×1216.

[0055] S4. Convert the dense depth map into a dense pseudo-point cloud.

[0056] Using projection matrix P rect inverse matrix Dense depth image D dense For each pixel, project it onto the camera coordinate system, and then use the inverse matrix Tr_velo_to_cam of the extrinsic parameter matrix Tr_velo_to_cam from the LiDAR sensor to the RGB camera. -1 By projecting the pseudo-point cloud in the camera coordinate system onto the coordinate system of the LiDAR sensor, a dense pseudo-point cloud X is obtained. Pseudo In the LiDAR coordinate system, 3D coordinates {x,y,z}, where X Pseudo The calculation formula is: Pseudo-point cloud X PseudoEach point in the image contains information in the range {x,y,z,r,g,b,u,v,d}, where {x,y,z} are the 3D coordinates of the pseudo-point cloud in the LiDAR coordinate system, and {r,g,b} are the corresponding RGB images of the pseudo-point cloud. rgb The RGB information of the pixels in the middle, {u,v} is the RGB image I corresponding to the pseudo point cloud. rgb The coordinate information of the pixels in the middle, {d} is the dense depth image D corresponding to the pseudo point cloud. dense Depth information of the middle pixels.

[0057] S5. Use the dense pseudo-point cloud to supplement the LiDAR point cloud, and generate candidate boxes based on the supplemented LiDAR point cloud.

[0058] Use pseudo-point clouds to supplement LiDAR point clouds, and X... LiDAR First, voxelization is performed, then sparse convolution (spconv) is used to extract voxel features. Next, the voxel features are stacked along the Z-axis to convert them into a dense BEV feature map. Then, a 2D network is used to process the BEV feature map to generate candidate boxes (Regions of Interest, ROIs). In this embodiment, the first-stage pipeline of the two-stage object detection network Voxel R-CNN is used for further explanation:

[0059] Since the number of LiDAR point clouds on distant, occluded target surfaces is relatively small, a first-stage candidate bounding box is generated by supplementing the LiDAR point cloud with a portion of pseudo-point clouds. The dense pseudo-point cloud X... Pseudo The corresponding dense depth image D dense Convert to top view to obtain pseudo point cloud top view T Pseudo (T Pseudo The resolution is 1600×1216), and the LiDAR point cloud X LiDAR The corresponding sparse depth map D sparse Convert to top-view to obtain LiDAR point cloud top view T Lidar (T Lidar The resolution is 1600×1216), then T Pseudo and T Lidar The pseudo-point cloud top view subgraph is obtained by dividing the graph into different subgraphs. LiDAR point cloud top view sub-map i = 1,...,4. Subgraph ( (The resolution is 400×1216) and ( Taking a resolution of 400×1216 as an example, the sub-image and Divide into identical sub-grids (50×202 sub-grids), if If there are pixels with non-zero values ​​within a subgrid, then according to The subgrid coordinates are used to calculate the subgraph in reverse. The local origin coordinates of the neutron grid are finally determined based on the subgraph. The coordinates in the image are used to select the corresponding pseudo-point cloud to supplement the LiDAR point cloud. (Sub-image) The subgrid in the subgraph randomly retains 4 points, and the subgraph The subgrid in the subgraph randomly retains 15 points, and the subgraph The subgrid in the subgraph randomly retains 30 points, and the subgraph... All points in the sub-grid are retained. Based on the coordinates and corresponding depth information of all previously retained points, the dense depth image D is queried. dense The corresponding pixels in the image are extracted and converted into a pseudo-point cloud to supplement the LiDAR point cloud, generating a first-stage candidate box (using the 3D coordinates {x,y,z} of the pseudo-point cloud, with reflectance values ​​set to 0). This is because the dense depth image D... dense Convert to top-view to get T Pseudo The resolution changed from 352×1216 to 1600×1216, therefore T Pseudo Not all pixels in T necessarily contain information; if the retained points are in T... Pseudo If a pixel has a value of 0, it is discarded.

[0060] S6. In parallel, determine whether each point in the dense pseudo-point cloud is within the candidate box. If yes, obtain the pseudo-point cloud within the candidate box; otherwise, do not perform any operation.

[0061] Use GPU parallel acceleration to determine pseudo-point cloud X pseudo To determine whether each point in the graph is within a candidate Region of Interest (ROI), we obtain the pseudo point cloud X within the ROI. pseudo_in_ROIs .

[0062] Determine if X is a pseudo-point cloud pseudo To determine whether each point in the algorithm is within a candidate Region of Interest (ROI), the following steps are performed:

[0063] Input: Pseudo-point cloud X pseudo Candidate Regions of Interest (ROIs)

[0064] Output: Pseudo-point cloud X within the candidate ROIs pseudo_in_ROIs

[0065] for x pseudo in pseudo-point cloud X pseud o

[0066] for roi in candidate regions ROIs

[0067] Calculate pseudo point cloud x pseudo The absolute value of the distance Δz between the z-axis coordinate of the roi and the z-axis coordinate of the center point of the roi.

[0068] if Δz <= (roi height / 2)

[0069] Pseudo-point cloud x pseudo Convert global coordinates (coordinates in the LiDAR sensor coordinate system) to local coordinates within the ROI

[0070] if local coordinates x-axis coordinate <= (roi length / 2) and local coordinate The y-axis coordinate is less than or equal to (roi width / 2).

[0071] Then x pseudo In candidate regions of interest (ROIs), and save x pseudo To pseudo-point cloud X pseudo_in_ROIs

[0072] S7. Parallel query the nearest LiDAR points of each point in the pseudo point cloud within the candidate box to obtain the nearest LiDAR point cloud.

[0073] Using the pseudo point cloud X within the candidate box Pseudo_in_ROIs Using each point in the sphere as its center, a sphere query with radius r is used to query the nearest neighbor LiDAR point clouds within the sphere, which are then used as the pseudo-point clouds. This step uses the GPU to query the LiDAR point cloud as the nearest neighbor LiDAR point cloud for the pseudo point cloud. The algorithm steps are as follows:

[0074] Input: Pseudo-point cloud X Pseudo_in_ROIs LiDAR point cloud X LiDAR

[0075] Output: Pseudo-point cloud X pseudo_in_ROIs Nearest neighbor real LiDAR point cloud

[0076] for x pseudo in pseudo-point cloud X pseudo

[0077] for x LiDAR in LiDAR point cloud X LiDAR

[0078] Calculate x pseudo and x LiDAR Euclidean distance Δd

[0079] if Δd <= r

[0080] Keep x LiDARAs a pseudo-point cloud x pseudo The nearest neighbor, and save x LiDAR To pseudo-point cloud

[0081] S8. Query neighboring pseudo-point clouds based on the pseudo-point clouds in the candidate box, and obtain the pseudo-point cloud features in the candidate box based on the pseudo-point clouds in the candidate box, the neighboring pseudo-point clouds, and the nearest LiDAR.

[0082] Using pseudo-point cloud X Pseudo_in_ROIs The image coordinate information contained therein is used to query the neighboring pseudo-point clouds of each pseudo-point cloud. The PointNet module is used to extract homomorphic features between each pseudo-point cloud and its neighboring pseudo-point clouds, and heteromorphic features between each pseudo-point cloud and its neighboring LiDAR points. These homomorphic and heteromorphic features are then fused together as the features of each pseudo-point cloud. The network structure of the PointNet module used in this embodiment is as follows: Figure 2 As shown, the network structure consists of two one-dimensional convolutional layers (Conv1d), two one-dimensional batch normalization layers (BatchNorm1d), and a ReLU activation function.

[0083] Further explanation of this step:

[0084] By calculating the Manhattan distance between the coordinates {u,v} of the pixels in the corresponding RGB image for each pseudo-point cloud, pseudo-point clouds with a Manhattan distance less than a threshold θ are retained as the nearest neighbor pseudo-points of each pseudo-point cloud. In the process of extracting homomorphic and heteromorphic features from pseudo-point clouds and LiDAR point clouds, N represents the number of pseudo-point clouds. Each pseudo-point cloud queries its nine nearest neighbor pseudo-point clouds to extract homomorphic features between the pseudo-point cloud and its nearest neighbor pseudo-point clouds. For example... Figure 3 The diagram shows the network structure used by the PointNet module to extract homomorphic and heteromorphic features between pseudo-point clouds and LiDAR point clouds. Figure 3 The dimension of the pseudo-point cloud features is [N,6], containing {x,y,z,r,g,b} information of N pseudo-point clouds. Figure 3 The dimension of the residual between the pseudo point cloud and the nearest pseudo point cloud is [N, n_p, 6]. The {x, y, z, u, v, d} of each pseudo point cloud is subtracted from the {x, y, z, u, v, d} of the 9 nearest pseudo point clouds. Figure 3 The dimension of the residual between the pseudo point cloud and the nearest LiDAR point cloud is [N, n_l, 6]. The {x, y, z} of each pseudo point cloud is subtracted from the {x, y, z} of the 9 nearest LiDAR point clouds. Figure 3In this context, `in` and `out` represent the input and output channels of the PointNet module, respectively. `[N, n_p, out_channels]` (or `[N, n_l, out_channels]`) indicates that the output dimension of the PointNet module is `[number of pseudo-point clouds, number of nearest neighbor point clouds, out_channels]`. Since some PointNet modules do not need to consider nearest neighbor point clouds, their output dimension is `[N, out_channels]`.

[0085] S9. Query the voxel features of the nearest LiDAR point cloud and obtain the LiDAR point cloud ROI features by aggregation.

[0086] Within each candidate Regions of Interest (ROIs) generated in the first stage, n×n×n grid points are generated. Each grid point queries its nearest LiDAR voxel features using a voxel query, and the PointNet module aggregates the nearest LiDAR voxel features as the ROI features f of the LiDAR point cloud. i ROI_LiDAR .

[0087] S10. Based on the pseudo-point cloud features within the candidate box, obtain the pseudo-point cloud ROI features within the candidate box using an aggregation method.

[0088] In the candidate ROIs generated in the first stage, n×n×n sub-voxels are generated within each ROI. The pseudo-point cloud features within each sub-voxel are aggregated using a max-pooling strategy to obtain the features of each sub-voxel, and thus the ROI features f of the pseudo-point cloud within the candidate bounding box are obtained. i ROI_Pseudo .

[0089] S11. The candidate box is optimized by fusing the LiDAR point cloud ROI features and the pseudo point cloud ROI features within the candidate box.

[0090] Use an attention mechanism to fuse the ROI features of LiDAR point clouds one ROI at a time. (n is the number of ROIs) and the ROI features of the pseudo-point cloud. (where n is the number of ROIs), and the fused features F are obtained. fusion This is used to optimize the candidate ROIs generated in the first stage to obtain 3D object detection boxes.

[0091] Furthermore, this embodiment further illustrates the above method: First, 2D images and LiDAR point clouds are acquired, the LiDAR point clouds are projected onto a sparse depth map, and adjacent image pairs are used to train the unsupervised depth completion model KBNet. After the depth completion model is trained, the model parameters remain unchanged, and in subsequent 3D object detection tasks, it is directly used to convert the sparse depth map into a dense depth image. The 3D object detection model is implemented using the open-mmlab open-source Openpcdet framework. The configuration file for the KITTI dataset in the Openpcdet framework is used. The model is trained on the entire KITTI 3D object detection dataset for 100 epochs with a batch size of 4.

[0092] This embodiment tests the 3D object detection performance of the proposed method on the Car category of the val set of the publicly available KITTI 3D object detection dataset. Table 1 below compares the performance of the proposed method with the Voxel-RCNN detection model. Voxel-RCNN is a classic two-stage 3D object detection network model, and it is used as the baseline for performance comparison of the proposed method. Voxel-RCNN+Homomorphic Features uses Voxel-RCNN and pseudo-point clouds; when optimizing the first-stage candidate boxes, only the homomorphic features of the pseudo-point clouds (features of each pseudo-point cloud and its nearest neighbor) are considered. The proposed method uses Voxel-RCNN and pseudo-point clouds; when optimizing the first-stage candidate boxes, it considers not only the homomorphic features of the pseudo-point clouds (features of each pseudo-point cloud and its nearest neighbor) but also the heteromorphic features of the pseudo-point clouds and LiDAR point clouds (features of each pseudo-point cloud and its nearest LiDAR point clouds).

[0093] Table 1. Performance comparison (AP%) of the proposed method and the Voxel-RCNN detection model in this embodiment.

[0094]

[0095] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0097] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0101] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A 3D target detection method based on virtual point cloud enhancement, characterized in that, Includes the following steps: Acquire 2D images and LiDAR point clouds; The LiDAR point cloud is projected onto a 2D plane to obtain a sparse depth map; Based on the depth completion method, and with the assistance of the 2D image, the sparse depth map is converted into a dense depth map; Convert the dense depth map into a dense pseudo-point cloud; The dense pseudo-point cloud is used to supplement the LiDAR point cloud, and candidate boxes are generated based on the supplemented LiDAR point cloud. In parallel, determine whether each point in the dense pseudo-point cloud is within the candidate box. If it is, obtain the pseudo-point cloud within the candidate box; otherwise, do not perform any operation. Parallel queries are performed on the nearest LiDAR points of each point in the pseudo-point cloud within the candidate box to obtain the nearest LiDAR point cloud. The pseudo point cloud in the candidate box is queried by querying the neighboring pseudo point cloud, and the pseudo point cloud features in the candidate box are obtained based on the pseudo point cloud in the candidate box, the neighboring pseudo point cloud, and the nearest LiDAR point cloud. Query the voxel features of the nearest LiDAR point cloud and obtain the ROI features of the LiDAR point cloud by aggregation. Based on the pseudo-point cloud features within the candidate box, the pseudo-point cloud ROI features within the candidate box are obtained by aggregation. The LiDAR point cloud ROI features and the pseudo point cloud ROI features within the candidate boxes are fused together, and the candidate boxes are optimized to obtain 3D target detection boxes. The specific steps for supplementing the LiDAR point cloud include: Convert the dense depth map corresponding to the dense pseudo-point cloud into a pseudo-point cloud top view; Convert the sparse depth map corresponding to the LiDAR point cloud into a top view of the LiDAR point cloud. The pseudo point cloud top view and the LiDAR point cloud top view are respectively divided into different sub-images, namely the pseudo point cloud top view sub-image and the LiDAR point cloud top view sub-image; The pseudo point cloud top view sub-image and the LiDAR point cloud top view sub-image are divided into the same sub-grid; Search for pixels with non-zero values ​​in the sub-grid of the LiDAR point cloud top view sub-image, and calculate the local origin coordinates of the sub-grid in the pseudo-point cloud top view sub-image based on the coordinates in the LiDAR point cloud top view sub-image. Based on the local origin coordinates of the subgrid in the top view sub-image of the pseudo-point cloud, a corresponding dense pseudo-point cloud is selected to supplement the LiDAR point cloud.

2. The 3D target detection method based on virtual point cloud enhancement according to claim 1, characterized in that, The 2D images and LiDAR point clouds were acquired using an RGB camera and a LiDAR sensor, respectively.

3. The 3D target detection method based on virtual point cloud enhancement according to claim 2, characterized in that, It also includes calibrating the intrinsic and extrinsic parameters of the RGB camera and LiDAR sensor before acquiring the 2D image and LiDAR point cloud.

4. The 3D target detection method based on virtual point cloud enhancement according to claim 3, characterized in that, The dense depth map is converted into a dense pseudo-point cloud using extrinsic parameters from an RGB camera and a LiDAR sensor.

5. The 3D target detection method based on virtual point cloud enhancement according to claim 1, characterized in that, Candidate boxes are generated using a one-stage process of a two-stage 3D object detection model.

6. The 3D target detection method based on virtual point cloud enhancement according to claim 1, characterized in that, Using each point in the pseudo-point cloud within the candidate box as the center of a sphere, within a preset sphere radius, the LiDAR point cloud within the sphere is taken as the nearest neighbor LiDAR point cloud of the pseudo-point cloud within the candidate box.

7. The 3D target detection method based on virtual point cloud enhancement according to claim 1, characterized in that, The specific steps for obtaining pseudo-point cloud features within the candidate bounding box include: Extract the homomorphic structure features of the pseudo-point cloud within the candidate box and the neighboring pseudo-point clouds; Extract the heteromorphic structural features between the pseudo point cloud and the nearest LiDAR point cloud within the candidate box; The homomorphic and heteromorphic structural features are fused together as pseudo-point cloud features within the candidate bounding box.

8. The 3D target detection method based on virtual point cloud enhancement according to claim 7, characterized in that, Point cloud network module is used for extraction.

9. The 3D target detection method based on virtual point cloud enhancement according to claim 1, characterized in that, An attention mechanism is used to fuse the LiDAR point cloud ROI features and the pseudo point cloud ROI features within the candidate bounding boxes.

Images