Improved multimodal 3D target detection method and system based on virtual point cloud

Abstract

The present invention discloses an improved multimodal three-dimensional target detection method based on virtual point clouds, comprising: performing depth completion based on input laser radar point cloud data and image data to generate a virtual point cloud; performing feature extraction on the laser radar point cloud and the virtual point cloud respectively to obtain real point cloud voxel features and virtual point cloud voxel features; weighting the obtained virtual point cloud voxel features with centroid offsets, and fusing the real point cloud voxel features and the weighted virtual point cloud voxel features with an attention fusion mechanism; constructing a density-aware grid proposal pool based on the fused features to perform feature aggregation; and performing target detection based on the aggregated features. The present invention suppresses interference caused by virtual noise points through centroid offset weighting, improves perception of smaller and farther objects through a multi-scale attention fusion mechanism, and reduces feature loss caused by ignoring point density information during network processing by constructing a density-aware grid proposal pool.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an improved multimodal three-dimensional target detection method and system based on virtual point cloud. Background Art

[0002] Three-dimensional object detection is a core technology in fields such as autonomous driving and robotic perception. Its core task is to accurately identify and localize three-dimensional objects from multimodal sensor data (such as lidar point clouds and camera images). Currently, methods based on cross-modal fusion of point clouds and cameras have attracted widespread attention. However, due to the fundamental differences in the representation, distribution characteristics, and information density of the two data types, efficient feature alignment and fusion remain significant challenges. LiDAR point clouds consist of points distributed unevenly in three-dimensional space, providing precise geometric information but lacking texture detail. The point cloud density decreases exponentially with increasing detection range (typically, the point density at 50 meters is only 5%-10% of that at near-field locations), resulting in a significant loss of geometric structure information for distant objects. While camera images provide rich texture and semantic information, they are limited by two-dimensional projections and suffer from depth blurring and occlusion. Existing methods typically generate virtual point clouds through depth completion (such as PENet and CaDDN) to achieve spatial alignment with the lidar data. However, completion accuracy is significantly affected by factors such as illumination and occlusion (depth errors can reach over 20% at long distances).

[0003] Among the current mainstream methods, voxel-based methods (such as VoxelNet and SECOND) quantize point clouds through regular voxel grids, but have the following defects: (1) Loss of details: fixed-size voxels are difficult to adapt to the distance changes in point cloud density, and small objects at a distance (such as pedestrians and bicycles) are easily filtered by downsampling during the voxelization process; (2) Computational redundancy: although strategies such as farthest point sampling (FPS) can retain key points, they increase computational complexity (such as the O(n²) time complexity of FPS) and limit the number of candidate points in the second stage proposal refinement; (3) Ignoring point density: relying only on the quantization method of the point cloud, details are easily lost, resulting in reduced performance.

[0004] Methods based on pseudo point clouds (such as PointPainting and MVP) project image features into point cloud space, but they face the problems of noise sensitivity and modality misalignment: (1) The virtual point cloud introduced by depth completion has positioning errors (such as artifact points at the edge of the image), and direct fusion will pollute the geometric authenticity of the lidar; (2) The difference in the original feature distribution between the point cloud and the image (such as the sparsity of the point cloud and the density of the image) leads to poor results of simple feature concatenation or summation fusion.

[0005] The existing technology has the following problems:

[0006] The first type is due to the problem of noise error introduced when depth completion generates virtual point clouds:

[0007] The feature distributions of point clouds and images differ significantly. Point cloud data consists of irregularly distributed points in three-dimensional space, while image data consists of regularly distributed pixels in two-dimensional space. To achieve data alignment between image and point cloud information, this study used a deep completion network (PENet) to generate dense virtual point clouds for images. However, because features from virtual points are affected by the performance of deep completion, relying solely on sparse convolution in the merged point cloud space for feature extraction would affect fusion performance. Furthermore, the noise introduced by deep completion can also have a significant impact.

[0008] The second category is due to the problem of low accuracy in detecting and recognizing targets at a distance:

[0009] 1. Directly concatenating the features of real point voxels and virtual point voxels will lead to insufficient fusion, and the fusion weights during the initial integration will have a great impact on the fusion quality. Therefore, the information from the lidar and camera cannot be fully utilized, which leads to poor detection results in sparse areas of the fused point cloud.

[0010] 2. When LiDAR collects data, the points and the radar naturally diverge as the angular offset between the lasers and the distance increase. Because the point density of LiDAR data varies significantly with distance, detailed information about distant points can be easily lost during voxelization, resulting in a decrease in density. This density information is not factored into the network features, and as a result, information about distant and smaller objects is easily lost. Summary of the Invention

[0011] To overcome the shortcomings of the above-mentioned existing technologies, the present invention provides an improved multimodal three-dimensional target detection method based on virtual point clouds. It suppresses the interference caused by virtual noise points through weighted centroid offset, improves the perception of farther and smaller objects through a multi-scale attention fusion mechanism, and reduces feature loss caused by ignoring point density information during network processing by constructing a density-aware grid proposal pool.

[0012] According to one aspect of the present invention, a multimodal three-dimensional target detection method based on virtual point cloud improvement is provided, comprising:

[0013] Perform depth completion based on the input lidar point cloud data and image data to generate a virtual point cloud;

[0014] Feature extraction is performed on the lidar point cloud and the virtual point cloud respectively to obtain the real point cloud voxel features and the virtual point cloud voxel features;

[0015] The obtained virtual point cloud voxel features are weighted by centroid offset, and the real point cloud voxel features and the weighted virtual point cloud voxel features are fused using the attention fusion mechanism;

[0016] Based on the fused features, a density-aware grid proposal pool is constructed for feature aggregation;

[0017] Based on the aggregated features, target detection is performed.

[0018] As a further technical solution, the obtained virtual point cloud voxel features are weighted by centroid offset, including:

[0019] The centroid offset of the virtual point cloud voxel features is weighted according to the centroid offset of the body.

[0020] As a further technical solution, the attention fusion mechanism is used to fuse the real point cloud voxel features and the weighted virtual point cloud voxel features, including:

[0021] Feature fusion is performed through two multi-scale attention mechanisms, as shown below:

[0022] ,

[0023] in, represents the attention weights generated by the multi-scale channel attention mechanism, Represents the real point cloud voxel features, Represents the weighted virtual point cloud voxel features.

[0024] As a further technical solution, based on the fused features, a density-aware grid proposal pool is constructed for feature aggregation, including:

[0025] Combined with the kernel density estimation method, after encoding the point density features, the voxel features adjacent to the grid points are aggregated to achieve enhanced pooling and obtain the aggregated features.

[0026] As a further technical solution, the method further includes:

[0027] Each generated proposal is divided into U × U × U Uniform grid points , the aggregated grid point density characteristics are:

[0028] ,

[0029] represents the fused voxel features, Represents grid points The set of voxels in a sphere with radius R, represents the actual number of points in each voxel, represents the number of virtual points in each voxel, Represents the probability density calculated by the kernel density estimation method.

[0030] As a further technical solution, after obtaining the aggregated features, the method further includes:

[0031] Take a sphere query of different radius to get each grid point Features , and obtain grid point features of different scales, connect the obtained features together, and generate the final features of each grid point based on the additional features of different voxel layers.

[0032] As a further technical solution, feature extraction is performed on the lidar point cloud and the virtual point cloud respectively to obtain the real point cloud voxel features and the virtual point cloud voxel features, including:

[0033] After voxelizing the LiDAR point cloud data, sparse convolution and feature extraction are performed to generate real point cloud voxel features;

[0034] After depth completion, the image data is subjected to sparse convolution and feature extraction to generate virtual point cloud voxel features.

[0035] According to one aspect of the present invention, there is provided a multimodal three-dimensional object detection system based on virtual point cloud improvement, comprising:

[0036] The first main module is used to perform depth completion based on the input lidar point cloud data and image data to generate a virtual point cloud;

[0037] The second main module is used to extract features from the lidar point cloud and the virtual point cloud respectively to obtain the real point cloud voxel features and the virtual point cloud voxel features;

[0038] The third main module is used to perform centroid offset weighting on the obtained virtual point cloud voxel features, and use the attention fusion mechanism to perform feature fusion on the real point cloud voxel features and the weighted virtual point cloud voxel features;

[0039] The fourth main module is used to build a density-aware grid proposal pool based on the fused features for feature aggregation;

[0040] The fifth main module is used to perform target detection based on the aggregated features.

[0041] According to one aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the improved multimodal three-dimensional target detection method based on virtual point cloud.

[0042] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the improved multimodal three-dimensional target detection method based on virtual point cloud.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) To address the problem of noise error introduced when depth completion generates virtual point clouds in the first category, this paper proposes a virtual point cloud enhancement block, which extracts features from the original point cloud, the original points, and the virtual points generated from the image through sparse convolution, and reduces the noise interference of virtual points through a centroid offset weighting module and performs feature fusion through a self-attention fusion mechanism.

[0045] (2) The present invention addresses the problem of low accuracy in detecting and recognizing distant targets in the second category of problems. It adopts a feature attention fusion structure to fuse features from two modalities, fully utilizes information from lidar and camera, improves the perception of distant smaller objects, and avoids noise interference caused by inaccurate virtual points. In addition, the density information of the point cloud is obtained in the perception grid pooling stage and attached to the features, thereby achieving better recognition of distant smaller targets. Combined with the kernel density estimation method, the point density features are encoded and the voxel features adjacent to the grid points are aggregated to achieve enhanced pooling.

[0046] (3) The present invention was tested on the KITTI dataset, which is currently the most widely used test dataset in the field of autonomous driving and contains 7481 training samples and 7518 test samples. The pedestrian detection results of the 3D object detection algorithm proposed in the present invention were improved by 6.93% compared with the baseline network PVRCNN. The improvement was particularly significant for samples with a large degree of occlusion in the dataset. The experimental results show that the method proposed in the present invention has achieved a significant improvement in the detection accuracy of smaller objects, which is of great value for applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A flowchart of an improved multimodal three-dimensional target detection method based on virtual point cloud provided by an embodiment of the present invention.

[0049] Figure 2Schematic diagram of the framework of the improved multimodal 3D target detection method based on virtual point cloud provided by an embodiment of the present invention.

[0050] Figure 3 A schematic diagram of virtual point cloud enhancement provided by an embodiment of the present invention.

[0051] Figure 4 Schematic diagram of the feature attention module provided in an embodiment of the present invention.

[0052] Figure 5 Schematic diagram of the MS-CAM module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The terms "including" and "having" and any variations thereof in the description and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatus.

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0055] See also Figure 1 An embodiment of the present invention provides an improved multimodal three-dimensional target detection method based on virtual point cloud. First, depth completion is performed based on the input lidar point cloud data and image data to generate a virtual point cloud. Then, feature extraction is performed on the lidar point cloud and the virtual point cloud respectively to obtain real point cloud voxel features and virtual point cloud voxel features. Then, the obtained virtual point cloud voxel features are weighted by centroid offset, and the real point cloud voxel features and the weighted virtual point cloud voxel features are fused using the attention fusion mechanism. Subsequently, based on the fused features, a density-aware grid proposal pool is constructed for feature aggregation. Finally, based on the aggregated features, target detection is performed.

[0056] See also Figure 2 , which presents an algorithmic framework for an improved multimodal 3D target detection method based on virtual point cloud. The framework includes a 3D backbone network for depth completion and feature extraction of input lidar point cloud data and image data, including a virtual point cloud enhancement module; a 2D backbone network and a region proposal network (RPN) for bird's-eye view (BEV) conversion and candidate box generation; voxel grid pooling for feature aggregation based on density-aware grid proposal pool; and a detection head for target detection.

[0057] based on Figure 2 The proposed algorithm framework inputs lidar point cloud data and RGB image data when performing three-dimensional target detection, generates a pseudo cloud (i.e., virtual point cloud) through depth completion, extracts features from the image and lidar point cloud through the virtual point cloud enhancement block, uses the attention fusion mechanism to fuse the virtual point cloud voxel features with the real point cloud voxel features, combines point density-based RoI grid pooling for feature aggregation, and finally generates detection results through the detection head.

[0058] In order to achieve data alignment of image and point cloud information, the embodiment of the present invention first generates a dense virtual point cloud of the image based on the depth completion network. Since the features from the virtual point cloud are affected by the depth completion performance, directly relying entirely on the sparse convolution in the merged point cloud space for feature extraction will affect the fusion performance. In addition, the noise introduced by the depth completion will also have a significant impact. Therefore, the method of the present invention proposes a virtual point cloud enhancement module, namely the VE module. The specific structure of the VE module is as follows: Figure 2 As shown in the figure, features are extracted from the original point cloud and the virtual point cloud generated from the image by sparse convolution, and feature fusion is performed by the proposed centroid offset weighting to reduce the virtual point noise interference and self-attention fusion mechanism.

[0059] Regarding the weighting of the center of mass offset, the details are as follows:

[0060] In order to fully perceive the virtual point cloud voxel features while reducing the errors caused by erroneous information, an embodiment of the present invention weights the virtual point cloud voxel features according to the body centroid offset, so that the virtual point cloud voxel features that are closer to the point cloud distribution in the original voxel, that is, with higher confidence, have a higher weight, while the weight of the virtual point cloud voxel features with excessively large centroid differences is reduced.

[0061] The embodiment of the present invention first calculates the body centroid. When there are only virtual point clouds or only real point clouds in the voxel, the voxel features are directly retained. When all voxels are non-empty voxels, the following formula can be used for calculation.

[0062] No. The sequence of non-empty voxels in a layer can be expressed as:

[0063]

[0064] in Representing voxels The 3D voxel index of represents the aggregated voxel feature vector, It is The number of non-empty voxels in the layer. According to the 3D voxel index, the point cloud sequence in the same voxel can be represented as , the calculation formula of physical fitness center is as follows (2):

[0065]

[0066] in Represents the voxel midpoint The spatial coordinates of the virtual point cloud and the real point cloud are calculated separately and expressed as and , weight The calculation formula is as follows (3):

[0067]

[0068] in Represents the distance between the virtual point cloud voxel and the real point cloud voxel centroid, Representative The diagonal length of the voxels in the layer. When the distance is close to 0, the weight is close to 1. When the distance is close to the diagonal length, it indicates that the offset is too large, so the weight is close to 0. After strengthening the virtual point voxel features with the weights, the virtual point cloud voxel features can be obtained.

[0069] See also Figure 3 , a feature attention fusion structure is used to fuse features from the two modalities, making full use of information from the lidar and camera to improve the perception of smaller objects farther away, while avoiding noise interference caused by inaccurate virtual points. Among them, the real point cloud voxel features and the virtual point cloud voxel features weighted by the centroid offset can be expressed as and , It represents the virtual point cloud features obtained from the image directly through the backbone network.

[0070] In order to avoid the great influence on the fusion quality caused by the initial integration fusion weight, the embodiment of the present invention does not use the direct cascade or summation method to set the initial aggregation feature, but uses the following method: Figure 4 The structure shown in the figure performs feature fusion through two multi-scale attention mechanisms, which can be expressed by Formula 4:

[0071]

[0072] in Represents the attention weight generated by the multi-scale channel attention mechanism (MS-CAM), the specific structure is as follows Figure 5 As shown in Figure 2. The global channel attention mechanism has a clearer distribution for large objects, but tends to ignore the features of smaller objects. However, small targets are often the bottleneck of network performance detection. Therefore, MS-CAM is used to aggregate multi-scale context features to alleviate the problems caused by scale changes and small objects. The input features are , the calculation formula is as follows:

[0073]

[0074] in The local details can be preserved, while Preserve global context information. Represents the Sigmoid function. It can be calculated by the following formula 6:

[0075]

[0076] in represents batch normalization, Represents a linear rectified unit.

[0077] It should be noted that LiDAR point cloud data exhibits significant distance-dependent density decay: as the detection distance increases, the spatial distribution density of sampling points per unit solid angle decreases exponentially. This nonlinear density variation leads to two key problems in conventional voxelization: (1) the spatial resolution of point clouds in distant areas decreases dramatically, resulting in a severe loss of geometric details; and (2) fixed-size voxel grids struggle to adapt to dynamically changing point spacing, resulting in incomplete feature representation of small-scale targets (such as pedestrians and traffic signs). To address this multi-scale representation challenge, embodiments of the present invention propose a density-aware feature enhancement method.

[0078] After obtaining the fused real point cloud and virtual point cloud features, a density-aware grid proposal pool is constructed. Combined with the kernel density estimation method, the point density features are encoded (the encoding process is Formula 7), and the voxel features adjacent to the grid points are aggregated to achieve enhanced pooling. Each generated proposal will be divided into U × U × U Uniform grid points The aggregated grid point density characteristics are shown in the following formula (7):

[0079]

[0080] in represents the fused voxel features, Represents grid points The set of voxels in a sphere with radius R, represents the actual number of points in each voxel, represents the number of virtual points in each voxel, It represents the probability density calculated by the kernel density estimation method, and the calculation formula is as follows:

[0081]

[0082] in, is the bandwidth, The calculation formula is shown in formula (9):

[0083]

[0084] in, 、 The components of the center coordinates of the k-th voxel and the i-th voxel in the l-th layer feature in the d-dimensional (x / y / z) dimension. It is an independent kernel function kernel in the three dimensions d of xyz. After obtaining the aggregated features, a ball query with different radii is used to obtain each grid point Features , and obtain grid point features of different scales, and connect the output features together. As shown in Equation (10), the final feature of each grid point is obtained based on the additional features of different voxel layers (the additional features refer to the connected grid point features obtained by Equation (10)).

[0085]

[0086] in Indicates that from Features obtained at the voxel level.

[0087] The implementation of each embodiment of the present invention is based on programmed processing performed by a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, and in addition to the aforementioned embodiments, an embodiment of the present invention provides a multimodal 3D object detection system based on virtual point clouds. This system is used to implement the multimodal 3D object detection method based on virtual point clouds described in the aforementioned method embodiments.

[0088] The system includes: a first main module, which is used to perform depth completion based on the input lidar point cloud data and image data to generate a virtual point cloud; a second main module, which is used to extract features from the lidar point cloud and the virtual point cloud respectively to obtain real point cloud voxel features and virtual point cloud voxel features; a third main module, which is used to weight the center of mass offset of the obtained virtual point cloud voxel features, and use the attention fusion mechanism to fuse the real point cloud voxel features and the weighted virtual point cloud voxel features; a fourth main module, which is used to construct a density-aware grid proposal pool based on the fused features for feature aggregation; and a fifth main module, which is used to perform target detection based on the aggregated features.

[0089] The improved multimodal three-dimensional target detection system based on virtual point cloud provided by the embodiment of the present invention is aimed at the problems of noise errors introduced when depth completion generates virtual point clouds and low accuracy of target detection and recognition at a distance. It adopts several of the aforementioned modules, suppresses the interference caused by virtual noise points through weighted centroid offset, improves the perception of smaller objects at a distance through a multi-scale attention fusion mechanism, and reduces feature loss caused by ignoring point density information during network processing by constructing a density-aware grid proposal pool.

[0090] It should be noted that the system embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference lies in the setting of corresponding functional modules, and the principles thereof are basically the same as the principles of the above-mentioned system embodiments provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned system embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, improve the modules in the above-mentioned system embodiments to obtain corresponding system class embodiments for implementing the methods in other method class embodiments.

[0091] Based on the same inventive concept as the aforementioned embodiment, an embodiment of the present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the improved multimodal three-dimensional target detection method based on virtual point cloud.

[0092] Based on the same inventive concept as the aforementioned embodiment, an embodiment of the present invention also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the improved multimodal three-dimensional target detection method based on virtual point cloud.

[0093] In summary, in order to achieve higher-precision three-dimensional target detection based on cross-modal fusion of point clouds and cameras, the present invention adopts an efficient and novel method to realize cross-modal feature fusion of the two. The specific operations are as follows: First, the original point cloud stream voxelizes the input point cloud and performs sparse convolution and feature extraction; the pseudo point cloud stream (i.e., virtual point cloud stream) forms a pseudo point cloud through the input image through a deep completion network and performs feature extraction, wherein the pseudo point cloud features are processed by the centroid weighting module in the pseudo point cloud enhancement module. Then, after obtaining the processed pseudo point cloud and real point cloud features, a multi-scale attention fusion mechanism is used to fuse the features of the two modalities. Finally, after the features are fused, the point density perception module is used to further aggregate the features during the candidate box grid pooling, and the final prediction result is obtained through the detection head.

[0094] Compared with the prior art, the innovation of the present invention is:

[0095] (1) As there is currently no unified fixed paradigm to handle two types of data with misaligned features, an efficient and novel method is adopted to achieve cross-modal feature fusion of the two, fully perceiving the information from both point cloud and image modal data.

[0096] (2) A centroid offset weighting module is designed. Combining the centroid of the virtual point voxels, the weights are calculated to balance the virtual point voxels. This can simply and effectively suppress the interference caused by virtual point noise and fully retain the characteristics of effective information.

[0097] (3) A multi-scale attention fusion mechanism is adopted to avoid the great impact of the fusion weight during the initial integration on the fusion quality, make full use of the information from the lidar and camera, reduce the mutual interference during information fusion and the introduction of irrelevant noise, thereby improving the perception of farther and smaller objects.

[0098] (4) Combined with the kernel density estimation method, after encoding the point density features, the voxel features adjacent to the grid points are aggregated to achieve enhanced pooling, and the density information is combined to fully obtain the features of farther and smaller targets, thereby reducing the feature loss caused by ignoring the point density information during network processing.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. An improved multimodal 3D target detection method based on virtual point cloud, characterized in that: include: Perform depth completion based on the input lidar point cloud data and image data to generate a virtual point cloud; Feature extraction is performed on the lidar point cloud and the virtual point cloud respectively to obtain the real point cloud voxel features and the virtual point cloud voxel features; The obtained virtual point cloud voxel features are weighted by centroid offset, and the real point cloud voxel features and the weighted virtual point cloud voxel features are fused using the attention fusion mechanism; Based on the fused features, a density-aware grid proposal pool is constructed for feature aggregation, including: combining the kernel density estimation method to encode the point density features, aggregating the voxel features adjacent to the grid points to achieve enhanced pooling, and obtaining the aggregated features. Each generated proposal is divided into U × U × U Uniform grid points , the aggregated grid point density characteristics are: ， represents the fused voxel features, Represents grid points The set of voxels in a sphere with radius R, represents the actual number of points in each voxel, represents the number of virtual points in each voxel, represents the probability density calculated by the kernel density estimation method; Based on the aggregated features, target detection is performed.

2. The improved multimodal 3D target detection method based on virtual point cloud according to claim 1, characterized in that: The obtained virtual point cloud voxel features are weighted by centroid offset, including: The centroid offset of the virtual point cloud voxel features is weighted according to the centroid offset of the body.

3. The improved multimodal 3D target detection method based on virtual point cloud according to claim 1, characterized in that: The attention fusion mechanism is used to fuse the real point cloud voxel features and the weighted virtual point cloud voxel features, including: Feature fusion is performed through two multi-scale attention mechanisms, as shown below: ， in, represents the attention weights generated by the multi-scale channel attention mechanism, Represents the real point cloud voxel features, Represents the weighted virtual point cloud voxel features.

4. The improved multimodal 3D target detection method based on virtual point cloud according to claim 1, characterized in that: After obtaining the aggregated features, the method further includes: Take a sphere query of different radius to get each grid point Features , and obtain grid point features of different scales, connect the obtained features together, and generate the final features of each grid point based on the additional features of different voxel layers.

5. The improved multimodal 3D target detection method based on virtual point cloud according to claim 1, characterized in that: Feature extraction is performed on the lidar point cloud and virtual point cloud respectively to obtain the real point cloud voxel features and virtual point cloud voxel features, including: After voxelizing the LiDAR point cloud data, sparse convolution and feature extraction are performed to generate real point cloud voxel features; After depth completion, the image data is subjected to sparse convolution and feature extraction to generate virtual point cloud voxel features.

6. The improved multimodal 3D target detection system based on virtual point cloud is characterized by: include: The first main module is used to perform depth completion based on the input lidar point cloud data and image data to generate a virtual point cloud; The second main module is used to extract features from the lidar point cloud and the virtual point cloud respectively to obtain the real point cloud voxel features and the virtual point cloud voxel features; The third main module is used to perform centroid offset weighting on the obtained virtual point cloud voxel features, and use the attention fusion mechanism to perform feature fusion on the real point cloud voxel features and the weighted virtual point cloud voxel features; The fourth main module is used to build a density-aware grid proposal pool based on the fused features for feature aggregation, including: combining the kernel density estimation method to encode the point density features, aggregating the voxel features adjacent to the grid points to achieve enhanced pooling, and obtaining the aggregated features. Each generated proposal is divided into U × U × U Uniform grid points , the aggregated grid point density characteristics are: ， represents the fused voxel features, Represents grid points The set of voxels in a sphere with radius R, represents the actual number of points in each voxel, represents the number of virtual points in each voxel, represents the probability density calculated by the kernel density estimation method; The fifth main module is used to perform target detection based on the aggregated features.

7. An electronic device, characterized in that: It includes a memory and a processor, the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the improved multimodal three-dimensional target detection method based on virtual point cloud as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, which enable the computer to execute the improved multimodal three-dimensional target detection method based on virtual point cloud as described in any one of claims 1 to 5.

Images