A method for 3D object detection assisted by image information
By employing a block-based self-attention mechanism and a multimodal feature fusion method, the problem of noise and feature extraction imbalance in 3D object detection is solved, achieving high-precision 3D object detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2024-05-09
- Publication Date
- 2026-05-26
AI Technical Summary
Existing 3D target detection methods suffer from problems such as significant noise impact, unbalanced modal feature extraction, insufficient noise processing, and inadequate feature fusion when utilizing point cloud and image information, resulting in insufficient detection accuracy and robustness.
A block-based self-attention mechanism is used for multimodal feature fusion to generate a foreground point mask and perform multi-scale feature extraction. Combined with a two-stage detection module, image information is used to supplement point cloud features, and feature enhancement is performed through an improved Set Abstraction structure and a multilayer perceptron.
It improves the performance and robustness of 3D object detection, reduces computational complexity and storage requirements, enhances the ability to capture details of foreground objects, and significantly improves detection accuracy.
Smart Images
Figure CN118506351B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a three-dimensional object detection method, and more particularly to an image-assisted three-dimensional object detection method. Background Technology
[0002] This section provides only background information relevant to this disclosure and is not necessarily prior art.
[0003] 3D object detection involves identifying objects of interest in a 3D scene and estimating their type and pose. It is crucial for tasks such as autonomous driving, real-time localization and mapping (RTD), and augmented reality. In recent years, with the rapid development of artificial intelligence, improved hardware computing power, and the increasingly widespread application of 3D sensors, 3D object detection has received increasing attention.
[0004] Despite the numerous object detection algorithms proposed for point clouds, these methods, which rely on a single modality as input, struggle to overcome the inherent limitations of point cloud data itself. On one hand, factors such as specular reflection and object occlusion often result in varying degrees of missing objects in point clouds, making it difficult to guarantee spatial continuity. On the other hand, the lack of texture and other features means that objects with similar shapes but different semantic categories often exhibit low discriminative power in point clouds. To address these issues, methods incorporating image information have received considerable attention. Compared to point clouds, images provide rich texture and semantic features, effectively complementing point cloud features.
[0005] However, introducing multimodal features also brings new problems. First, due to the noise inherent in the point cloud itself and the effects of filling and pooling modules during image feature extraction, the image feature extraction process based on the relationship between the point cloud and image projection often contains a lot of noise. Second, different modal networks often have different generalization and overfitting balance points for the same task, posing a challenge to balancing different modalities. Existing multimodal detection methods, on the one hand, ignore the noise in the feature extraction process, and on the other hand, mostly only use mid-term fusion methods for features or late-term fusion methods for results, failing to fully utilize the information gain brought by multimodality.
[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0007] Purpose of the invention: The technical problem to be solved by the present invention is to provide an image information-assisted three-dimensional object detection method to address the shortcomings of the existing technology.
[0008] To address the aforementioned technical problems, this invention discloses an image-assisted three-dimensional object detection method, comprising the following steps:
[0009] Step 1: Extract features from the point cloud containing the 3D object to be detected and its corresponding image.
[0010] Step 2: Based on the block self-attention mechanism, the features of the point cloud and the image are fused to obtain the fused multimodal features;
[0011] Step 3: Based on the fused multimodal features, generate a foreground point mask and use it to extract multi-scale features with a focus on the foreground, so as to obtain feature maps of the point cloud at different resolutions.
[0012] Step 4: Generate an object detection proposal. This involves interpolating feature maps at different resolutions using the three nearest neighbor interpolation method, fusing the interpolated features with multimodal features using a multilayer perceptron, and then feeding the results into a two-stage detection module to generate an object detection proposal, thus completing the image-assisted 3D object detection.
[0013] Furthermore, the feature extraction described in step 1 specifically includes:
[0014] Step 1-1: Perform preliminary feature extraction on the point cloud containing the 3D object to be detected to obtain a subset of the point cloud after downsampling. The subset of the point cloud contains the geometric features of each point.
[0015] Steps 1-2: Use a feature extraction network to generate a feature map containing semantic and texture information for the image corresponding to the point cloud;
[0016] Steps 1-3: Based on the projection relationship between the point cloud and the image determined by the camera's intrinsic and extrinsic parameters, obtain the pixel coordinates of each point in the image within the point cloud subset output in Step 1-1, and use bilinear interpolation to obtain the image features of each point on the image feature map output in Step 1-2.
[0017] Furthermore, the feature extraction network described in steps 1-2 consists of a downsampling part composed of a deep convolutional network and an upsampling part composed of three layers of transposed convolutions with residual structures, wherein:
[0018] The downsampling part, i.e. the backbone network, is any deep convolutional network trained on the ImageNet classification task;
[0019] The upsampling part is as follows: Four feature maps with progressively decreasing resolution are uniformly selected from the outputs of each part of the backbone network to form the input of the upsampling part. The upsampling part uses transposed convolution to connect every two feature maps, and performs cross-layer feature fusion by concatenating the output of the transposed convolution with the original output of the corresponding module of the downsampling network and then using a single-layer convolution processing method.
[0020] Furthermore, the feature fusion between point cloud and image based on the block self-attention mechanism described in step 2 specifically includes:
[0021] Step 2-1: The geometric features of each point obtained in Step 1-1 are concatenated with the image features of each point obtained in Step 1-3, and a single-layer convolution is used for preliminary fusion to obtain the preliminary fused features.
[0022] Step 2-2: Divide the point cloud into multiple overlapping blocks in space;
[0023] Steps 2-3: Within each block, feature aggregation is performed based on a self-attention mechanism;
[0024] Steps 2-4: For each point in the point cloud, the final feature of that point, i.e., the aggregated feature, is obtained by averaging the features obtained by aggregating the features across all the blocks to which it is divided.
[0025] Step 2-5: The preliminary fused features obtained in Step 2-1 are processed by single-layer convolution and added as residuals to the aggregated features obtained in Step 2-4, which are the fused multimodal features.
[0026] Furthermore, step 2-2, which involves spatially dividing the point cloud into multiple overlapping blocks, specifically includes:
[0027] For N points in the point cloud, M block centers are obtained by sampling the farthest point, and then K intra-block points are obtained by using nearest neighbor search for each block center;
[0028] During the block segmentation process, M×K≥4N is taken, and the maximum radius of each block is limited. The portion of the K points within a block that exceeds the preset radius does not participate in feature aggregation.
[0029] Furthermore, step 3, which involves generating a foreground view mask and using it for foreground-emphasis-based multi-scale feature extraction, specifically includes:
[0030] Step 3-1: For each point in the point cloud containing the 3D object to be detected, based on the fused multimodal features, a multilayer perceptron is used to generate a foreground probability prediction, and a foreground point segmentation mask is obtained based on a preset threshold.
[0031] Step 3-2: Using a mask-guided sampling method, generate three subsets of the point cloud at different scales with the number of points halved in succession;
[0032] Step 3-3: Extract features sequentially from the three subsets of different scales in descending order of resolution to obtain feature maps at different resolutions.
[0033] Furthermore, the mask-guided sampling method described in step 3-2 generates three subsets of the point cloud at different scales with the number of points successively halved, specifically including:
[0034] Based on the foreground point segmentation mask, the coordinates of all background points are reduced by a preset ratio and moved to the vicinity of any foreground point. Then, by performing farthest point sampling on the adjusted point cloud with three different sampling numbers, three subsets of the point cloud at different scales are obtained.
[0035] Furthermore, the feature extraction described in step 3-3 involves using the improved Set Abstraction structure to extract features from subsets at each scale sequentially.
[0036] Furthermore, the improved Set Abstraction structure described in step 3-3 specifically includes:
[0037] Step 3-3-1: Given a high-resolution source point cloud and a low-resolution target point cloud, find the neighbor point set from the source point cloud for each point in the target point cloud according to the radius of the sphere.
[0038] Step 3-3-2: For each set of neighboring points, generate an offset code based on the local offset between the neighboring points and the sampling center, add it to the original features of each point, and then send it to the multilayer perceptron for processing.
[0039] Step 3-3-3 introduces a mask-based background point feature masking method, which sets the background point features in each neighbor point set to zero based on the foreground point mask, and merges the features of each neighbor point set through max pooling to obtain the features of each point in the target point cloud.
[0040] Furthermore, step 4, generating the object detection proposal, specifically includes:
[0041] Step 4-1: Using the feature maps at three different resolutions output in Step 3-3 as feature sources, and the coordinates of the point cloud subset obtained by downsampling in Step 1-1 as targets, the features at different resolutions of each point are obtained by three nearest neighbor interpolation. These features are then concatenated with the multimodal features of each point obtained in Step 2-5 and fused using a multilayer perceptron fusion processing method to obtain the fused point cloud.
[0042] Step 4-2: Based on the voting mechanism, a set of key regions that may be the center of the object are aggregated from the fused point cloud output in Step 4-1, and initial features are generated for each key region through the improved Set Abstraction structure described in Step 3-3.
[0043] Step 4-3: Based on the mutual attention mechanism, the key regions are used as queries and the point clouds are used as keys and values to enhance the features of each key region from the object semantic level. The self-attention mechanism is applied between key regions to enhance the features of key regions from the scene semantic level. Finally, a preliminary detection result is given for each key region based on the enhanced features.
[0044] Step 4-4: Based on the preliminary detection results, use the RBG feature aggregation module to aggregate the geometric features of the specific object, thereby optimizing the preliminary detection results and obtaining the final target detection proposal.
[0045] Beneficial effects:
[0046] 1) This method introduces additional image information to supplement the features of a single point cloud modality, improving the performance and robustness of object detection. Simultaneously, it exhibits excellent adaptability to the backbone network based on image features, requiring no additional training for the detection task and imposing no requirements on its network structure. It can fully utilize a large number of existing models that have been fully trained on other tasks such as image classification.
[0047] 2) This method proposes a feature fusion approach based on a block-based self-attention mechanism. By dividing the complete point cloud into multiple blocks and applying the attention mechanism independently within each block, the noise in the modal feature extraction process is effectively reduced and the semantic information of each point is enhanced. Compared with the mode of directly applying the self-attention mechanism to the entire scene, it has faster computational efficiency and lower storage space usage. At the same time, it limits the attention range of each point, thereby greatly reducing the difficulty of network training.
[0048] 3) This method uses information gain from multimodal approaches to generate a foreground mask and improves the Set Abstraction structure based on this. This improves the network's ability to model minute details of foreground objects while allowing the feature extraction process to focus more on the foreground region, thus enhancing the algorithm's performance.
[0049] 4) This method considers feature aggregation at the target object from both semantic and geometric structural information levels, which improves the robustness of feature representation and significantly enhances detection accuracy and performance. Attached Figure Description
[0050] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.
[0051] Figure 1 This is a flowchart of image-assisted 3D object detection based on a block-based self-attention mechanism.
[0052] Figure 2 This is a schematic diagram of an image feature extraction network using the InceptionV3 network as the backbone network in one embodiment.
[0053] Figure 3 This is a schematic diagram of the foreground point mask generated based on multimodal features and the sampling results that the mask should produce.
[0054] Figure 4 This is a schematic diagram of the detection results of the image information-assisted three-dimensional object detection method proposed in this invention. Detailed Implementation
[0055] This invention proposes an image-assisted 3D object detection method based on a block-based self-attention mechanism. This method fully utilizes the information gain from multimodal processing to achieve high accuracy in both semantics and geometry. The specific technical solution is as follows: An image-assisted 3D object detection method based on a block-based self-attention mechanism first utilizes a block-based self-attention mechanism for multimodal feature fusion. This reduces noise during multimodal extraction while enhancing the semantic information of the point cloud. Next, it uses the multimodal features to generate a relatively accurate foreground point mask, which guides multi-layer feature extraction. This allows the process to focus more on foreground objects while capturing more details in the foreground region. Finally, a two-stage object detection module enhances object features at both the semantic and geometric levels, thereby achieving high-accuracy 3D object detection results. Specifically, it includes the following steps:
[0056] Step 1: Independent multimodal feature extraction: Two independent feature extraction networks are used to extract features from the input point cloud and the image, and the image features corresponding to each point are extracted from the image feature map.
[0057] Step 2: Multimodal feature fusion based on block self-attention mechanism: The input point cloud is split into multiple overlapping blocks. In each block, feature aggregation is performed point by point based on the self-attention mechanism. Finally, all blocks are merged back into the structure of the input point cloud.
[0058] Step 3: Multi-scale feature extraction with emphasis on foreground: Based on multi-modal features, a more accurate foreground point mask is generated for each point. Based on this mask, a multi-scale sampling subset of the foreground-emphasized region is obtained using a mask-guided sampling method. Finally, feature extraction is performed on each subset using an improved Set Abstraction structure.
[0059] Step 4: Two-stage object detection process: First, for the output point cloud of the target scale, three nearest neighbor interpolation and multilayer perceptron are used to extract and fuse its features at different scales. Next, based on a voting mechanism, a set of key regions that may contain foreground objects are generated on the output point cloud. Finally, a two-stage detection process is used to enhance the object features in terms of semantic information and geometric structure information, respectively, to obtain more accurate detection results.
[0060] Step 1 involves independent feature extraction from data of different modalities, laying the foundation for subsequent multimodal feature fusion. This process consists of the following steps:
[0061] Step 1-1: Use the Set Abstraction structure (Reference: Qi CR, Yi L, Su H, et al. Pointnet++: Deep hierarchical feature learning on point sets in ametric space[J]. Advances in neural information processing systems, 2017, 30.) to perform preliminary feature extraction on the input point cloud, and obtain a subset of the point cloud containing geometric features in the local space after downsampling.
[0062] Steps 1-2 involve generating feature maps for the input image using an upsampling network consisting of a downsampling backbone network and three layers of transposed convolutions with residual structures. The backbone network is any network trained on the ImageNet classification task (reference: Deng J, Dong W, Socher R, et al. Imagenet: A large-scale hierarchical imagedatabase[C] / / 2009IEEE conference on computer vision and pattern recognition. IEEE, 2009:248-255.), requiring no fine-tuning for the detection task; its parameters are not involved in gradient optimization during target detection network training. For the upsampling part, four feature maps with progressively decreasing resolution are uniformly selected from the outputs of each part of the backbone network to form the input of the upsampling network. The upsampling network uses transposed convolutions to connect every two adjacent feature maps, and performs cross-layer feature fusion by concatenating the output of the transposed convolutions with the original outputs of the corresponding modules of the downsampling network and then performing single-layer convolution processing.
[0063] Steps 1-3: Based on the projection relationship between the point cloud and the image, the pixel coordinates of each point in the point cloud subset output in step 1-1 are obtained on the image. Furthermore, bilinear interpolation is used to obtain the image features of each point on the image feature map output in step 1-2. The projection relationship between the point cloud and the image is established by the intrinsic and extrinsic parameters provided by the camera and LiDAR sensors.
[0064] Step 2 adaptively fuses the geometric and image features of the point cloud obtained in Step 1, reducing noise during multimodal feature extraction while enhancing the semantic similarity between points belonging to the same object within a local region. This process consists of the following steps:
[0065] Step 2-1: The geometric features of each point obtained in Step 1-1 are stitched together with the image features of each point obtained in Step 1-3, and a single-layer convolution is used for preliminary fusion to obtain the preliminary fused features.
[0066] Step 2-2 involves spatially dividing the point cloud into multiple overlapping blocks. For the input N points, M block centers are first obtained by sampling the farthest point. Then, for each block center, a nearest neighbor search is used to find the K points closest to that block center, thus forming a block. To ensure the block segmentation process covers the input point cloud and allows for non-empty intersections between different blocks to ensure feature flow, M×K≥4N is used in the block segmentation process. To reduce interference from outliers and limit the attention range of each point to reduce the difficulty of network training, the maximum radius of each block is further limited. Points exceeding the maximum radius within a block will not participate in feature aggregation.
[0067] Steps 2-3 involve using a multi-branch self-attention mechanism (Reference: Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30.) to perform intra-block feature aggregation. During the aggregation process, a positional encoding is generated for each point using its global coordinates in the point cloud rather than its local coordinates within the block. This guides the feature aggregation process, ensuring spatial consistency across the blocks when points are divided into multiple blocks.
[0068] Steps 2-4 involve merging all blocks to restore the original point cloud structure. During this process, for each point in the input point cloud, the average of the aggregated features from all the blocks to which it belongs is taken as the final feature of that point, thus completing the process of restoring each block to the input point cloud.
[0069] Steps 2-5 involve processing the preliminary fused features obtained in Step 2-1 using a single-layer convolution, and then adding the resulting residual to the aggregated features obtained in Step 2-4. This process reduces the risk of gradient vanishing in shallow networks. Furthermore, since the block-based mode cannot guarantee 100% coverage of the output point cloud, residual connections can prevent feature gaps caused by some points not being covered by the block-based mode.
[0070] Step 3 aims to fully utilize the information gain from multimodal extraction to guide the multi-layer feature extraction process, enabling it to focus more on the foreground region of the scene while capturing more geometric details of foreground objects. This process consists of the following steps:
[0071] Step 3-1: For each point in the point cloud, a foreground probability prediction is generated using a multilayer perceptron (Reference: Rosenblatt F. The perceptron: a probabilistic model for information storage and organization in the brain[J]. Psychological review,1958,65(6):386.) based on the fused multimodal features. The foreground point segmentation mask is obtained based on a predetermined threshold, with the threshold selection mainly focusing on the recall rate of foreground points.
[0072] Step 3-2: Using mask-guided farthest point sampling, subsets of the input point cloud are generated at different scales. The distance from a point to a point set is defined as the minimum distance from that point to all points in the set. Farthest point sampling first randomly selects one point from the input point cloud as the initial sampled point set and uses the remaining points as candidate point sets. Then, the point farthest from the sampled point set in the candidate point set is removed from the candidate point set and added to the sampled point set. This process is repeated until the number of points in the sampled point set reaches the preset sampling number. For the input point cloud, according to the foreground point segmentation mask obtained in Step 3-1, the coordinates of all background points are reduced by a preset ratio (in some specific implementations, it can be one ten-thousandth of the original) and moved to any foreground point. Under the farthest point sampling mechanism, this process will prioritize sampling on foreground points. Sampling will only continue on background points if there are no foreground points or the number of foreground points is less than the sampling number, thus preserving more details of the foreground in the sampled subset. Meanwhile, scaling and translation operations do not disrupt the spatial proportions of the point cloud, allowing the principle of covering as many areas of the scene as possible to be followed when sampling the background after selecting all foreground points.
[0073] Step 3-3 involves extracting features sequentially at each scale, from highest to lowest resolution, using an improved Set Abstraction structure. First, given a high-resolution source point cloud and a low-resolution target point cloud, each point in the target point cloud is used to find neighboring point sets from the source point cloud based on the sphere radius. Further, for each neighboring point set, an offset code is generated based on the local offset between the neighboring point and the sampling center, and this code is added to the original features of each point. This code is then fed into a multilayer perceptron for processing, thereby enhancing the module's ability to model minute details. Finally, before merging the features of each neighboring point set using max pooling to obtain the features of each point in the target point cloud, a mask-based background point feature masking is introduced. This involves setting the background point features within each neighboring point set to zero based on a foreground point mask, thus reducing their interference with the foreground features.
[0074] Step 4 interpolates the feature maps of different resolutions obtained in Step 3, then uses a multilayer perceptron to fuse the interpolated features with the multimodal features obtained in Step 2, and feeds them into the two-stage detection module to generate an object detection proposal. This process consists of the following steps:
[0075] Step 4-1: Using the three point clouds at different scales output in Step 3-3 as feature sources and the coordinates of the point cloud subsets downsampled in Step 1-1 as targets, features at different resolutions for each point are obtained through three nearest neighbor interpolation. In three nearest neighbor interpolation, given a target interpolation point, the three nearest neighbor points to the target location are first found in the source point cloud. Then, the reciprocal of the squared distance from the normalized neighbor points to the target location is used as a weight to calculate a weighted sum of the neighbor point features, thus obtaining the interpolated features of the target location. Finally, the interpolated features of the target point cloud at different scales are fused together with the multimodal features of each point obtained in Step 2-5 in parallel and then processed using a multilayer perceptron to obtain the fused point cloud.
[0076] Step 4-2: Based on the voting mechanism (Reference: Qi CR, Litany O, He K, et al. Deep houghvoting for 3d object detection in point clouds[C] / / proceedings of the IEEE / CVF International Conference on Computer Vision.2019:9277-9286.), a set of key regions that may be the center of the object are aggregated from the output point cloud of Step 4-1, and initial features are generated for each key region through the Set Abstraction structure.
[0077] Step 4-3: Based on the mutual attention mechanism, the key regions are used as queries and the point clouds are used as keys and values to enhance the features of each key region from the object semantic level. The self-attention mechanism is applied between key regions to enhance the features of key regions from the scene semantic level. Finally, a preliminary detection result is given for each key region based on the enhanced features.
[0078] Step 4-4: Based on the preliminary detection results, the RBG feature aggregation module (Reference: Wang H, Shi S, Yang Z, et al. Rbgnet: Ray-based grouping for 3d object detection[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2022:1110-1119.) is used to aggregate the geometric features of specific objects, thereby enhancing the features of key regions at the geometric structure level and optimizing the preliminary detection results to obtain the final target detection proposal.
[0079] Example:
[0080] like Figure 1 As shown, a 3D object detection method based on image information assisted by a block self-attention mechanism is presented. Specifically, it includes the following steps:
[0081] Step 1: Extract features from the point cloud containing the 3D object to be detected and its corresponding image;
[0082] Step 2: Fuse the geometric features and image features of the point cloud based on the block self-attention mechanism;
[0083] Step 3: Generate a foreground point mask and use it for multi-scale feature extraction with a focus on the foreground;
[0084] Step 4: Fuse the features from each layer and feed them into the two-stage detection module to generate target proposals.
[0085] Step 1 involves independent feature extraction from data of different modalities, laying the foundation for subsequent multimodal feature fusion. This process consists of the following steps:
[0086] Step 1-1: Use the Set Abstraction structure to perform preliminary feature extraction on the input point cloud containing the 3D object to be detected, and obtain a subset of the point cloud containing local geometric features after downsampling.
[0087] Steps 1-2 involve using a feature extraction network to generate feature maps for the image corresponding to the point cloud scene. The feature extraction network uses the InceptionV3 backbone network, fully trained on the ImageNet classification task, as the downsampling network. The outputs of each sub-part of the backbone network are taken as inputs to the upsampling network from the outputs of four different sub-modules: Conv2d_4a_3x3, Mixed_5d, Mixed_6e, and Mixed_7c. The upsampling network uses transposed convolutions to upsample between every two adjacent feature maps, and then fuses the outputs of the transposed convolutions with the original outputs of the corresponding modules of the upsampling network using a single-layer convolution. The overall structure of this image feature extraction network is as follows: Figure 2 As shown.
[0088] Steps 1-3: Based on the projection relationship between the point cloud and the image determined by the camera's intrinsic and extrinsic parameters, obtain the pixel coordinates of each point output in Step 1-1 on the image, and further use bilinear interpolation to obtain the image features of each point on the image feature map output in Step 1-2.
[0089] Step 2 fuses the geometric and image features of the point cloud obtained in Step 1, reducing multimodal extraction noise while enhancing the semantic similarity of points belonging to the same object within local regions. This process consists of the following steps:
[0090] Step 2-1: The geometric features of each point in the point cloud obtained in Step 1-1 are stitched together with the image features of each point in the point cloud obtained in Step 1-3, and a single-layer convolutional structure is used for preliminary feature fusion.
[0091] Step 2-2 involves spatially dividing the point cloud into multiple overlapping blocks. For an input point cloud consisting of N points, M block centers are first obtained by sampling the farthest point. Then, for each block center, a nearest neighbor search is used to find the K points closest to that center, forming a block. To ensure coverage of the input point cloud during the block division process, and to allow for partial intersection between different blocks to ensure feature flow, M×K≥4N is used in the block division process. To reduce interference from outliers and limit the attention range of each point to reduce the difficulty of network training, the maximum radius of each block is further limited; points exceeding the maximum radius within a block will not participate in feature aggregation.
[0092] Steps 2-3 involve using a multi-branch self-attention mechanism to aggregate intra-block features within each block. During the aggregation process, a positional encoding is generated for each point using its global coordinates in the input point cloud rather than its local intra-block coordinates to guide feature aggregation, ensuring spatial consistency when features are aggregated across different blocks.
[0093] Steps 2-4 involve merging all blocks to restore the original point cloud structure. During this process, for each point in the point cloud, the average of the aggregated features from all the blocks to which it belongs is taken as the final feature of that point.
[0094] Steps 2-5 involve processing the preliminary fused features obtained in Step 2-1 using a single-layer convolution, and then adding the resulting residual to the aggregated features obtained in Step 2-4. This process reduces the risk of gradient vanishing in shallow networks. Furthermore, since the block-based mode cannot guarantee 100% coverage of the output point cloud, residual connections can prevent feature gaps caused by some points not being covered by the block-based mode.
[0095] Step 3 utilizes the information gain from multimodal processing to guide the multi-layer feature extraction process, enabling it to focus more on the foreground region while capturing more details of the foreground object, and simultaneously obtaining feature maps of the point cloud at different resolutions. This process consists of the following steps:
[0096] Step 3-1: For each point in the point cloud, a multilayer perceptron is used to generate a foreground probability prediction based on the fused multimodal features, and a foreground point segmentation mask is obtained according to a predetermined threshold. The threshold selection is mainly based on the recall rate of foreground points.
[0097] Step 3-2 uses mask-guided farthest-point sampling to generate three subsets of the input point cloud at different scales, with the number of points halved in succession. By skillfully reducing the coordinates of background points in the input point cloud to one ten-thousandth of their original size and shifting them to the vicinity of any foreground point, the farthest-point sampling mechanism prioritizes sampling at foreground points, thus preserving more details of the foreground in the sampled subsets.
[0098] Step 3-3: Following the order of resolution from high to low, feature extraction is performed sequentially at each scale using the improved Set Abstraction structure. The improvement consists of two parts: first, an additional local offset embedding module is added to the local feature extraction part to enhance the features of each point with offsets before feeding them into the multilayer perceptron, thereby improving the module's ability to model minute detail features; second, when using max pooling for local feature fusion, the features of background points within the region are masked based on foreground point masks, thereby reducing their interference with foreground features.
[0099] Figure 3The diagram illustrates two different scenarios: the original point cloud and foreground object annotations; the point-by-point foreground mask generated based on the multimodal fusion features of the input point cloud in step 3-1; the original unguided farthest point sampling result; and the mask-guided farthest point sampling result proposed in step 3-2. In the mask, green represents foreground points and red represents background points. In the sampling results, the gray area represents the input scene used for reference, and the red area represents the actual sampling points. The diagram shows that the mask-guided sampling process proposed in this invention focuses more on the foreground region of the scene, and the sampled subset retains more detailed features of the foreground objects.
[0100] Step 4 involves interpolating the feature maps obtained in Step 3 at different resolutions. Then, a multilayer perceptron is used to fuse the interpolated features with the multimodal features obtained in Step 2. A two-stage detection structure is employed to enhance the features of the foreground object from both semantic and geometric information perspectives, thereby achieving more accurate detection results. This process consists of the following steps:
[0101] Step 4-1: Using the three point clouds of different scales output in Step 3-3 as feature sources, and the coordinates of the point cloud subset obtained by downsampling in Step 1-1 as targets, features of different resolutions of each point are obtained by three nearest neighbor interpolation. These features are then combined with the multimodal features of each point obtained in Step 2-5 and fused using a multilayer perceptron to obtain the fused point cloud.
[0102] Step 4-2: Based on the voting mechanism, a set of key regions that may be the center of the object are aggregated from the output point cloud of Step 4-1, and initial features are generated for each key region through the Set Abstraction structure.
[0103] Step 4-3 uses two modules, mutual attention between key regions and input point cloud, and self-attention between key regions, to perform semantic feature aggregation at the object level and scene level, respectively, and provides preliminary detection results.
[0104] Step 4-4: Using the detection proposal obtained in Step 4-3 as a priori, the geometric structure features of the foreground object corresponding to the key region are extracted with the help of the RBG feature aggregation module, and more accurate detection results are generated.
[0105] Figure 4 This paper presents the detection results of the image-assisted 3D object detection method proposed in this invention in four different scenarios. The first column shows the ground truth annotation information of the scene, the second column shows the detection results of the 3D object detection method based on a single point cloud modality, and the third column shows the detection results of the method proposed in this invention. The comparison results show that the method of this invention has better accuracy in both semantic category and bounding box geometry.
[0106] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the inventive content of the image information-assisted three-dimensional object detection method provided by this invention, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0107] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MCU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.
[0108] This invention provides an idea and method for image-assisted 3D object detection. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.
Claims
1. A method of image information assisted three-dimensional object detection, characterized by, Includes the following steps: Step 1: Extract features from the point cloud containing the 3D object to be detected and its corresponding image. Step 2: Based on the block self-attention mechanism, the features of the point cloud and the image are fused. The input point cloud is divided into multiple overlapping blocks. In each block, feature aggregation is performed point by point based on the self-attention mechanism. Finally, all blocks are merged back into the structure of the input point cloud to obtain the fused multimodal features. Step 3: Based on the fused multimodal features, a foreground point mask is generated and used to extract multi-scale features with a focus on the foreground, resulting in feature maps of the point cloud at different resolutions. Specifically, a foreground point mask is generated for each point based on the multimodal features. A mask-guided sampling method is then used to obtain a multi-scale sampling subset of the foreground-focused region. Finally, an improved Set Abstraction structure is used to extract features from each subset. The improved Set Abstraction structure specifically includes: Step 3-3-1: Given a high-resolution source point cloud and a low-resolution target point cloud, find the neighbor point set from the source point cloud for each point in the target point cloud according to the radius of the sphere. Step 3-3-2: For each set of neighboring points, generate an offset code based on the local offset between the neighboring points and the sampling center, add it to the original features of each point, and then send it to the multilayer perceptron for processing. Step 3-3-3 introduces a mask-based background point feature masking method, which sets the background point features in each neighbor point set to zero based on the foreground point mask, and merges the features of each neighbor point set through max pooling to obtain the features of each point in the target point cloud. Step 4: Generate an object detection proposal. This involves interpolating feature maps at different resolutions using the three nearest neighbor interpolation method, fusing the interpolated features with multimodal features using a multilayer perceptron, and feeding the results into a two-stage detection module. The two-stage detection process enhances the object features in terms of semantic information and geometric structure information, thereby generating an object detection proposal and completing the image-assisted 3D object detection.
2. The image-assisted 3D object detection method as described in claim 1, characterized in that, The feature extraction described in step 1 specifically includes: Step 1-1: Perform preliminary feature extraction on the point cloud containing the 3D object to be detected to obtain a subset of the point cloud after downsampling. The subset of the point cloud contains the geometric features of each point. Steps 1-2: Use a feature extraction network to generate a feature map containing semantic and texture information for the image corresponding to the point cloud; Steps 1-3: Based on the projection relationship between the point cloud and the image determined by the camera's intrinsic and extrinsic parameters, obtain the pixel coordinates of each point in the image within the point cloud subset output in Step 1-1, and use bilinear interpolation to obtain the image features of each point on the image feature map output in Step 1-2.
3. The image-assisted 3D object detection method according to claim 2, characterized in that, The feature extraction network described in steps 1-2 consists of a downsampling part composed of a deep convolutional network and an upsampling part composed of three transposed convolutional layers with residual structures, wherein: The downsampling part, i.e. the backbone network, is any deep convolutional network trained on the ImageNet classification task; The upsampling part is as follows: Four feature maps with progressively decreasing resolution are uniformly selected from the outputs of each part of the backbone network to form the input of the upsampling part. The upsampling part uses transposed convolution to connect every two feature maps, and performs cross-layer feature fusion by concatenating the output of the transposed convolution with the original output of the corresponding module of the downsampling network and then using a single-layer convolution processing method.
4. The image-assisted three-dimensional object detection method according to claim 3, characterized in that, Step 2, which involves fusing features from point clouds and images using a block-based self-attention mechanism, specifically includes: Step 2-1: The geometric features of each point obtained in Step 1-1 are concatenated with the image features of each point obtained in Step 1-3, and a single-layer convolution is used for preliminary fusion to obtain the preliminary fused features. Step 2-2: Divide the point cloud into multiple overlapping blocks in space; Steps 2-3: Within each block, feature aggregation is performed based on a self-attention mechanism; Steps 2-4: For each point in the point cloud, the final feature of that point, i.e., the aggregated feature, is obtained by averaging the features obtained by aggregating the features across all the blocks to which it is divided. Step 2-5: The preliminary fused features obtained in Step 2-1 are processed by single-layer convolution and added as residuals to the aggregated features obtained in Step 2-4, which are the fused multimodal features.
5. The image-assisted three-dimensional object detection method according to claim 4, characterized in that, Step 2-2, which involves spatially dividing the point cloud into multiple overlapping regions, specifically includes: For N points in the point cloud, M block centers are obtained by sampling the farthest point, and then K intra-block points are obtained by using nearest neighbor search for each block center; During the block segmentation process, M×K≥4N is taken, and the maximum radius of each block is limited. The portion of the K points within a block that exceeds the preset radius does not participate in feature aggregation.
6. The image-assisted three-dimensional object detection method according to claim 5, characterized in that, Step 3, which involves generating a foreground point mask and using it for multi-scale feature extraction with a focus on the foreground, specifically includes: Step 3-1: For each point in the point cloud containing the 3D object to be detected, based on the fused multimodal features, a multilayer perceptron is used to generate a foreground probability prediction, and a foreground point segmentation mask is obtained based on a preset threshold. Step 3-2: Using a mask-guided sampling method, generate three subsets of the point cloud at different scales with the number of points halved in succession; Step 3-3: Extract features sequentially from the three subsets of different scales in descending order of resolution to obtain feature maps at different resolutions.
7. The image-assisted three-dimensional object detection method according to claim 6, characterized in that, Step 3-2, which describes using a mask-guided sampling method to generate three subsets of the point cloud at different scales with the number of points successively halved, specifically includes: Based on the foreground point segmentation mask, the coordinates of all background points are reduced by a preset ratio and moved to the vicinity of any foreground point. Then, by performing farthest point sampling on the adjusted point cloud with three different sampling numbers, three subsets of the point cloud at different scales are obtained.
8. The image-assisted three-dimensional object detection method according to claim 7, characterized in that, Step 4, generating object detection proposals, specifically includes: Step 4-1: Using the feature maps at three different resolutions output in Step 3-3 as feature sources, and the coordinates of the point cloud subset obtained by downsampling in Step 1-1 as targets, the features at different resolutions of each point are obtained by three nearest neighbor interpolation. These features are then concatenated with the multimodal features of each point obtained in Step 2-5 and fused using a multilayer perceptron fusion processing method to obtain the fused point cloud. Step 4-2: Based on the voting mechanism, a set of key regions that may be the center of the object are aggregated from the fused point cloud output in Step 4-1, and initial features are generated for each key region through the improved Set Abstraction structure described in Step 3-3. Step 4-3: Based on the mutual attention mechanism, the key regions are used as queries and the point clouds are used as keys and values to enhance the features of each key region from the object semantic level. The self-attention mechanism is applied between key regions to enhance the features of key regions from the scene semantic level. Finally, a preliminary detection result is given for each key region based on the enhanced features. Step 4-4: Based on the preliminary detection results, use the RBG feature aggregation module to aggregate the geometric features of the specific object, thereby optimizing the preliminary detection results and obtaining the final target detection proposal.