Aerial guided alignment consistency three-dimensional object detection method and system

Abstract

The application discloses a kind of aerial guide alignment consistency three-dimensional target detection method and system, comprising: the rule voxel grid data generated is utilized 3D backbone network to carry out multiple downsampling to it, after each downsampling, further extract sparse feature, utilize aerial projection fusion module to carry out three-dimensional convolution and aerial view projection after two-dimensional convolution to sparse feature respectively, after two-dimensional convolution result is restored to voxel space, it is fused with three-dimensional convolution result, generates fusion feature, to obtain multiscale voxel feature in this way.Voxel feature is input 2D backbone network and carries out two-dimensional convolution feature extraction, generates aerial view feature and inputs regional proposal generation module, generates target candidate region and is input to ROI network with multiscale voxel feature simultaneously to carry out feature alignment and extraction, finally output target detection result.The application can effectively improve three-dimensional target detection precision, and reduce missed detection and misjudgment phenomenon to some extent, further improve the stability and reliability of detection result.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a bird's-eye view-guided method and system for aligning and consistent 3D target detection. Background Technology

[0002] 3D target detection data is widely used in autonomous driving, smart logistics, and security monitoring due to its rich spatial information and accurate target location. However, in real-world applications, targets exhibit varying scales and are susceptible to scene interference, making detection tasks challenging. To address this issue, researchers have proposed various improvement schemes, primarily focusing on multimodal information fusion and enhanced feature representation, aiming to improve the accuracy and robustness of target detection in complex scenes.

[0003] Numerous studies have focused on feature fusion between images and point clouds in multimodal 3D object detection to improve the detection performance and robustness of models. Feature projection fusion methods include CenterFusion, PointPainting, EPNET, and BEVFusion. CenterFusion utilizes a frustum association mechanism to accurately associate sparse radar points with the center of objects in the image, effectively leveraging radar depth and velocity characteristics to supplement the spatial semantics in the image. PointPainting first performs semantic segmentation on the camera image, then uses calibration relationships to map each LiDAR point to image coordinates, reads the segmented semantic information at its corresponding pixel, and adds this semantic information as an additional attribute to the point cloud, forming an enhanced point cloud with semantic labels, which is finally input into the 3D detector to improve detection results. EPNET establishes a fine correspondence between point cloud and image features in a point-by-point manner, adaptively estimating the importance of image features to enhance point features and suppress interference information. BEVFusion innovatively extracts features from cameras and LiDAR independently and maps them to a unified Bird's-Eye View (BEV) space, achieving decoupled fusion within this space. This allows different modalities to perceive independently yet collaboratively, significantly improving the system's robustness under sensor failure or noise. Attention-based fusion methods include Bridged Transformer, IGNet, and LidarIG. Bridged Transformer uses the same positional encoding for image and point cloud data to achieve attention bridging, fusing point cloud and image information at the feature level. IGNet focuses on deep modeling of correlations between different modalities, utilizing a bidirectional interactive attention mechanism and a fusion feature pyramid network to enhance the complementary capabilities of image and LiDAR features. LidarIG employs a quantization perception mechanism to coarsely align pixel blocks and point clouds, then refines spatial features through a dynamic Gaussian distribution, combining posterior density perception to filter high-quality Region of Interest (RoI) features, effectively mitigating information loss caused by point cloud sparsity.

[0004] In addition, some studies explore enhancing feature representation. The SEED method proposes a deformable mesh attention module that divides the reference box into a mesh and predicts offsets, obtaining a more flexible receptive field to capture more informative features. PIPC-3Ddet effectively improves detection accuracy and spatial semantic representation in complex backgrounds through viewpoint information embedding and proposal relevance inference. FocalsConv dynamically selects sparse regions to be expanded through a learnable importance guidance mechanism, thereby improving the feature representation of foreground objects while maintaining efficiency. GE-FSOD aims to generate richer and more accurate multi-scale feature representations by dynamically generating spatially adaptive weights to balance contributions from high-level semantic features, mid-level features, and current-level detail features. Relation-DETR explicitly models the positional relationship between detection boxes using normalized relative geometric features and integrates this relationship as a learnable bias into the self-attention mechanism of the Transformer decoder.

[0005] Existing methods have made some progress in feature fusion and feature representation enhancement for multimodal 3D object detection, effectively improving the detection performance and robustness of the models. However, these methods still have certain limitations: feature projection fusion methods are prone to introducing alignment bias when the depth estimation error is large, making it difficult for cross-modal information to be fully complementary; attention-based fusion methods can promote cross-modal information exchange, but their computational complexity may be high; and methods for enhancing feature representation often focus on expanding the receptive field of voxel / point cloud features, resulting in relatively insufficient utilization of the 2D image space. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides a bird's-eye view-guided alignment-consistency 3D target detection method, system, and storage medium. It aims to overcome the problems of insufficient utilization of cross-modal information, unstable feature alignment, and difficulty in effectively applying supplementary image information to 3D representation learning in complex scenes, which are problems encountered by traditional multimodal 3D target detection methods. This invention can effectively improve the accuracy of 3D target detection, reduce missed and false detections to a certain extent, and further enhance the stability and reliability of the detection results.

[0007] According to a first aspect of the present disclosure, a bird's-eye view-guided method for detecting aligned and consistent 3D targets is provided, the method comprising the following steps: Acquire real point cloud and virtual point cloud data, and perform point cloud voxelization processing on the real point cloud and virtual point cloud data to generate regular voxel mesh data; Regular voxel grid data is input into a 3D backbone network. The 3D backbone network performs multiple downsampling operations on the input data. After each downsampling, the corresponding sparse features are extracted through sparse convolution. The bird's-eye view projection fusion module is then used to perform 3D convolution and 2D convolution operations on the sparse features after the bird's-eye view projection. The 2D convolution results are restored to voxel space and fused with the 3D convolution results of the corresponding stage to generate fused features. Finally, a multi-scale voxel feature composed of multiple fused features is obtained. The multi-scale voxel feature is input into a 2D backbone network for 2D convolution feature extraction to generate bird's-eye view features. The bird's-eye view features are input into the region proposal generation module. Based on the prediction results of the foreground and background features of the bird's-eye view, the module selects regions where objects to be detected may exist, interacts and integrates the features of the regions, and further updates the global features. Based on the prediction results of the foreground and background features of the bird's-eye view, the module selects initial candidate boxes. Using the updated global features, the module refines the initial candidate boxes in the decoder to obtain the final target candidate proposal. The target candidate proposals and multi-scale voxel features are input into the ROI network for feature alignment and extraction, and finally the target detection results are output.

[0008] In some embodiments, the bird's-eye view projection fusion module splits sparse features into a first feature group and a second feature group. It performs 3D convolution operations on the first feature group to extract 3D spatial geometric features. Simultaneously, it projects the second feature group onto the bird's-eye view and performs 2D convolution operations to extract 2D semantic features. The 2D semantic features are then backfilled into the original 3D voxel space and stitched together with the 3D spatial geometric features to obtain stitched features. Finally, a second 3D convolution operation is performed on the stitched features to output the fused bird's-eye view projection feature map.

[0009] In some embodiments, the bird's-eye view projection fusion module implements the following steps: Obtaining 3D indexes of non-empty voxels and its corresponding eigenvectors ,in For the number of non-empty voxels, Input the number of channels; The feature vector is convolved through the first 3D submanifold. Encode to obtain the first three-dimensional geometric features ; The 3D index is compressed along the height direction to the bird's-eye view plane, and all voxel features projected onto the same plane are aggregated to obtain the deduplicated bird's-eye view index. and their corresponding aggregation features ,in This represents the number of unique bird's-eye view indexes after deduplication. The aggregated features are obtained by 2D submanifold convolution. Encode the geometric features of the bird's-eye view to obtain the geometric features. ; Based on the mapping relationship between the original non-empty voxel 3D index and the deduplicated bird's-eye view index, the bird's-eye view geometric features are backfilled to each non-empty voxel position to obtain bird's-eye view geometric features corresponding one-to-one with each voxel. ; The first three-dimensional geometric features With the aforementioned bird's-eye view geometry By stitching along the channel dimension, the fused features are obtained. ; The fused features are obtained by convolution of a second 3D submanifold. Encode and output the enhanced voxel features. .

[0010] In some embodiments, aggregation features This can be achieved in the following ways: Calculate the coordinates of each plane Corresponding features and With voxel count , For the number of non-empty voxels, Indicates the location The corresponding feature vector ; The aggregation feature is obtained by dividing the sum of the number of voxels and the minimum constant by the sum of the features. ,in It is a very small constant.

[0011] In some embodiments, the three-dimensional convolution result With 2D convolution results They are processed centrally separately to obtain , , , , This represents the mean vector within the same batch. and and With indicates the first Characteristics of individual element location; A centralized contrast consistency loss function is constructed to constrain the alignment relationship of cross-modal features in the representation space in the bird's-eye view projection fusion module. The specific expression is as follows: , This represents the number of point clouds in the current batch. It is the weight assigned to the loss.

[0012] In some embodiments, the method further includes constructing loss functions for the region proposal generation module and the ROI network respectively, wherein the loss function of the region proposal generation module is the sum of the first classification branch loss, the first regression loss, and the contrast loss, and the loss function of the ROI network is the sum of the second classification branch loss and the second regression loss. The first classification branch loss uses focus loss to alleviate the imbalance between positive and negative samples, the first regression loss uses smoothing L1 loss to constrain the prediction box parameters and rotation angle error, and the second classification branch loss uses binary cross-entropy loss; regression loss... Smoothing L1 loss is used to optimize the rotation angle and bounding box regression error.

[0013] In some embodiments, the method is specifically implemented by including the following steps: The image is fed into a pre-trained virtual point cloud generation network to obtain a virtual point cloud estimated from the image. The virtual point cloud is then stitched together with the real point cloud at the point level to form an expanded point cloud. Perform a voxelization operation on the expanded point cloud to obtain a voxel representation; In the voxel domain, the input convolutional block is used to encode the voxel representation to obtain the initial sparse voxel features; The initial sparse voxel features are sequentially input into multiple projection convolutional blocks for multi-scale feature extraction. Each projection convolutional block consists of a downsampling layer, two sparse convolutional blocks, and a bird's-eye view projection fusion module, outputting features at the corresponding scale. , Indicates the number of projected convolutional blocks; Features of the highest level Extract bird's-eye view features from a 2D backbone network; The bird's-eye view features are input into the region proposal generation module, and the target candidate proposals are obtained after processing.

[0014] Multi-scale voxel features are used to aggregate features and refine candidate boxes for target candidate proposals, resulting in the final detection results.

[0015] According to a second aspect of the present disclosure, a bird's-view guided alignment consistency 3D target detection system is provided, the system comprising: The data input module is used to acquire real point cloud and virtual point cloud data, and to perform point cloud voxelization processing on the real point cloud and virtual point cloud data to generate regular voxel mesh data. A 3D backbone network is used to perform multiple downsampling operations on the input data. After each downsampling, the corresponding sparse features are extracted through sparse convolution. The bird's-eye view projection fusion module is used to perform 3D convolution and 2D convolution operations on the sparse features after the bird's-eye view projection. The 2D convolution results are restored to the voxel space and fused with the 3D convolution results of the corresponding stage to generate fused features. Finally, a multi-scale voxel feature composed of multiple fused features is obtained. A 2D backbone network is used to extract multi-scale voxel features by performing two-dimensional convolutional features to generate bird's-eye view features. The region proposal generation module is used to filter out regions where objects to be detected may exist based on the prediction results of the foreground and background features of the bird's-eye view. It interacts and integrates the features of the regions, further updates the global features, filters out initial candidate boxes based on the prediction results of the foreground and background features of the bird's-eye view, and refines the initial candidate boxes in the decoder using the updated global features to obtain the final target candidate proposal. The ROI network is used to align and extract features from the target candidate proposals, and finally output the target detection results.

[0016] According to a third aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described bird's-eye view-guided alignment consistency three-dimensional target detection method.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein computer instructions are stored on the storage medium, and when executed by a processor, the instructions implement the steps of the above-described bird's-eye view-guided alignment consistency three-dimensional target detection method.

[0018] This disclosure provides a bird's-eye view-guided alignment-consistency 3D target detection method, system, and storage medium, aiming to overcome the problems of insufficient utilization of cross-modal information, unstable feature alignment, and difficulty in effectively applying image supplementary information to 3D representation learning in complex autonomous driving scenarios by traditional multimodal 3D target detection methods. Specifically, this invention effectively couples image semantic information with point cloud spatial geometric information through spatial projection and consistency constraints. This approach not only alleviates the problem of insufficient discriminative information caused by limited point cloud resolution and missing details at long distances, thereby improving the accuracy of 3D target detection, but also enhances the consistency and separability of cross-modal features during the fusion process, reducing the risk of false detection and false negatives caused by calibration errors, occlusion, and differences in modal distribution, and fully leveraging the supplementary role of image features in point cloud detection. Specifically, this invention establishes a correspondence between 3D voxels (or voxel centers) and 2D image plane positions through geometric projection, and constructs a projection-view feature fusion mechanism during the feature fusion stage, enabling 3D voxel features to obtain appearance and semantic cues from the corresponding image representations. Based on this, a centralized contrast consistency loss is designed to impose consistency constraints on 3D voxel features and their 2D projection features: cross-modal features at the same corresponding position should remain similar in the embedding space, and features at different corresponding positions should have good separability. Simultaneously, the centralization operation suppresses global shifts in cross-modal features, improving the stability of consistency learning. Through the organic combination of these key mechanisms, this invention can provide a more accurate, stable, and robust solution for 3D object detection in complex autonomous driving scenarios.

[0019] In summary, the beneficial effects of this invention specifically include: establishing a correspondence between point clouds and images through spatial projection and feature alignment mechanisms, organically fusing the rich semantic information in images with the spatial geometric perception capabilities of point clouds; simultaneously, designing an efficient multimodal fusion module to achieve deep complementarity of cross-modal features, thereby enhancing the network's overall representation ability and discrimination accuracy for targets at different scales. Compared to traditional methods that only use point clouds, the method of this invention can effectively improve the accuracy of 3D target detection in various complex driving scenarios, and to a certain extent reduce missed detections and false detections, further improving the stability and reliability of the detection results.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0022] Figure 1This is a flowchart of the bird's-eye view-guided alignment consistency 3D target detection method in an embodiment of the present invention; Figure 2 This is a bird's-eye view-guided alignment consistency 3D target detection network architecture diagram in an embodiment of the present invention; Figure 3 This is a schematic diagram of the alignment consistency three-dimensional target detection system guided by bird's-eye view in an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present invention are shown in the drawings, not the entire structure.

[0025] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0026] 3D object detection plays a crucial role in safety-critical applications such as autonomous driving and intelligent robotics. However, in complex and ever-changing road scenarios, the overall discrimination capability and localization accuracy of detection models remain challenging due to limitations in the spatial resolution and sparse sampling characteristics of point cloud data, as well as the influence of environmental changes and sensor noise. Especially when dealing with distant, small-scale targets and incomplete information caused by occlusion, the limited number of usable points and insufficient geometric features in the point cloud make it difficult to effectively characterize the fine-grained semantics and texture cues of objects, easily leading to missed detections and false detections, thus limiting the robustness and reliability of the system in complex scenarios.

[0027] This invention addresses the challenges of limited point cloud resolution, lack of long-distance detail, and difficulty in effectively aligning cross-modal information in complex autonomous driving scenarios. It proposes a bird's-view guided alignment-consistency feature extraction network (BVACNet) method for 3D object detection. This method explicitly enhances the consistency between 3D voxel features and their 2D projected representations through projection fusion of 3D features to 2D space and a centralized contrast consistency loss constraint. This allows for more efficient utilization of supplementary semantic information from image features, effectively improving the overall accuracy of 3D object detection in complex scenes and reducing false positives and false negatives. 3D object detection technology shows broad application prospects in various practical scenarios, including autonomous driving, robot navigation and logistics, smart cities, and medical and rehabilitation assistance.

[0028] (1) Autonomous driving In autonomous driving scenarios, vehicles need to perceive surrounding targets in real time within dynamic and complex traffic environments, such as distant vehicles, pedestrians, and partially occluded targets. 3D target detection can provide more accurate spatial geometric information and distance estimation, improving the ability to identify and locate key targets. This provides reliable input for decision-making modules such as path planning, obstacle avoidance, lane changing, and following, thereby enhancing the safety and stability of the system.

[0029] (2) Robot Navigation and Logistics In scenarios such as warehousing and logistics, park delivery, and factory production lines, mobile robots need to perform autonomous localization, path planning, and material handling. 3D object detection can help robots identify shelves, pallets, packages, and various dynamic obstacles, and achieve more robust obstacle avoidance and collaborative operation in narrow passages or environments with mixed pedestrian traffic, thereby improving the efficiency and safety of logistics automation.

[0030] (3) Smart City In urban governance and public services, 3D object detection can be used for tasks such as traffic element identification, road and municipal facility inspection, 3D map updating, and congestion and anomaly analysis. Through precise detection and positioning of urban spatial elements, more efficient infrastructure management and more timely safety warnings can be supported, improving the precision and intelligence of urban operations.

[0031] (4) Medical and Rehabilitation Assistance In tasks such as 3D reconstruction of medical images, preoperative assessment, and intraoperative navigation, 3D target detection can be used to locate lesions, organs, or key anatomical structures, assisting doctors in quantitative analysis and precise operation. In rehabilitation training and movement assessment, it can also be combined with 3D perception to detect and track human posture and key parts, providing objective basis for assessment and intervention.

[0032] With the improvement of point cloud acquisition and processing capabilities, as well as the continuous development of deep learning and multimodal fusion methods, 3D object detection will continue to expand its application boundaries in the above-mentioned fields and provide more reliable technical support for high-precision perception and intelligent decision-making.

[0033] A bird's-eye view-guided method for aligned and consistent 3D object detection, such as Figure 1 As shown, it includes the following steps: S1. Acquire real point cloud and virtual point cloud data, and perform point cloud voxelization processing on the real point cloud and virtual point cloud data to generate regular voxel mesh data. S2. Input the regular voxel grid data into the 3D backbone network. The 3D backbone network performs multiple downsampling operations on the input data. After each downsampling, the corresponding sparse features are extracted through sparse convolution. The bird's-eye view projection fusion module is used to perform three-dimensional convolution and two-dimensional convolution operations on the sparse features after the bird's-eye view projection. The two-dimensional convolution results are restored to the voxel space and fused with the three-dimensional convolution results of the corresponding stage to generate fused features. Finally, a multi-scale voxel feature composed of multiple fused features is obtained. S3. Input the multi-scale voxel features into the 2D backbone network to extract two-dimensional convolutional features and generate bird's-eye view features; S4. The bird's-eye view features are input into the region proposal generation module. This module first filters out regions where objects to be detected may exist based on the prediction results of the foreground and background features of the bird's-eye view. Then, the features of these regions are interacted and integrated to update the global features. Subsequently, initial candidate boxes are selected based on the foreground and background prediction results. Finally, the updated global features are used to refine the initial candidate boxes in the decoder to obtain the final target candidate proposals. S5. Input the target candidate proposals and multi-scale voxel features into the ROI network for feature alignment and extraction, and finally output the target detection results.

[0034] In the specific implementation process, the overall network architecture of the method is as follows: Figure 2As shown, the framework consists of a data input module, a 3D backbone network, a 2D backbone network, a region proposal generation module, and an ROI network. Based on this, two new key designs are introduced into the 3D backbone network: (a) a bird's-eye view projection fusion module, which organically combines the rich semantic information of the image with the spatial perception capability of the point cloud, effectively injecting 2D appearance and semantic cues into the 3D feature learning process while maintaining the 3D geometric structure representation; (b) a centralized contrast consistency loss, used to explicitly enhance the consistency between 3D voxel features and their 2D projection representations, suppress cross-modal feature distribution shifts, and improve the separability of non-corresponding features through a contrastive learning mechanism, thereby enhancing the stability and discriminative ability of the fused representation.

[0035] The bird's-eye view projection fusion module splits sparse features into a first feature group and a second feature group. It performs 3D convolution operation on the first feature group to extract 3D spatial geometric features. At the same time, it projects the second feature group onto the bird's-eye view and performs 2D convolution operation to extract 2D semantic features. The 2D semantic features are backfilled into the original 3D voxel space and stitched with the 3D spatial geometric features to obtain stitched features. The stitched features are then subjected to a second 3D convolution operation to output the fused bird's-eye view projection feature map.

[0036] The implementation steps of the bird's-eye view projection fusion module include: Obtaining 3D indexes of non-empty voxels and its corresponding eigenvectors ,in For the number of non-empty voxels, Input the number of channels; The feature vector is convolved through the first 3D submanifold. Encode to obtain the first three-dimensional geometric features ; The 3D index is compressed along the height direction to the bird's-eye view plane, and all voxel features projected onto the same plane are aggregated to obtain the deduplicated bird's-eye view index. and their corresponding aggregation features ,in This represents the number of unique bird's-eye view indexes after deduplication. The aggregated features are obtained by 2D submanifold convolution. Encode the geometric features of the bird's-eye view to obtain the geometric features. ; Based on the mapping relationship between the original non-empty voxel 3D index and the deduplicated bird's-eye view index, the bird's-eye view geometric features are backfilled to each non-empty voxel position to obtain bird's-eye view geometric features corresponding one-to-one with each voxel. ; The first three-dimensional geometric features With the aforementioned bird's-eye view geometry By stitching along the channel dimension, the fused features are obtained. ; The fused features are obtained by convolution of a second 3D submanifold. Encode and output the enhanced voxel features. .

[0037] Aggregation features This can be achieved in the following ways: Calculate the coordinates of each plane Corresponding features and With voxel count , The number of non-empty voxels; The aggregation feature is obtained by dividing the sum of the number of voxels and the minimum constant by the sum of the features. ,in It is a very small constant.

[0038] In practical implementation, during the early stages of multimodal fusion, geometric projection is often used to introduce image features at corresponding locations into point cloud features, thereby utilizing richer semantic and textural information in the image. However, if features are directly organized and aggregated on the image plane, the depth compression caused by perspective projection will make the local neighborhood relationships on the image plane inconsistent with the three-dimensional geometric neighborhood: pixel positions that are close in 2D may correspond to different depths or even different object surfaces, thus introducing contextual interference that does not conform to the three-dimensional structure, weakening the stability and effectiveness of cross-modal feature fusion. To address this, this invention adopts a "projection to bird's-eye view (BEV)" strategy to organize cross-modal interaction, that is, mapping supplementary information from the image to a unified BEV coordinate system, and completing feature encoding and fusion in the BEV space. Since the BEV representation is based on ground plane coordinates, its neighborhood structure is closer to the spatial distribution and motion relationships of targets in autonomous driving scenarios, which can alleviate the neighborhood mismatch problem introduced by the camera perspective to a certain extent. Based on this idea, the present invention designs a bird's-eye view projection fusion module: First, the cross-modal correspondence is unified to the BEV plane, and the features are encoded in a set and interactively modeled on the plane; then the fusion result of the BEV space is backfilled into the original three-dimensional voxel set and further fused with the 3D geometric features to obtain a more discriminative and robust three-dimensional voxel representation, and to provide more stable geometric-semantic joint features for the subsequent detection head.

[0039] In the specific implementation process, the 3D index of the non-empty voxel is set as follows: Its corresponding feature vector ,in For the number of non-empty voxels, The number of input channels is given. First, the geometric features of the 3D voxels are encoded using 3D submanifold convolution, as shown in Equation (1).

[0040] (1) To fully utilize image features and establish geometrically consistent correspondences within the Bird's-Eye View (BEV) space, this section organizes the 3D voxel index by directly projecting it onto the BEV plane. Specifically, given the input 3D voxel index... Compress it along the Z-axis, i.e., the height direction, to obtain the corresponding planar index. Because voxels at different altitudes may correspond to the same... Position, after projection, will result in index overlap. To eliminate this overlap and form a stable BEV representation, for positions with the same index... The voxel features are averaged and aggregated to obtain representative features for that BEV location. As shown in equation (2).

[0041] (2) (3) (4) in, This corresponds to each 2D index position after deduplication. coordinate, This represents the number of 2D indexes after deduplication. To avoid dividing by zero, the smallest constant.

[0042] Obtain the deduplicated BEV index and its aggregation characteristics Then, use 2D submanifold convolution kernels. Encoding yields geometric features at the bird's-eye view level. Subsequently, based on the mapping relationship from voxels to BEV indices, the... Backfill to the original The location of each voxel is used to obtain the bird's-eye view geometric features that correspond one-to-one with the voxel. .

[0043] After obtaining the three-dimensional and two-dimensional geometric features, they are concatenated along the feature channels to obtain a fused feature vector. Finally, 3D submanifold convolution is applied again to the fused features to further interact and integrate cross-spatial information, outputting enhanced voxel features. .

[0044] In a preferred embodiment, the three-dimensional convolution result With 2D convolution results They are processed centrally separately to obtain , , , , This represents the mean vector within the same batch. and and With indicates the first Characteristics of individual element location; A centralized contrast consistency loss function is constructed to constrain the alignment relationship of cross-modal features in the representation space in the bird's-eye view projection fusion module. The specific expression is as follows: , This represents the number of point clouds in the current batch. It is the weight assigned to the loss.

[0045] Specifically, in multimodal 3D object detection tasks, the texture, edge, and high-level semantic information contained in image modalities can serve as important supplements to point cloud geometric features. On the one hand, these appearance cues can provide more sufficient discriminative information when point clouds are sparse or lack distant details; on the other hand, for targets that are relatively easy to detect, image features can also help improve the stability of classification and regression prediction, thereby further improving detection accuracy. This invention introduces a centered contrast consistency loss to explicitly enhance the consistency between 3D voxel features and their 2D projected representations, so that the texture and edge cues provided by image modalities can more stably supplement the 3D geometric representation.

[0046] Unlike methods that model local relationships solely within a 3D voxel mesh, 3D voxels are first projected onto a 2D image plane. This allows features to establish correspondences in 2D space based on imaging geometry, mitigating to some extent the representational fragmentation caused by features that are similar in 2D but dissimilar in 3D space. Building upon this, consistency constraints are applied to both 3D and 2D features, ensuring that cross-modal features at corresponding projection locations remain similar in representation space, while features at non-corresponding locations remain separable. This explicitly strengthens cross-modal alignment and enhances the discriminativeness and stability of the fused features. Furthermore, considering the potential systematic shifts in scale and distribution of cross-modal features, both types of features are centered before calculating contrastive consistency to weaken the impact of global bias. This makes the consistency learning process more stable and focused on the alignment of discriminative structures.

[0047] Specifically, for the backbone network in the first... 3D voxel features output by the layer projection bird's-eye view feature fusion module and the two-dimensional features corresponding to its projection position. Since cross-modal features may exhibit systematic shifts in scale and distribution, to mitigate the impact of global bias on similarity metrics, the two types of features are centered separately: (5) (6) (7) (8) in, and This represents the mean vector within the same batch. and and With indicates the first Characteristics of individual element location.

[0048] Based on this, a centralized comparison consistency loss is constructed, as shown in Equation (9).

[0049] (9) in, This represents the number of point clouds in the current batch. The weights are assigned to the loss, by minimizing This makes the 3D and 2D embeddings at the same projection position closer in the feature space and suppresses the similarity with other non-corresponding 2D features, thereby explicitly strengthening the cross-modal alignment relationship. This allows 2D image features to supplement 3D voxel representation in a more stable way, ultimately improving the discriminativeness and robustness of the fused features.

[0050] In a preferred embodiment, the method further includes constructing loss functions for the region proposal generation module and the ROI network respectively. The loss function for the region proposal generation module is the sum of the first classification branch loss, the first regression loss, and the contrast loss. The loss function for the ROI network is the sum of the second classification branch loss and the second regression loss. The first classification branch loss uses focus loss to alleviate the imbalance between positive and negative samples. The first regression loss uses smoothing L1 loss to constrain the prediction box parameters and rotation angle error. The second classification branch loss uses binary cross-entropy loss. Smoothing L1 loss is used to optimize the rotation angle and bounding box regression error.

[0051] In practical implementation, the overall loss function can be expressed as: (10) (11) (12) in, The centralization consistency loss is used to constrain the alignment relationship of cross-modal features in the representation space. and These correspond to the training objectives of the Region Proposal Generation (RPN) module and the ROI network, respectively. For the RPN module, its classification branch loss... Focal loss is used to mitigate the imbalance between positive and negative samples; regression loss. A smoothed L1 loss is used to constrain the predicted bounding box parameters (including bounding box regression errors such as position / size) and rotation angle errors. Furthermore, a contrastive loss is introduced as auxiliary supervision to bring the predictions of ground truth queries and normal queries in the denoised group (DN) closer in the feature space and further separate them from non-matching samples, thereby improving the stability of query matching and feature discriminativeness. For the ROI network, its classification loss... Binary cross-entropy loss is used; regression loss is employed. The same smoothed L1 loss is used to optimize the rotation angle and bounding box regression error, further improving the localization accuracy of the candidate boxes. The algorithm flow is shown in Table 1 for a clearer illustration of the training process.

[0052] Table 1

[0053] In a preferred embodiment, the method specifically includes the following steps: The image is fed into a pre-trained virtual point cloud generation network to obtain a virtual point cloud estimated from the image. The virtual point cloud is then stitched together with the real point cloud at the point level to form an expanded point cloud. Perform a voxelization operation on the expanded point cloud to obtain a voxel representation; In the voxel domain, the input convolutional block is used to encode the voxel representation to obtain the initial sparse voxel features; The initial sparse voxel features are sequentially input into multiple projection convolutional blocks for multi-scale feature extraction. Each projection convolutional block consists of a downsampling layer, two sparse convolutional blocks, and a bird's-eye view projection fusion module, outputting features at the corresponding scale. , Indicates the number of projected convolutional blocks; Features of the highest level Extract bird's-eye view features from a 2D backbone network; The bird's-eye view features are input into the region proposal generation module, and the target candidate proposals are obtained after processing.

[0054] Multi-scale voxel features are used to aggregate features and refine candidate boxes for target candidate proposals, resulting in the final detection results.

[0055] In a specific example, the overall process is as follows: First, the input image is fed into a pre-trained virtual point cloud generation network to obtain a virtual point cloud estimated from the image. This virtual point cloud is then concatenated with the original LiDAR point cloud at the point level to form an expanded point cloud P'. Subsequently, voxelization is performed on P' to obtain a voxel representation V. In the voxel domain, input convolutional blocks are used to encode features in V, obtaining initial sparse voxel features. Next, Four projection convolutional blocks are sequentially input for multi-scale feature extraction. Each projection convolutional block consists of a downsampling layer, two sparse convolutional blocks, and a bird's-eye view projection fusion module, outputting features at the corresponding scale. Among them, the highest level features Further input into a 2D backbone network to extract bird's-eye view features Then, Input Region Proposal Network (RPN) generates initial candidate boxes Finally, multi-scale features are utilized. right Feature aggregation and candidate box refinement are performed to obtain the final detection results. .

[0056] Another embodiment provides a bird's-eye view-guided alignment consistency 3D target detection system 300, comprising: The data input module 310 is used to acquire real point cloud and virtual point cloud data, and to perform point cloud voxelization processing on the real point cloud and virtual point cloud data to generate regular voxel mesh data. The 3D backbone network 320 is used to perform multiple downsampling operations on the input data. After each downsampling, the corresponding sparse features are extracted through sparse convolution. The bird's-eye view projection fusion module is used to perform 3D convolution and 2D convolution operations on the sparse features after the bird's-eye view projection. The 2D convolution results are restored to the voxel space and fused with the 3D convolution results of the corresponding stage to generate fused features. Finally, a multi-scale voxel feature composed of multiple fused features is obtained. A 2D backbone network 330 is used to extract multi-scale voxel features through two-dimensional convolutional features to generate bird's-eye view features. The region proposal generation module 340 is used to filter out regions where objects to be detected may exist based on the prediction results of the foreground and background features of the bird's-eye view, interact and integrate the features of the regions, further update the global features, filter out the initial candidate boxes based on the prediction results of the foreground and background features of the bird's-eye view, and refine the initial candidate boxes in the decoder using the updated global features to obtain the final target candidate proposal. The ROI network 350 is used to perform feature alignment and extraction of the target candidate proposals using multi-scale voxel features, and finally output the target detection results.

[0057] In addition to the modules described above, the bird's-eye view-guided alignment consistency 3D target detection system 300 may also include other components; however, since these components are not relevant to the embodiments of this disclosure, their illustrations and descriptions are omitted herein. Other specific working processes for image feature-guided 3D target detection using the above-described bird's-eye view-guided alignment consistency 3D target detection system 300 are described in the above-described bird's-eye view-guided alignment consistency 3D target detection method embodiment and will not be repeated here.

[0058] Another embodiment illustrates that the system of the present invention can also be achieved by means of... Figure 4 The architecture of the computing device shown is used to implement this. Figure 4 The architecture of the computing device is shown. For example... Figure 4 As shown, the computer system 410 includes a system bus 430, one or more CPUs 440, input / output 420, and memory 450. Memory 450 can store various data or files used for computer processing and / or communication, as well as program instructions executed by the CPU, including bird's-eye view guided alignment consistency 3D target detection methods. Figure 4 The architecture shown is merely exemplary and should be adjusted according to actual needs when implementing different devices. Figure 4 One or more components are included. The memory 450, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the bird's-eye view-guided aligned consistency 3D target detection method described above in this embodiment of the invention. One or more CPUs 440 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions, and modules stored in the memory 450.

[0059] Of course, the processor of the server provided in the embodiments of the present invention is not limited to performing the method operations described above, but can also perform related operations in the alignment consistency three-dimensional target detection method using the above-described bird's-eye view guidance provided in any embodiment of the present invention.

[0060] The memory 450 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on terminal usage. Furthermore, the memory 450 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 450 may further include memory remotely configured relative to one or more CPUs 440, these remote memories being connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0061] Input / output 420 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Input / output 420 may also include a display device such as a display screen.

[0062] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements the bird's-eye view-guided alignment consistency 3D target detection method described in the above embodiments. The computer-readable storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0063] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0064] The program code contained on the storage medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0065] Furthermore, other specific operational processes of a non-transitory computer-readable storage medium are described in the above-described embodiments of the bird's-eye view-guided alignment consistency 3D target detection method, and will not be repeated here.

[0066] This invention addresses the problem of insufficient utilization of cross-modal features caused by "projection neighborhood mismatch" in multimodal 3D object detection for autonomous driving scenarios. It proposes a feature fusion framework, BVACNet, based on bird's-eye view spatial organization. By uniformly modeling cross-modal correspondences on the BEV plane, a bird's-eye view projection fusion module is introduced to more rationally aggregate and encode features. Furthermore, a centralized contrast consistency loss is used to constrain the fused representation. The model improves feature geometric consistency while suppressing noise interference caused by projection and aggregation, thereby obtaining a more stable and discriminative geometric-semantic joint representation. Experimental results demonstrate that the proposed method achieves leading performance in both 3D and 2D detection metrics.

[0067] To further verify the effectiveness of this invention, Table 1 shows the evaluation metrics (AP_R40@0.70, 0.70, 0.70) of BVACNet on the KITTI dataset. Experimental results show that BVACNet achieves the best overall performance across multiple metrics in both 3D and 2D detection: it achieves the highest overall mAP (89.22) in 3D detection and the highest AP (95.12) on easy difficulty, further improving detection accuracy compared to existing methods; simultaneously, it is on par with the second-best methods on medium and hard difficulty, demonstrating stability and robustness in complex scenarios such as occlusion and sparse point clouds. In 2D detection, BVACNet also maintains a leading position in both overall performance and difficulty levels (overall mAP reaches 97.40), further validating that the proposed method can effectively improve the quality of cross-modal feature fusion, thereby achieving simultaneous improvement in 2D and 3D detection performance.

[0068] Table 1. KITTI dataset detection results (AP_R40@0.70, 0.70, 0.70)

[0069] Table 2 shows the comparison results under the KITTI dataset (AP_R40@0.70, 0.50, 0.50). It can be seen that BVACNet achieves the highest overall mAP in both 3D and 2D detection tasks, and maintains a leading position across most difficulty levels. Overall, these results demonstrate that BVACNet can still consistently improve detection accuracy even with more lenient IoU evaluation settings.

[0070] Table 2. KITTI dataset detection results (AP_R40@0.70, 0.50, 0.50)

[0071] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such a step or method.

[0072] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A bird's-eye view-guided method for aligning and consistent 3D targets, characterized in that, The method includes the following steps: Acquire real point cloud and virtual point cloud data, and perform point cloud voxelization processing on the real point cloud and virtual point cloud data to generate regular voxel mesh data; Regular voxel grid data is input into a 3D backbone network. The 3D backbone network performs multiple downsampling operations on the input data. After each downsampling, the corresponding sparse features are extracted through sparse convolution. The bird's-eye view projection fusion module is then used to perform 3D convolution and 2D convolution operations on the sparse features after the bird's-eye view projection. The 2D convolution results are restored to voxel space and fused with the 3D convolution results of the corresponding stage to generate fused features. Finally, a multi-scale voxel feature composed of multiple fused features is obtained. The multi-scale voxel feature is input into a 2D backbone network for 2D convolution feature extraction to generate bird's-eye view features. The bird's-eye view features are input into the region proposal generation module. Based on the prediction results of the foreground and background features of the bird's-eye view, the module selects regions where objects to be detected may exist, interacts and integrates the features of the regions, and further updates the global features. Based on the prediction results of the foreground and background features of the bird's-eye view, the module selects initial candidate boxes. Using the updated global features, the module refines the initial candidate boxes in the decoder to obtain the final target candidate proposal. The target candidate proposals and multi-scale voxel features are input into the ROI network for feature alignment and extraction, and finally the target detection results are output.

2. The bird's-eye view-guided alignment consistency 3D target detection method according to claim 1, characterized in that, The bird's-eye view projection fusion module splits sparse features into a first feature group and a second feature group. It performs 3D convolution operation on the first feature group to extract 3D spatial geometric features. At the same time, it projects the second feature group onto the bird's-eye view and performs 2D convolution operation to extract 2D semantic features. The 2D semantic features are then backfilled into the original three-dimensional voxel space and stitched together with the 3D spatial geometric features to obtain stitched features. A second 3D convolution operation is performed on the stitched features to output the fused bird's-eye view projection feature map.

3. The bird's-eye view-guided alignment consistency 3D target detection method according to claim 1, characterized in that, The steps for implementing the bird's-eye view projection fusion module include: Obtaining 3D indexes of non-empty voxels and its corresponding eigenvectors ,in For the number of non-empty voxels, Input the number of channels; The feature vector is convolved through the first 3D submanifold. Encode to obtain the first three-dimensional geometric features ; The 3D index is compressed along the height direction to the bird's-eye view plane, and all voxel features projected onto the same plane are aggregated to obtain the deduplicated bird's-eye view index. and their corresponding aggregation features ,in This represents the number of unique bird's-eye view indexes after deduplication. The aggregated features are obtained by 2D submanifold convolution. Encode the geometric features of the bird's-eye view to obtain the geometric features. ; Based on the mapping relationship between the original non-empty voxel 3D index and the deduplicated bird's-eye view index, the bird's-eye view geometric features are backfilled to each non-empty voxel position to obtain bird's-eye view geometric features corresponding one-to-one with each voxel. ; The first three-dimensional geometric features With the aforementioned bird's-eye view geometry By stitching along the channel dimension, the fused features are obtained. ; The fused features are obtained by convolution of a second 3D submanifold. Encode and output the enhanced voxel features. .

4. The bird's-eye view-guided alignment consistency 3D target detection method according to claim 3, characterized in that, Aggregation features This can be achieved in the following ways: Calculate the coordinates of each plane Corresponding features and With voxel count , For the number of non-empty voxels, Indicates the location The corresponding feature vector ; The aggregation feature is obtained by dividing the sum of the number of voxels and the minimum constant by the sum of the features. ,in It is a very small constant.

5. The bird's-eye view-guided alignment consistency 3D target detection method according to claim 1, characterized in that, 3D convolution results With 2D convolution results They are processed centrally separately to obtain , , , , This represents the mean vector within the same batch. and and With indicates the first Characteristics of individual element location; A centralized contrast consistency loss function is constructed to constrain the alignment relationship of cross-modal features in the representation space in the bird's-eye view projection fusion module. The specific expression is as follows: , This represents the number of point clouds in the current batch. It is the weight assigned to the loss.

6. The bird's-eye view-guided alignment consistency 3D target detection method according to claim 1, characterized in that, The method further includes constructing loss functions for the region proposal generation module and the ROI network respectively. The loss function for the region proposal generation module is the sum of the first classification branch loss, the first regression loss, and the contrast loss. The loss function for the ROI network is the sum of the second classification branch loss and the second regression loss. The first classification branch loss uses focus loss to alleviate the imbalance between positive and negative samples. The first regression loss uses smoothing L1 loss to constrain the prediction box parameters and rotation angle error. The second classification branch loss uses binary cross-entropy loss. Smoothing L1 loss is used to optimize the rotation angle and bounding box regression error.

7. The bird's-eye view-guided alignment consistency 3D target detection method according to claim 1, characterized in that, The specific implementation process of the method includes the following steps: The image is fed into a pre-trained virtual point cloud generation network to obtain a virtual point cloud estimated from the image. The virtual point cloud is then stitched together with the real point cloud at the point level to form an expanded point cloud. Perform a voxelization operation on the expanded point cloud to obtain a voxel representation; In the voxel domain, the input convolutional block is used to encode the voxel representation to obtain the initial sparse voxel features; The initial sparse voxel features are sequentially input into multiple projection convolutional blocks for multi-scale feature extraction. Each projection convolutional block consists of a downsampling layer, two sparse convolutional blocks, and a bird's-eye view projection fusion module, outputting features at the corresponding scale. , Indicates the number of projected convolutional blocks; Features of the highest level Extract bird's-eye view features from a 2D backbone network; The bird's-eye view features are input into the region proposal generation module, and the target candidate proposals are obtained after processing. Multi-scale voxel features are used to aggregate features and refine candidate boxes for target candidate proposals, resulting in the final detection results.

8. A bird's-eye view-guided alignment consistency 3D target detection system, characterized in that, The system includes: The data input module is used to acquire real point cloud and virtual point cloud data, and to perform point cloud voxelization processing on the real point cloud and virtual point cloud data to generate regular voxel mesh data. A 3D backbone network is used to perform multiple downsampling operations on the input data. After each downsampling, the corresponding sparse features are extracted through sparse convolution. The bird's-eye view projection fusion module is used to perform 3D convolution and 2D convolution operations on the sparse features after the bird's-eye view projection. The 2D convolution results are restored to the voxel space and fused with the 3D convolution results of the corresponding stage to generate fused features. Finally, a multi-scale voxel feature composed of multiple fused features is obtained. A 2D backbone network is used to extract multi-scale voxel features through two-dimensional convolutional features to generate bird's-eye view features. The region proposal generation module is used to filter out regions where objects to be detected may exist based on the prediction results of the foreground and background features of the bird's-eye view. It interacts and integrates the features of the regions, further updates the global features, filters out initial candidate boxes based on the prediction results of the foreground and background features of the bird's-eye view, and refines the initial candidate boxes in the decoder using the updated global features to obtain the final target candidate proposal. The ROI network is used to perform feature alignment and extraction of the target candidate proposals using multi-scale voxel features, and finally output the target detection results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the bird's-eye view-guided aligned and consistent 3D target detection method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, wherein computer instructions are stored on the storage medium, characterized in that, When the instructions are executed by the processor, they implement the steps of the bird's-eye view-guided aligned and consistent 3D target detection method as described in any one of claims 1 to 7.

Images