Calibration-free three-dimensional detection method based on visual angle guidance and axial attention

Abstract

The invention relates to a calibration-free three-dimensional detection method based on visual angle guidance and axial attention, and belongs to the technical field of three-dimensional target detection. The method comprises the following steps: extracting image features of a collected single-frame image through a ResNet18 backbone network; the image features are decoupled into front view features and bird's-eye view features through a feature decoder, binary classification segmentation supervision is applied to the bird's-eye view features, and multi-class foreground segmentation supervision is applied to the front view features; the height dimension of the front view is reserved, a global feature corresponding relation is established between front view features and aerial view features through a cross-view-angle axial attention module, and object contour features in the front view features are fused into the aerial view features to enhance semantic features of the aerial view features; and inputting the fused aerial view features into a three-dimensional detection head, and outputting the following three types of three-dimensional detection results. According to the method, the practicability and accuracy of calibration-free three-dimensional target detection are effectively improved, and the method has wide engineering popularization value and commercial application prospects.

Description

A calibration-free 3D detection method based on viewpoint guidance and axial attention Technical Field

[0001] This invention relates to a calibration-free 3D detection method based on viewpoint guidance and axial attention, belonging to the field of 3D target detection technology. Background Technology

[0002] Monocular 3D object detection aims to recover the 3D spatial position and size of objects from a single 2D image, and has significant application value in autonomous driving, robot navigation, and related fields. Compared with traditional 2D detection, monocular 3D detection can provide more accurate environmental perception. However, existing 3D detection methods mainly focus on vehicle-centric scenes, and vehicle-mounted cameras require precise factory calibration and fixed installation angles. In contrast, roadside cameras are usually installed at higher positions, with a wider field of view, capable of capturing richer scene information and covering the blind spots of vehicle-centric vision systems. However, these advantages also bring two key challenges: (1) a significant increase in target density; and (2) greater susceptibility to changes in camera calibration parameters. Therefore, many algorithms designed for vehicle-centric scenes have limited effectiveness when applied to roadside perception environments, and there is an urgent need to develop high-precision 3D detection methods under uncalibrated or weakly calibrated conditions.

[0003] A Chinese patent (CN202411890586.3) proposes a multimodal 3D target detection method based on an attention mechanism. This method utilizes feature extraction from images and LiDAR point clouds, along with a bidirectional information interaction fusion strategy, to enhance feature expressiveness and redundancy suppression. The method performs refined processing within the region of interest, achieving multimodal target identification and localization through candidate box generation, classification, and regression. Finally, it combines the non-maximum suppression (NMS) algorithm to output high-confidence detection results, thus achieving significant improvements in both fusion efficiency and detection accuracy.

[0004] Another Chinese patent (CN202510016818.0) proposes a memory-efficient 3D target detection method based on bird's-eye view representation. By introducing a reversible backbone network and a forward-backward view transformation module, it significantly reduces the memory overhead during image BEV feature extraction. This method utilizes panoramic RGB images and laser point clouds to construct BEV features for the image and point clouds respectively, then performs feature fusion on the BEV plane, and achieves 3D object detection through an encoder and decoder structure. The forward-backward transformation module enhances the representation capability of the forward view projection, thereby further compressing memory usage while maintaining accuracy, making it suitable for resource-constrained scenarios.

[0005] While these methods have made progress in the accuracy and speed of 3D object detection, they all lack the dependence of the detection model on camera calibration parameters. Once the camera parameters of the model are perturbed or incorrect, the detection performance of the model will suffer a catastrophic decline. To fill this research gap, this invention proposes a calibration-free 3D detection method. Summary of the Invention

[0006] The calibration-free 3D detection method based on viewpoint guidance and axial attention adopted in this invention solves the problem that in actual road deployment, due to the diverse installation positions and unfixed postures of roadside cameras, and their susceptibility to external factors (such as wind and snow), the calibration process is difficult and prone to failure, which seriously affects the detection accuracy and robustness.

[0007] The present invention adopts the following technical solution:

[0008] The present invention discloses a calibration-free 3D detection method based on viewpoint guidance and axial attention.

[0009] S1: Use a monocular camera installed on the roadside to capture single-frame images, and extract image features from the captured single-frame images through the ResNet18 backbone network.

[0010] S2: The image features obtained in step S1 are decoupled into front view features and bird's-eye view (BEV) features by the feature decoder, and an auxiliary supervision task is introduced to improve the decoupling effect;

[0011] S3: Divide the front view features and bird's-eye view features into 256×256 grids respectively; apply binary classification segmentation supervision to the bird's-eye view features and multi-class foreground segmentation supervision to the front view features;

[0012] S4: Retain the height dimension of the front view, and establish a global feature correspondence between the front view features and the bird's-eye view features through the cross-view axis attention module. Integrate the object outline features in the front view features into the bird's-eye view features to enhance the semantic features of the bird's-eye view features.

[0013] S5: Input the fused bird's-eye view features into the 3D detection head, and use a three-layer convolutional network regression structure to output the following three types of 3D detection results.

[0014] In the calibration-free 3D detection method based on viewpoint guidance and axial attention described in this invention, the input size of the acquired single-frame image is set to 1024×1024 pixels in step S1.

[0015] The acquired raw image can be represented as a three-dimensional tensor:

[0016]

[0017] Where B represents the batch size, 3 represents the number of RGB channels, H=1024, W=1024;

[0018] After extraction by the ResNet18 backbone network, the image is converted into a high-dimensional feature map:

[0019]

[0020] Where H′=64, W′=64 are the dimensions of the original image after downsampling, and 512 is the channel dimension.

[0021] The calibration-free 3D detection method based on viewpoint guidance and axial attention described in this invention

[0022] In step S2, the front view features and the bird's-eye view features are generated into front view features using separate decoders. Bird's-eye view features ;

[0023] Decoder embedding front view perspective embedding value Embedded values ​​with bird's-eye view Mapping global semantic features back to spatial resolution is represented as follows:

[0024]

[0025] The resulting feature maps satisfy the following conditions: .

[0026] The calibration-free 3D detection method based on viewpoint guidance and axial attention described in this invention, in step S3, employs a foreground / background binary classification segmentation strategy for the supervision of the bird's-eye view features.

[0027] The bird's-eye view features are supervised using the following binary classification loss (supervision of bird's-eye view features, supervision of front-view features):

[0028]

[0029] in, The model predicts the probability that an object exists in each cell, where N is the total number of cells.

[0030] The multi-class foreground segmentation supervision mechanism based on foreground view features extends traditional binary classification to multi-class overlapping encoding. The label of each foreground view grid is defined as the number of targets at that location, expressed as follows:

[0031]

[0032] in, Indicates the first The number of overlapping foreground objects in each grid. The maximum number of categories is set.

[0033] To adapt to model training, all label maps were uniformly resampled to a size of 256×256 to maintain consistency with the feature map spatial resolution, and the cross-entropy loss function was used for supervision.

[0034]

[0035] The calibration-free 3D detection method based on viewpoint guidance and axial attention described in this invention includes the following step in step S4, which involves cross-viewpoint information fusion via a cross-viewpoint axial attention module:

[0036] Front view features and bird's-eye view features Expanding along the spatial dimension into a sequence representation yields the result, achieving a dimensionality reduction from 4 to 3 dimensions for the feature dimension. , in, ;

[0037] Linear projections are performed on the bird's-eye view features and the front view features respectively to generate three sets of inputs for the attention matrix:

[0038]

[0039] in, The projective weight matrix is ​​a learnable matrix.

[0040] Construct a multi-head axial attention mechanism and obtain the attention output through the following calculation:

[0041]

[0042]

[0043]

[0044] The multi-head attention outputs are concatenated and processed through residual connections and normalization to finally output fused features:

[0045]

[0046] LayerNorm represents the layer normalization operation, used for stable training;

[0047] After processing by the cross-view axis attention module, a BEV feature map fused with front view semantic information is obtained, denoted as:

[0048]

[0049] The calibration-free 3D detection method based on viewpoint guidance and axial attention described in this invention uses a three-layer convolutional network regression structure in step S5 to output the following three types of 3D detection results:

[0050] The x, y, z coordinates of the center point of the target location;

[0051] The target's length, width, and height dimensions;

[0052] The target's y-angle;

[0053] Target multiple classification categories;

[0054] Training is performed using a multi-task loss function. The target's position and size are lossed using L1 loss, the target's orientation angle using Multi-Bin loss, the target category using Focal Loss loss, and the auxiliary segmentation map loss function using CrossEntropy loss. The combined loss function is as follows:

[0055]

[0056] in, These are adjustable weighting coefficients.

[0057] Beneficial effects

[0058] Compared with the prior art, the present invention has the following significant advantages:

[0059] Completely eliminates reliance on camera calibration: It does not use any camera intrinsic or extrinsic parameters, making it suitable for roadside camera installation environments with different orientations, heights, and angles, significantly reducing deployment costs.

[0060] Enhanced target perception in occluded scenes: The multi-category front view segmentation module effectively preserves fine-grained boundary information of multiple overlapping targets, which is significantly better than the traditional foreground / background segmentation strategy, and is particularly suitable for dense traffic scenes.

[0061] Preserving height information and achieving global alignment: Introducing a cross-view axis attention mechanism to achieve semantic alignment between the front view and the bird's-eye view while preserving the height dimension and improving the accuracy of 3D spatial modeling.

[0062] Improved detection accuracy: On large-scale real-world 3D target detection datasets such as DAIR-V2X-I and DAIR-V2X-C, the method of this invention achieves an accuracy improvement of 1.0%–2.5% on metrics such as AP3D (IoU=0.5) and APBEV compared to the existing calibration-free method CBR, verifying the effectiveness and advancement of the method.

[0063] Computational efficiency compatible with practical deployment: By adopting a lightweight network structure (such as the ResNet18 backbone) and combining it with an attention mechanism optimization scheme, it can run on actual edge devices and meet real-time requirements.

[0064] In summary, this invention effectively improves the practicality and accuracy of calibration-free 3D target detection, and has broad engineering promotion value and commercial application prospects. Attached Figure Description

[0065] Figure 1. Flowchart of the calibration-free 3D detection method based on viewpoint guidance and axial attention of the present invention;

[0066] Figure 2. Schematic diagram of foreground segmentation construction in the calibration-free 3D detection method based on viewpoint guidance and axial attention of the present invention;

[0067] Figure 3 is a schematic diagram of the axis cross attention mechanism in the calibration-free 3D detection method based on view guidance and axial attention of the present invention;

[0068] Figure 4. Schematic diagram of the three-dimensional detection frame in the calibration-free three-dimensional detection method based on view guidance and axial attention of the present invention;

[0069] Figure 5. Schematic diagram of model detection results in the calibration-free 3D detection method based on viewpoint guidance and axial attention of the present invention;

[0070] Figure 6. A schematic diagram of the experimental results of the calibration-free 3D detection method based on view guidance and axial attention of the present invention. Detailed Implementation

[0071] To make the objectives and technical solutions of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0072] The present invention discloses a calibration-free 3D detection method based on viewpoint guidance and axial attention.

[0073] S1: Use a monocular camera installed on the roadside to capture single-frame images, and extract image features from the captured single-frame images through the ResNet18 backbone network.

[0074] The input size of the acquired single-frame image is set to 1024×1024 pixels;

[0075] The acquired raw image can be represented as a three-dimensional tensor:

[0076]

[0077] Where B represents the batch size, 3 represents the number of RGB channels, H=1024, W=1024;

[0078] After extraction by the ResNet18 backbone network, the image is converted into a high-dimensional feature map:

[0079]

[0080] Where H′=64, W′=64 are the dimensions of the original image after downsampling, and 512 is the channel dimension.

[0081] S2: Generate front view features from front view features and bird's-eye view features using separate decoders. Bird's-eye view features ; Decoder embedding front view perspective embedding value Embedded values ​​with bird's-eye view Mapping global semantic features back to spatial resolution is represented as follows:

[0082]

[0083] The resulting feature maps satisfy the following conditions: .

[0084] S3: Divide the front view features and bird's-eye view features into 256×256 grids respectively; supervise the binary classification segmentation of the bird's-eye view features, using a foreground / background binary classification segmentation strategy.

[0085] The bird's-eye view features are supervised using the following binary classification loss (supervision of bird's-eye view features, supervision of front-view features):

[0086]

[0087] in, The model predicts the probability that an object exists in each cell, where N is the total number of cells.

[0088] The multi-class foreground segmentation supervision mechanism based on foreground view features extends traditional binary classification to multi-class overlapping encoding. The label of each foreground view grid is defined as the number of targets at that location, expressed as follows:

[0089]

[0090] in, Indicates the first The number of overlapping foreground objects in each grid. The maximum number of categories is set.

[0091] To adapt to model training, all label maps were uniformly resampled to a size of 256×256 to maintain consistency with the feature map spatial resolution, and the cross-entropy loss function was used for supervision.

[0092] .

[0093] S4: Features of the front view and bird's-eye view features Expanding along the spatial dimension into a sequence representation yields the result, achieving a dimensionality reduction from 4 to 3 dimensions for the feature dimension. , in, ;

[0094] Linear projections are performed on the bird's-eye view features and the front view features respectively to generate three sets of inputs for the attention matrix:

[0095]

[0096] in, The projective weight matrix is ​​a learnable matrix.

[0097] Construct a multi-head axial attention mechanism and obtain the attention output through the following calculation:

[0098]

[0099]

[0100]

[0101] The multi-head attention outputs are concatenated and processed through residual connections and normalization to finally output fused features:

[0102]

[0103] LayerNorm represents the layer normalization operation, used for stable training;

[0104] After processing by the cross-view axis attention module, a BEV feature map fused with front view semantic information is obtained, denoted as:

[0105]

[0106] S5: Input the fused bird's-eye view features into the 3D detection head. The detection head uses a three-layer convolutional network regression structure and outputs the following three types of 3D detection results:

[0107] The x, y, z coordinates of the center point of the target location;

[0108] The target's length, width, and height dimensions;

[0109] The target's y-angle;

[0110] Target multiple classification categories;

[0111] Training is performed using a multi-task loss function. The target's position and size are lossed using L1 loss, the target's orientation angle using Multi-Bin loss, the target category using Focal Loss loss, and the auxiliary segmentation map loss function using CrossEntropy loss. The combined loss function is as follows:

[0112]

[0113] in, These are adjustable weighting coefficients.

[0114] Example 1

[0115] An embodiment of a calibration-free 3D detection method based on viewpoint guidance and axial attention provided by this invention:

[0116] First, input image The input is fed into the feature extraction network. The backbone network uses a ResNet18 structure to extract multi-level semantic features.

[0117] The network outputs feature maps at four scales, namely:

[0118] , ,

[0119] Features from each layer are resampled and standardized to a size of 32×32 for subsequent feature fusion.

[0120] Next, the resampled multi-layer feature input spatial projection module. This module maps image features to a specific viewpoint space through learnable transformations, generating preliminary viewpoint feature representations and providing a foundation for feature generation of the front view or bird's-eye view.

[0121] Subsequently, the features are fed into the decoder for layer-by-layer upsampling. The decoder outputs a higher-resolution feature map at each stage until a feature representation of size [B, 64, H / 4, W / 4] is obtained. Learnable viewpoint tokens are introduced during the decoding process, with each viewpoint corresponding to an independent token. These tokens interact with the decoding layer features to dynamically adjust the spatial distribution and are fused with the feature map at the end. Simultaneously, the front view branch employs a multi-classification strategy for supervision to enhance the ability to distinguish occluded scenes and strengthen feature decoupling.

[0122] After completing the viewpoint feature decoding, the front view features are obtained respectively. and bird's-eye view features The two feature inputs are fused into a cross-view feature matching module. This module first calculates global dependencies along the spatial dimension through a cross-view axial attention mechanism, realizing information complementarity and feature interaction between different perspectives, and generating a fused BEV representation.

[0123] Finally, the fused features are input into the 3D detection head for prediction. The detection head includes classification and regression branches, which output information such as the target's category, 3D coordinates, size parameters, and orientation angle.

[0124] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A calibration-free 3D detection method based on viewpoint guidance and axial attention, characterized in that, S1: Use a monocular camera installed on the roadside to capture single-frame images, and extract image features from the captured single-frame images through the ResNet18 backbone network. S2: Decouple the image features obtained in step S1 into front view features and bird's-eye view features using the feature decoder. S3: Divide the front view features and bird's-eye view features into 256×256 grids respectively; apply binary classification segmentation supervision to the bird's-eye view features and apply multi-class foreground segmentation supervision to the front view features. S4: Retain the height dimension of the front view. Through the cross-view axis attention module, establish a global feature correspondence between the front view features and the bird's-eye view features. Integrate the object contour features in the front view features into the bird's-eye view features to enhance the semantic features of the bird's-eye view features. S5: Input the fused bird's-eye view features into the 3D detection head. Use a three-layer convolutional network regression structure to output the following three types of 3D detection results.

2. The calibration-free 3D detection method based on viewpoint guidance and axial attention as described in claim 1, characterized in that, In step S1, the input size of the acquired single-frame image is set to 1024×1024 pixels; the acquired original image can be represented as a three-dimensional tensor: Where B represents the batch size, 3 represents the number of RGB channels, H=1024, W=1024; after extraction by the ResNet18 backbone network, the image is converted into a high-dimensional feature map: Where H′=64, W′=64 are the dimensions of the original image after downsampling, and 512 is the channel dimension.

3. The calibration-free 3D detection method based on viewpoint guidance and axial attention according to claim 1, characterized in that, In step S2, the front view features and the bird's-eye view features are generated into front view features using separate decoders. Bird's-eye view features ； Decoder embedding front view perspective embedding value Embedded values ​​with bird's-eye view Mapping global semantic features back to spatial resolution is represented as follows: The resulting feature maps respectively satisfy: 。 4. The calibration-free 3D detection method based on viewpoint guidance and axial attention as described in claim 1, characterized in that, In step S3, the binary classification segmentation supervision of the bird's-eye view features adopts a foreground / background binary classification segmentation strategy, and the bird's-eye view features are supervised using the following binary classification loss: ;in, The model predicts the probability of an object existing in each grid cell, where N is the total number of grid cells. A multi-class foreground segmentation supervision mechanism for the front view features extends traditional binary classification to multi-class overlapping encoding. The label for each front view grid cell is defined as the number of targets at that location, expressed as follows: ;in, Indicates the first The number of overlapping foreground objects in each grid. The maximum number of categories is set; all label images are uniformly resampled to a size of 256×256 to maintain consistency with the feature map spatial resolution, and the cross-entropy loss function is used for bird's-eye view feature supervision. The expression is as follows: 。 5. The calibration-free 3D detection method based on viewpoint guidance and axial attention according to claim 1, characterized in that, The S4 step, which involves cross-view axial attention module to achieve cross-view information fusion, is as follows: Front view features... and bird's-eye view features Expanding along the spatial dimension into a sequence representation yields the result, achieving a dimensionality reduction from 4 to 3 dimensions for the feature dimension. , in, Linear projections are performed on the bird's-eye view features and the front view features respectively to generate three sets of inputs for the attention matrix: ;in, Given a learnable projection weight matrix; construct a multi-head axial attention mechanism, and obtain the attention output through the following calculation: ； ； The multi-head attention outputs are concatenated and processed through residual connections and normalization to finally output fused features. Where LayerNorm represents the layer normalization operation, used to stabilize training; after processing by the cross-view axis attention module, the BEV feature map fused with front view semantic information is obtained, denoted as: 。 6. The calibration-free 3D detection method based on viewpoint guidance and axial attention according to claim 1, characterized in that, In step S5, the detection head uses a three-layer convolutional network regression structure to output the following three types of 3D detection results: x, y, z coordinates of the target's center point; the target's length, width, and height; the target's ry-orientation angle; and the target's multi-class classification. A multi-task loss function is used for training: L1 loss is used for the target's position and size, Multi-Bin loss is used for the target's orientation angle, Focal Loss is used for the target's class, and CrossEntropy loss is used for the auxiliary segmentation map. The combined results are as follows: ;in, These are adjustable weighting coefficients.

Images