A monocular 3D detection method based on analog three-view and geometric relation reasoning

Abstract

This invention discloses a monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning in the field of computer vision technology. The method includes: extracting multi-scale forward-looking features from a monocular image; decoupling the three-view space by mapping the multi-scale forward-looking features to simulated bird's-eye view space and side view space, generating bird's-eye view feature maps and side view feature maps, and finally generating bird's-eye view center probability priors and side view center probability priors; introducing a geometry-guided refinement module, first performing node gating on the query vector based on the bird's-eye view and side view center probability priors, then constructing a fully connected graph based on the 3D center distance predicted by the query vector for message passing and feature updating; finally, inputting the query vector into the detection head to obtain the 3D detection result. This invention effectively improves the accuracy and robustness of monocular 3D detection in occluded and dense scenes by simulating orthogonal view decoupling features and combining geometric relationship graphs for reasoning.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning. Background Technology

[0002] 3D object detection is one of the core technologies in fields such as autonomous driving, robot navigation, and augmented reality. Compared with solutions that rely on costly LiDAR or stereo vision, 3D object detection methods based on monocular images have the advantages of low cost and simple deployment, thus becoming a research hotspot in academia and industry.

[0003] However, monocular 3D detection faces a serious projection blur problem, meaning that recovering 3D information from a single 2D image is an ill-posed problem. The main challenges include: 1) Depth information uncertainty: Monocular images lose depth information, making it difficult to accurately estimate the position of objects in 3D space; 2) Occlusion and truncation: In dense scenes, target objects occlude each other or partially move out of the image, resulting in low perception accuracy for densely arranged targets; 3) Perspective distortion: The near-large-far-small phenomenon in images causes the size and position of targets to be highly coupled in image space, increasing the difficulty of regression.

[0004] To alleviate these problems, existing technologies attempt to improve performance through multi-view fusion and depth estimation optimization. However, most of these methods require real multi-view image input or only improve the depth regression module. They do not address the depth-size coupling and positioning drift issues caused by perspective distortion at the spatial geometric representation level, nor do they fully model the spatial dependencies between objects. Therefore, performance improvements are limited in occluded or densely packed target scenes. Summary of the Invention

[0005] The purpose of this invention is to provide a monocular 3D detection method based on simulated three-view and geometric relationship reasoning, which solves the technical problems of lack of three-dimensional geometric priors, sparse depth information, and unstable localization in occluded scenes. By simulating multi-view geometric reasoning with a single image, a decoupled three-dimensional geometric representation is constructed, and the spatial relationship between objects is modeled to achieve feature refinement, thereby improving the detection accuracy and robustness in occluded and dense target scenes.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: A monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning constructs the TriViewNet detection framework. The TriViewNet framework consists of multi-scale feature extraction, three-view spatial decoupling, and geometrically guided feature refinement, and includes the following steps: S1. Multi-scale feature extraction: Obtain the monocular input image, extract multi-scale front-view feature maps through the backbone network, flatten them into token sequences, add position encoding and hierarchical embedding, and then concatenate them. The encoded visual tokens are obtained by aggregation through a three-view encoder. S2. Three-view space decoupling: The multi-scale front view feature map is mapped to the simulated bird's-eye view space and side view space through geometric projection and cross attention mechanism, respectively, to generate bird's-eye view feature map and side view feature map. Based on the bird's-eye view feature map and side view feature map, bird's-eye view center probability prior and side view center probability prior are generated respectively. S3. Geometric Graph-Guided Feature Refinement: The multi-scale front-view feature map and the preset decoder query vector are input into the decoder, and the query vector is iteratively updated. During the decoding process, the query vector is enhanced by the geometric graph-guided refinement module G²R based on the bird's-eye view center probability prior, the side-view center probability prior, and the spatial relationship between the query vectors to obtain the refined query vector. The refined query vector is input into the detection head, and the three-dimensional detection result of the target is output.

[0007] Furthermore, in step S2, the multi-scale forward-looking feature map is mapped to the simulated bird's-eye view space and side-view view space through geometric projection and cross-attention mechanism, respectively, to generate the bird's-eye view feature map and the side-view feature map. Specifically, this includes: Create regular geometric grids for the bird's-eye view and the side view respectively; The coordinates of each grid point are encoded using location encoding and combined with a learnable token to generate a corresponding grid query vector; The multi-scale forward-looking feature map is compressed to obtain the compressed geometric subspace features. Using the grid query vector as the query and the geometric subspace features as the key and value, a cross-attention operation is performed to aggregate the bird's-eye view feature map and the side view feature map, respectively.

[0008] Furthermore, the generation of the side-view center probability prior in S2 specifically includes: inputting the side-view feature map into a side-view detection head with K output channels to generate a K-channel side-view heatmap, and obtaining the side-view center likelihood prior of K slices divided along the camera X-axis through Sigmoid activation; calculating the slice index using the 3D-annotated target center coordinates, drawing center supervision only in the corresponding slice channel with a Gaussian kernel, and training with pixel-wise BCE loss.

[0009] Furthermore, the slice index The calculation formula is:

[0010] in, The X-axis coordinates of the target center in 3D annotation. , The range of X-axis coordinates, and K is the number of slices.

[0011] Furthermore, the enhancement of the query vector by the geometry-guided refinement module in S3 includes the following steps: Node gating steps: For each query vector, based on its currently predicted 3D center, upsample the bird's-eye view center probability prior and the side view center probability prior to obtain the corresponding confidence level, and calculate the gating coefficient based on the confidence level. Use the gating coefficient to weight the query vector to obtain the gated query vector. Graph propagation steps: Treat all gated query vectors as nodes in the graph, calculate edge weights based on the 3D spatial distance between nodes, and use a message passing mechanism to enable each node to aggregate the features of its neighboring nodes and update its own features.

[0012] Furthermore, in the node gating step, the formula for calculating the gating coefficient is as follows:

[0013] in, Indicates the first Features of each query vector; and Control the relative contributions of the bird's-eye view center probability prior and the side view center probability prior; It is a stabilization gating coefficient used to scale the query vector; This is used to avoid setting the features of low-confidence nodes to zero.

[0014] Furthermore, in the graph propagation step, the formula for calculating edge weights based on the 3D spatial distance between nodes is as follows: Define the distance kernel :

[0015] in, For nodes With nodes The 3D center, Represents the Euclidean norm; This is the distance attenuation coefficient; The normalized adjacency matrix is:

[0016] in, It is a numerically stable term; Message aggregation and residual updates are as follows:

[0017] in, This is the gated query vector. To control the strength of message injection.

[0018] Furthermore, a monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning also includes end-to-end multi-task joint optimization, with the overall objective function being:

[0019] in, To monitor the loss of the bird's-eye view branch target center probability map, For the Side branch, use the target center probabilistic graph loss. For classifying losses, For size regression loss, The orientation angle regression loss is used.

[0020] By adopting the above technical solution, the present invention has the following advantages: 1. This invention provides a monocular 3D detection method based on simulated three-view and geometric relationship reasoning. Under monocular image conditions, it simulates multi-view geometric reasoning, inversely projects the front view features into orthogonal view features that are complementary to bird's-eye and side views, and establishes a globally consistent three-dimensional geometric representation. This effectively alleviates the depth-size coupling and positioning drift problems caused by perspective distortion and solves the problem of insufficient geometric priors in monocular 3D detection.

[0021] 2. This invention provides a monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning. The designed geometry graph-guided refinement module G²R uses the decoder query vector as graph nodes, combines bird's-eye view and side view geometric priors for node gating, and constructs a weighted geometric graph based on 3D distance. Feature refinement is achieved through spatial relationship reasoning, which strengthens high-confidence candidate features with geometric consistency and spatial proximity, suppresses outliers, and significantly improves the consistency and robustness of 3D localization in occluded, target-dense, and depth cue-sparse scenes.

[0022] 3. This invention provides a monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning. The overall objective function of the multi-task joint optimization is constructed, and dense geometric supervision of bird's-eye view and side view center probability maps is introduced. Combined with multi-task supervision of category, size, and orientation angle, the end-to-end optimization of the network is realized, ensuring the accuracy of the detection results.

[0023] 4. This invention provides a monocular 3D detection method based on simulated three-view and geometric relationship reasoning. The TriViewNet framework has achieved state-of-the-art performance on mainstream monocular 3D detection datasets such as KITTI and Rope3D. Compared with existing methods, it has achieved stable improvements in AP3D and APbev metrics, especially in challenging scenarios with medium to long distances, occlusion, and dense targets, and has practical engineering application value. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall structure of the TriViewNet detection framework of the present invention; Figure 2 This is a bird's-eye view structural diagram of the branch of the present invention; Figure 3 This is a schematic diagram of the side view of the branch structure of the present invention; Figure 4 This is a schematic diagram of the geometry-guided refinement (G²R) module of the present invention; Figure 5 This is a simulated orthogonal view feature map extracted by TVSD in this invention; Figure 6 This is a side view of the K-slice prediction head heatmap of the present invention; Figure 7a This is a comparison chart of the detection performance of the present invention and existing methods on the KITTI validation set in a certain type of scenario; Figure 7b This is a comparison chart of the detection performance of the present invention and existing methods on the KITTI validation set in another type of scenario; Figure 8 This is a comparison chart of the bird's-eye view detection performance of the present invention and existing methods on the KITTI validation set. Detailed Implementation

[0025] The technical solution of the present invention will be specifically described below with reference to the accompanying drawings. It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0026] This invention provides a monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning, constructing the TriViewNet detection framework. The TriViewNet detection framework consists of multi-scale feature extraction, three-view spatial decoupling, and geometric graph-guided feature refinement, specifically as follows: Figure 1 As shown, the process steps are as follows: Given an input image, the backbone network first extracts multi-scale features and inputs them into the three-view space decoupling branch. In order to simulate multi-view features from monocular images, the three-view space decoupling part first designs a bird's-eye view and a side view mapping mesh to achieve feature space alignment of the three views. Then, through cross-attention, it aggregates bird's-eye view simulated view features and side view simulated view features from monocular features respectively. Bird's-eye view detection head and side view detection head are introduced to predict the corresponding view features and output the bird's-eye view and side view center probability priors to provide geometric consistency guidance for target queries in the decoding stage. In the decoding prediction stage, the decoder, together with the multi-scale features output by the encoder, uses the deep embedding prior generated by the depth branch as a condition. The geometry graph-guided refinement (G²R) module treats the query vector nodes as graph nodes and constructs a three-dimensional fully connected graph based on geometric distance, thereby injecting geometric context between targets into each query vector. Finally, the detection head outputs the category and three-dimensional parameters, including three-dimensional center, size, and orientation. The specific process includes the following steps: S1. Multi-scale Feature Extraction: A monocular input image is acquired, and multi-scale front-view feature maps are extracted using a backbone network. These maps are flattened into token sequences, and positional encoding and hierarchical embedding are added before concatenation. The resulting sequence is then aggregated by a three-view encoder to obtain encoded visual tokens. Specifically, to explicitly construct a simulated multi-view geometric representation under monocular input, the front-view visual features are converted into two geometric branches: a bird's-eye view and a side view. The backbone network first extracts multi-scale features. Each layer of features is flattened into a token sequence, and positional encoding and hierarchical embedding are added before concatenation into a unified sequence. Subsequently, a three-view encoder based on MSDeformAttn is used to aggregate the cross-scale information, obtaining the encoded visual token representation, which serves as the input feature source for subsequent modules. To reduce the computational cost of the geometric branches, only the lowest-resolution encoded token is used, along with corresponding masks to filter invalid positions. Simultaneously, a linear layer maps its channel count to a smaller geometric branch dimension for more efficient subsequent computation.

[0027] S2. Three-view space decoupling: The multi-scale front view feature map is mapped to the simulated bird's-eye view space and side view space through geometric projection and cross attention mechanism, respectively, to generate bird's-eye view feature map and side view feature map. Based on the bird's-eye view feature map and side view feature map, the bird's-eye view center probability prior and the side view center probability prior are generated respectively. Specifically, the multi-scale forward-looking feature maps are mapped to simulated bird's-eye view space and side-view view space through geometric projection and cross-attention mechanisms to generate bird's-eye view feature maps and side-view feature maps. In the geometry view generation stage, regular geometric meshes are constructed for both the bird's-eye view and the side view. This step is crucial because it involves the alignment of the geometric space. The coordinates of each mesh location are then input into the MLP to obtain the corresponding mesh query. The coordinates of each grid point are encoded using location encoding and combined with a learnable token to generate a corresponding grid query vector; The multi-scale front-look feature map is compressed to obtain the compressed geometric subspace features. Using the grid query vector as the query and the geometric subspace features as the key and value, a cross-attention operation is performed to aggregate the bird's-eye view feature map and the side view feature map respectively.

[0028] Furthermore, based on the obtained bird's-eye view feature map and side view feature map, the features are respectively connected to the bird's-eye view detection head and the side view detection head, outputting the corresponding geometric prior probability maps. These maps are then trained using the labeled bird's-eye view and side view supervision signals, allowing the bird's-eye view branch to focus more on the distribution consistency on the ground plane, and the side view branch to focus more on the structural consistency of the sides. Finally, the features obtained from the bird's-eye view and side view branches are mapped back to the channel dimension of the main branch, facilitating fusion with the decoder's query features or for subsequent geometric consistency enhancement. It is worth noting that the side view exhibits a certain coupling relationship along the Z-axis; therefore, decoupling is implemented in the design of the side view detection head. Simultaneously, TVSD maintains a forward-looking depth branch in parallel. This branch structure follows the depth branch method of the baseline Monodtr, providing geometric information to the final decoder.

[0029] Bird's-eye view branch: Due to issues such as occlusion, truncation, and projection dimension reduction in monocular view, target feature extraction from monocular images is severely affected. Therefore, the goal of the bird's-eye view branch is to extract bird's-eye features from the TriView Encoder feature store to reduce coupling between target features and provide grid-aligned bird's-eye-eye prior information for downstream target prediction. The specific structure diagram of the bird's-eye view branch is as follows: Figure 2 As shown, " ”This represents element-wise addition, and the specific steps are as follows: First, after the backbone network extracts the front view features from the monocular RGB image, it inputs them into the three-view encoder to further extract target features. At this point, the feature storage still contains front view features. Therefore, in order to obtain the bird's-eye view features in the image space, it is necessary to construct a feature projection alignment grid to ensure that the features remain aligned in space after projection. Next, the features are flattened and learnable tokens are constructed. A coordinate conditional grid query is then constructed using a multilayer perceptron (MLP). Here, i refers to the position of the i-th grid in the bird's-eye view grid. , : No. A bird's-eye view grid query vector, No. A learnable bird's-eye view token. : No. The coordinates of a bird's-eye view grid point camera coordinate system.

[0030]

[0031] Perform cross-attention on the bird's-eye view grid to converge image memories onto the geometric grid.

[0032] To reduce the computational overhead of attention, we first... Projected onto a compact geometric subspace. The coarse-scale image memory tokens selected for the TriViewEncoder feature storage. For linear projection, For the memory of the projected geometric subspace. : Token sequence length, Cg: Channel dimension of the geometric subspace

[0033] Using this as the key and value, the aggregated output features of the bird's-eye view branch are obtained. :

[0034] in: , This represents the transpose of the Mg matrix.

[0035] Based on this, the bird's-eye view prediction head outputs the center likelihood prior: , This indicates a bird's-eye view of the inspection head.

[0036] And use it as a probability map of geometry perception for the decoder's G. 2 R-node and edge construction.

[0037] : Bird's-eye view feature map obtained from the bird's-eye view branch (usually by The mesh is reshaped back to a 2D grid and then subjected to several layers of convolution / FFN. Number of channels in the bird's-eye view feature map (not necessarily equal to) (It may have been mapped back to the main dimension). (The text says) ): Logits of the center heatmap output by the bird's-eye view inspection head. The output is commonly as follows: or (Depending on whether slice / channel decoupling is also performed). Sigmoid, converts logits to The central probability. The bird's-eye view of the center's likelihood prior is used as a geometrically perceptual probabilistic graph for subsequent applications (such as node / edge construction in G²R, query vector guidance, etc.).

[0038] Side-view branch: Unlike the bird's-eye view branch, the side-view branch involves coupling of different target depth information in the YZ plane after X-dimensional compression. Therefore, a decoupling operation needs to be designed for this branch. The specific structure diagram of the side-view branch is as follows: Figure 3 As shown, the process is as follows: If explicit generation is performed during the feature construction stage... If side-view features are obtained, the number of network parameters and computational cost will increase significantly. Therefore, the design decouples the side-view detection head part, and after obtaining the side-view feature map through cross-attention... , The side-view feature map (side-view domain feature) obtained after cross-attention of the side-view branch. : Number of feature channels. The height of the side view feature map (corresponding to) – The discrete resolution of a certain dimension of the grid. Width of the side view feature map (corresponding to) – (The other dimension of the grid is discrete resolution).

[0039] The generation of the side-view center probability prior specifically includes: inputting the side-view feature map into a side-view detection head with K output channels to generate a K-channel side-view heatmap; obtaining the side-view center likelihood prior for K slices divided along the camera X-axis through Sigmoid activation; calculating the slice index using the 3D-annotated target center coordinates; drawing the center supervision only in the corresponding slice channel using a Gaussian kernel; and training using pixel-wise BCE loss. The specific steps are as follows: A K-slice output head is used to introduce depth-direction decoupling in the output space, employing a lightweight design. Convolution will Mapped to Side view center heat map of the passageway:

[0040] No. One channel In the corresponding camera coordinate system The first axis A slice is used to characterize the target center falling within this horizontal interval and... The probability of a grid position. : Convolutional layers (linear projection) convert channels from Mapped to Weight dimension can be understood as , The Sigmoid activation function maps each pixel to... The probability value. Along the camera coordinate system The number of slices (horizontal segments) of the axis. Same as above, side view – Grid resolution.

[0041] Correspondingly, in order to further reduce the information coupling of targets at the same depth on the side, the target center in the 3D annotation is used. , The coordinates (usually in meters) of the target center point in the camera coordinate system within the 3D annotation. Calculate its slice index k.

[0042] and only The center of gravity is drawn using a Gaussian kernel. Finally, for... With the corresponding The channel GT heatmap uses pixel-wise BCE loss.

[0043] This supervision forces different slice channels to learn responses related to lateral position, thereby mitigating the inherent overlap in side-view projections. The target response at a location is distributed across different channels, reducing coupling interference.

[0044] S3. Geometric Graph-Guided Feature Refinement: The multi-scale front-view feature map and the preset decoder query vector are input into the decoder, and the query vector is iteratively updated. During the decoding process, the query vector is enhanced by the geometric graph-guided refinement module G²R based on the bird's-eye view center probability prior, the side-view center probability prior, and the spatial relationship between the query vectors to obtain the refined query vector. The refined query vector is input into the detection head, and the three-dimensional detection result of the target is output.

[0045] The enhancement of the query vector through the geometry-guided refinement module includes the following steps: Node gating steps: For each query vector, based on its currently predicted 3D center, upsample the bird's-eye view center probability prior and the side view center probability prior to obtain the corresponding confidence level, and calculate the gating coefficient based on the confidence level. Use the gating coefficient to weight the query vector to obtain the gated query vector. Graph propagation steps: Treat all gated query vectors as nodes in the graph, calculate edge weights based on the 3D spatial distance between nodes, and use a message passing mechanism to enable each node to aggregate the features of its neighboring nodes and update its own features.

[0046] A geometric relationship graph is introduced to model the inter-target dependencies between decoder queries, specifically as follows: Figure 4 As shown. Figure 4 This diagram illustrates the structure of the geometry-guided refinement (G²R) module of the present invention, given the decoder query features. Treat each query vector as a node in a graph. For each node... Its geometric descriptor This represents the current 3D center estimate. The 3D center is projected onto the bird's-eye view and side view meshes, and sampled from the corresponding prior probability maps to obtain... and (For side views, the channel is selected based on the current slice index). These priors define the node weights used for feature gating. In the node gating step, the formula for calculating the gating coefficient is:

[0047] in, Indicates the first Features of each query vector; and The relative contributions of bird's-eye view and side view priors are controlled; It is a stabilization gate coefficient used to scale the query vector; This is used to avoid completely zeroing out the features of low-confidence nodes. High-confidence query vectors are amplified, while low-confidence query vectors are suppressed but still preserved.

[0048] Subsequently, G²R constructs a fully connected graph over all query vectors: node confidence is injected using the gating mechanism described above, while edge weights are determined by the 3D distance between pairs of nodes. After row normalization, each node aggregates features from its neighbors and updates itself through residual connections, thereby reinforcing geometrically consistent, spatially proximate, and highly confident query vectors and diluting the influence of outliers.

[0049] In the graph propagation step, the formula for calculating edge weights based on the 3D spatial distance between nodes is as follows: First, define the distance kernel. :

[0050] in For nodes With nodes The 3D center; Represents the Euclidean norm; This represents the distance decay coefficient. The normalized adjacency matrix is:

[0051] in, This is a numerically stable term. Message aggregation and residual updates are as follows:

[0052] in, This is the gated query vector. Control the strength of message injection.

[0053] Furthermore, the method of this invention performs end-to-end multi-task joint optimization of the network and introduces cross-view geometric priors through bird's-eye view / side-view center heatmaps. The overall objective function is:

[0054] in, To monitor the loss of the bird's-eye view branch target center probability map, The side branch target center probability map loss is used. To inject dense geometric priors into the auxiliary viewpoints, the center heatmap outputs for both bird's-eye view and side view are supervised. and For the predicted logits of the corresponding branch, A pre-generated binary supervision map for the data is used, corresponding to bev_center and side_center in targets, respectively. We first apply a sigmoid function, and then calculate the mean binary cross-entropy for all pixels:

[0055] These two losses provide dense supervision aligned with the view, prompting the bird's-eye / side-view geometric representation obtained from the projection to form a more concentrated response in the central region of the target.

[0056] in addition, The classification loss is used to supervise the class / foreground probability of each decoder query vector, thereby improving the ability to determine the existence of the target and distinguish between classes. The size regression loss is used to regress the physical size parameters of the 3D bounding box (such as...). Regression constraints are applied to ensure the accuracy of the scaling estimate; The orientation angle regression loss is used to supervise the target orientation angle, thereby improving the stability and consistency of pose estimation and thus synergistically improving the overall 3D detection performance.

[0057] The technical solution of the present invention will be described in detail with reference to specific embodiments. In this embodiment, the KITTI dataset is used as the experimental object to train and test the TriViewNet framework. The scope of protection of the present invention is not limited to the following embodiments.

[0058] Example 1: Monocular 3D Car Target Detection on the KITTI Dataset

[0059] Step 1: Dataset Preparation

[0060] The KITTI 3D object detection dataset was selected, using the RGB images of the left camera and the corresponding 3D annotation information and camera calibration files. The training set of 7481 frames was divided into training set and validation set by 3712 / 3769, and the test set of 7518 frames was used. The 3D detection evaluation standard for the car category is 3D bounding box IoU > 70%, and the average precision AP_40 of 40 recall sampling points is used as the evaluation index.

[0061] Step 2: Network Setup

[0062] The TriViewNet network was built based on the PyTorch framework, with ResNet-50 as the backbone. The three-view encoder uses a 3-layer MSDeformAttn structure, and the decoder uses a 3-layer attention stacked structure with 50 query slots. The resolution of the bird's-eye view and side view meshes was set according to the detection range of the KITTI dataset, and the number of slices in the side view branch was K=8. The hyperparameters of the G²R module were set as follows: α=2, β=1, γ=2, λ=0.1. =0.1, δ=1e-6.

[0063] Step 3: Network Training

[0064] Network training was performed using the parameter settings described above: batch size 16, optimizer AdamW, learning rate 2×10⁻⁶. -4 Weight decay 1×10 -4 The learning rate was reduced to 1 / 10 of its original value at epochs 115 and 145, with a total of 200 training epochs. End-to-end training was performed using a single 4090 graphics card with a random seed of 444.

[0065] Step 4: Model Testing and Results

[0066] The trained TriViewNet model was inferenced on the KITTI validation and test sets without using any additional data (such as depth maps or LiDAR data). The results on the KITTI test set were provided by the official KITTI evaluation server. Table 1 reports the comparison results between AP3D and APBev.

[0067] Table 1: Performance Comparison of the "Cars" Category in the KITTI Dataset

[0068] Note: Best results are shown in bold, and second-best results are shown underlined. "-" indicates no relevant data. All performance is evaluated using IOU > 0.7 in R40 as the benchmark.

[0069] As shown in Table 1, we achieved a stable improvement on KITTI's AP3D and APbev, especially on samples at medium and long distances and with more severe occlusion, which verifies the importance of global geometric consistency modeling for monocular 3D localization.

[0070] The present invention also compared the 3D detection performance on the Rope3D dataset, as shown in Table 2 below.

[0071] Table 2.3D Detection Performance

[0072] Note: Best results are indicated in bold, and second-best results are indicated by underline. All metrics are measured using R40 as the benchmark.

[0073] As shown in Table 2, consistent with the conclusions in Table 1, existing monocular 3D object detection methods typically face greater challenges in roadside scenes: on the one hand, the roadside perspective brings stronger perspective and scale variations; on the other hand, denser targets, frequent occlusion, and sparser depth cues make strategies relying solely on front depth regression more prone to 3D localization drift and dimensional instability. Experimental results demonstrate that TriViewNet, relying on the relationship propagation mechanism of TVSD geometric representation and graph-guided refinement (G²R), can continuously improve the consistency and robustness of 3D bounding boxes in complex roadside scenes, thereby achieving robust monocular 3D detection performance.

[0074] As shown in Tables 1 and 2, TriViewNet of the present invention outperforms state-of-the-art methods on the KITTI validation and test sets, as well as the Rope3D dataset, in terms of AP_3D and AP_bev metrics. In particular, it shows significant improvement on the more difficult "Hard" level, verifying its robustness in occluded and long-distance scenes.

[0075] Example 2: Ablation Experiment Verification of Module Effectiveness

[0076] To verify the effectiveness of the Three-View Space Decoupling (TVSD) module and the Geometry-Guided Refinement (G²R) module in this invention, ablation experiments were conducted on the KITTI validation set to examine: (1) the role of each proposed module; (2) the effect of TVSD; and (3) the effect of Geometry-Guided Refinement. All these experiments were conducted on the KITTI validation set.

[0077] TriViewNet ablation experiments included two modules: TVSD and G. 2 R is shown in Table 3 below: Table 3 shows the ablation experiments using TriViewNet, which include two modules: TVSD and G. 2 R

[0078] The best results are shown in bold.

[0079] As can be seen from Table 3, while keeping the detection framework and training settings unchanged, adding G²R can perform secondary integration of decoder queries through prior confidence gating and distance decay graph message propagation, enhance geometrically consistent high-confidence candidates and suppress outlier hypotheses, thereby bringing stable AP improvement and improving congestion and false detection and localization jitter of long-range targets.

[0080] The specific impact of different branches on TVSD is shown in Table 4 below.

[0081] Table 4. Impact of different branches on the KITTI test set in TVSD

[0082] The best results are shown in bold.

[0083] As shown in Table 4, further breakdown of TVSD reveals that the bird's-eye view geometry primarily improves the alignment of the target's center regression and bird's-eye view projection in the (x)–(z) plane, thus significantly enhancing the bev AP and 3D center stability in distant scenes. The side-view orthogonal view, on the other hand, contributes more significantly to the vertical geometric representation of the target, effectively reducing height-related errors and improving robustness to pitch changes. When both are used together, the model can simultaneously establish geometric constraints in two complementary orthogonal domains (bev and side), ensuring that the decoder's 3D assumptions remain consistent across multiple view domains, thereby achieving a more comprehensive 3D accuracy gain.

[0084] G 2 The influence of R. To further analyze the effectiveness of geometry-guided refinement (G²R), we ablate nodes and graph-building mechanisms as shown in Table 5 below.

[0085] Table 5. Impact of G2R on the KITTI test set

[0086] The best result is shown in bold.

[0087] The results in Table 5 show that simply introducing node gating brings a stable performance improvement, indicating that the node weights constructed based on the bird's-eye view / side prior can effectively suppress low-confidence candidates and strengthen the geometrically consistent query vector representation. Further incorporating graph propagation further improves the overall performance, especially on more challenging samples, demonstrating that the edge weights jointly defined by the distance kernel and prior compatibility can propagate consistency information among candidates, thereby improving the stability of geometric estimation. Overall, The gain of G²R comes from the complementary effect of prior node weight selection (Node) and consistency refinement (Graph).

[0088] In G 2 In R, we introduce three hyperparameters as shown in Table 6 below to control the propagation strength, propagation locality, and the extent to which the prior perception gating mechanism is effective in geometry-guided refinement (G²R).

[0089] Table 6. The impact of variables on the KITTI test set in G2R

[0090] The best results are shown in bold.

[0091] As can be seen from Table 6, the residual weights Appeared In the middle, it is used to control the proportion of propagation messages injected into the query representation; a larger one It can result in a stronger refinement of relationships, but when it is too large, it may lead to over-smoothing of features. Distance decay coefficient Appeared Used to determine the locality of propagation: larger This will form a more local neighborhood, while a smaller one This allows for more global feature mixing. Power exponent Construction of node weights In this context, a larger α indicates a more stringent confidence constraint on the bev prior, because... An increase in the exponent will more strongly suppress low- and medium-confidence candidates, thus making the gating sharper and more biased towards retaining high-confidence hypotheses.

[0092] The ablation experiments, as shown in Tables 3, 4, 5, and 6 above, further demonstrate the effectiveness of the TVSD and G²R modules. The TVSD module provides richer geometric constraints for target localization by introducing bird's-eye view and side-view branches; the G²R module, through node gating and graph propagation, effectively utilizes the spatial relationships between objects for refinement, improving the accuracy and consistency of detection.

[0093] Visualization results: Figure 5 The diagram illustrates the simulated orthogonal view feature map extracted by the TVSD method of this invention. Red boxes represent the target ground truth, red circles represent the target region response in the feature map, and black circles represent targets without response. The front view features are extracted from the simulated orthogonal view bird's-eye view and side view feature maps as follows: Figure 5 As shown, a brighter feature map indicates a higher response and thus a higher prediction probability. Figure 5As can be seen from the first row, targets for which no features were extracted in the front view show a clear response in the simulated top-view feature map, and similarly, a response is observed in the side-view feature map. Figure 5 As can be seen from the second line, some targets that correspond to the side view do not respond to the bird's-eye view, while some targets respond to the bird's-eye view but not to the side view. Therefore, the bird's-eye view and side view branches can complement each other.

[0094] Figure 6 The side view heatmap of the K-silice prediction head is shown, where GT stands for ground truth. Orange dashed circles represent targets, and "+" indicates overlay. To decouple feature coupling in the yz plane caused by target overlap, the prediction head portion is divided along the x-axis. Figure 6 The image shows the distribution of the network's prediction head heatmap when performing a K=8 partition. Brighter areas indicate stronger feature responses, suggesting the network extracted more significant discriminative patterns in that region. Figure 6 As can be seen, after silice partitioning, the three targets do not interfere with each other. Compared with the case of target 2 and 3 overlapping in the side-view heatmap overlap, the k-silice decoupling method proposed in this invention can effectively avoid coupling between targets.

[0095] TriViewNet Detection Performance: The method described in this invention was further compared with representative high-performance monocular 3D detectors to evaluate its accuracy improvement and stability under different scene difficulties in monocular 3D object detection tasks. Therefore, we compared TriViewNet with MonoDETR, Movis, and MonoDGP, and the results are as follows: Figure 7a , Figure 7b As shown, TriViewNet achieves fewer false negatives and false positives by simulating multi-view inference with a single image, especially in scenes with dense targets, such as... Figure 7a Columns 1, 2, and 3 Figure 7b As shown in columns 1 and 3, the prediction of both target position and size is more accurate, as shown in columns 1 and 3. Figure 7a Column 1 and Figure 7b As shown in the second column, TriViewNet performs particularly well at medium to long distances. Furthermore, through a comparison of bird's-eye view detection results, TriViewNet demonstrates greater accuracy and detail in predicting the location of targets in the depth direction. Figure 8 As shown, in Figure 8 In the image, we present the bird's-eye view detection results of MonoDETR, MoVis, MonoDGP, and the network proposed by our method on the Kitti dataset. Green indicates the detection results, and red indicates the ground truth.

[0096] The visualization results are as follows: Figure 5 , Figure 6 , Figure 7a , Figure 7b , Figure 8 As shown, the advantages of the method of the present invention are intuitively demonstrated. Figure 5 This indicates that the orthogonal view features extracted by the TVSD module can still maintain a valid response in areas where the front view features fail (such as occlusion). Figure 6 It demonstrates how the K-slice decoupling mechanism separates targets at different lateral positions into different channels, avoiding overlapping interference from side-view projections. Figure 7a , Figure 7b and Figure 8 The comparison of detection results shows that the method of the present invention has fewer missed detections and false detections in dense and occluded scenes, and the 3D positioning and size prediction are more accurate.

[0097] In summary, this invention effectively improves the performance and robustness of monocular 3D target detection by simulating three-view diagrams for spatial decoupling and combining geometric relationship diagrams for feature refinement, and has high practical value.

[0098] Finally, it should be noted that although the present invention has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Various equivalent changes or substitutions can be made without departing from the concept of the present invention. Therefore, any changes or modifications to the above embodiments within the essential spirit of the present invention will fall within the scope of the claims of the present invention.

Claims

1. A monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning, characterized in that, The TriViewNet detection framework is constructed, which consists of multi-scale feature extraction, three-view space decoupling, and geometry-guided feature refinement, including the following steps: S1. Multi-scale feature extraction: Obtain the monocular input image, extract multi-scale front-view feature maps through the backbone network, flatten them into token sequences, add position encoding and hierarchical embedding, and then concatenate them. The encoded visual tokens are obtained by aggregation through a three-view encoder. S2. Three-view space decoupling: The multi-scale front view feature map is mapped to the simulated bird's-eye view space and side view space through geometric projection and cross attention mechanism, respectively, to generate bird's-eye view feature map and side view feature map. Based on the bird's-eye view feature map and side view feature map, bird's-eye view center probability prior and side view center probability prior are generated respectively. S3. Geometric Graph-Guided Feature Refinement: The multi-scale front-view feature map and the preset decoder query vector are input into the decoder, and the query vector is iteratively updated. During the decoding process, the query vector is enhanced by the geometric graph-guided refinement module G²R based on the bird's-eye view center probability prior, the side-view center probability prior, and the spatial relationship between the query vectors to obtain the refined query vector. The refined query vector is input into the detection head, and the three-dimensional detection result of the target is output.

2. The monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning according to claim 1, characterized in that, In step S2, the multi-scale forward-looking feature map is mapped to the simulated bird's-eye view space and side-view view space through geometric projection and cross-attention mechanism, respectively, to generate the bird's-eye view feature map and the side-view feature map. Specifically, this includes: Create regular geometric grids for the bird's-eye view and the side view respectively; The coordinates of each grid point are encoded using location encoding and combined with a learnable token to generate a corresponding grid query vector; The multi-scale forward-looking feature map is compressed to obtain the compressed geometric subspace features. Using the grid query vector as the query and the geometric subspace features as the key and value, a cross-attention operation is performed to aggregate the bird's-eye view feature map and the side view feature map, respectively.

3. The monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning according to claim 1, characterized in that, The generation of the side-view center probability prior in S2 specifically includes: inputting the side-view feature map into a side-view detection head with K output channels to generate a K-channel side-view heatmap, and obtaining the side-view center likelihood prior of K slices divided along the camera X-axis through Sigmoid activation; calculating the slice index using the 3D labeled target center coordinates, drawing center supervision only in the corresponding slice channel with a Gaussian kernel, and training with pixel-wise BCE loss.

4. The monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning according to claim 3, characterized in that, The slice index The calculation formula is: in, The X-axis coordinates of the target center in 3D annotation. , The range of X-axis coordinates, and K is the number of slices.

5. The monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning according to claim 1, characterized in that, The enhancement of the query vector by the geometry-guided refinement module in S3 includes the following steps: Node gating steps: For each query vector, based on its currently predicted 3D center, upsample the bird's-eye view center probability prior and the side view center probability prior to obtain the corresponding confidence level, and calculate the gating coefficient based on the confidence level. Use the gating coefficient to weight the query vector to obtain the gated query vector. Graph propagation steps: Treat all gated query vectors as nodes in the graph, calculate edge weights based on the 3D spatial distance between nodes, and use a message passing mechanism to enable each node to aggregate the features of its neighboring nodes and update its own features.

6. The monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning according to claim 5, characterized in that, In the node gating step, the formula for calculating the gating coefficient is: in, Indicates the first Features of each query vector; and Control the relative contributions of the bird's-eye view center probability prior and the side view center probability prior; It is a stabilization gating coefficient used to scale the query vector; This is used to avoid setting the features of low-confidence nodes to zero.

7. The monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning according to claim 5, characterized in that, In the graph propagation step, the formula for calculating edge weights based on the 3D spatial distance between nodes is as follows: Define the distance kernel : in, For nodes With nodes The 3D center, Represents the Euclidean norm; This is the distance attenuation coefficient; The normalized adjacency matrix is: in, It is a numerically stable term; Message aggregation and residual updates are as follows: in, This is the gated query vector. To control the strength of message injection.

8. The monocular 3D detection method based on simulated three-view diagrams and geometric relationship reasoning according to claim 1, characterized in that, It also includes end-to-end multi-task joint optimization, with the overall objective function being: in, To monitor the loss of the bird's-eye view branch target center probability map, For the side branch, the target center probability graph loss is used. For classifying losses, For size regression loss, The orientation angle regression loss is used.

Images