A monocular three-dimensional detection method, system, device and storage medium

Abstract

The application relates to the field of computer vision and intelligent traffic perception technology, and discloses a monocular three-dimensional detection method, a system, a device and a storage medium, wherein the method comprises the following steps: acquiring a monocular road scene image and corresponding camera internal parameters; inputting the monocular road scene image into a visual basic model with frozen original parameters to output visual features adapted to a task; inputting the monocular road scene image into a depth basic model with frozen original parameters to extract dense depth representation and encode, and output depth features; fusing the visual features and the depth features to generate a plurality of depth perception target query features, and predicting target related information according to the target query features; calculating final depth based on the camera internal parameters and the target related information, and generating a three-dimensional bounding box according to the final depth and the target related information, and outputting a target category and the three-dimensional bounding box. The application can efficiently realize monocular three-dimensional target detection of a road scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent transportation perception technology, and in particular to a monocular three-dimensional detection method, system, device and storage medium. Background Technology

[0002] Monocular 3D object detection aims to identify the categories of traffic participants from a single RGB image and estimate their position, size, and orientation in 3D space. This technology is of great significance for environmental perception, trajectory prediction, risk assessment, and path planning of autonomous vehicles. Due to the advantages of monocular cameras, such as low cost, easy installation, and easy large-scale deployment, 3D object detection based on monocular images has become an important research direction in the fields of intelligent transportation and autonomous driving perception.

[0003] In monocular 3D object detection tasks, the true spatial distance of an object needs to be indirectly recovered from 2D projection, appearance, scale, and contextual relationships. Existing methods attempt to improve monocular 3D detection performance through geometric constraints, depth-assisted supervision, Transformer Detection Framework, or visual base models.

[0004] For example, some methods estimate depth using camera projection relationships or prior object size; some methods enhance deep learning with the help of LiDAR depth or pseudo-depth labels; and some methods introduce the visual base model DINOv2 into monocular 3D detection tasks to obtain stronger visual representations.

[0005] However, due to the lack of direct depth observation in monocular images, the true spatial distance of a target needs to be indirectly recovered from two-dimensional projection, appearance, scale, and contextual relationships, thus creating inherent depth ambiguity. In real road scenes, distant vehicles occupy a small proportion of the image, occlusion and truncation reduce visible geometric information, and different cameras and different urban environments also introduce distribution differences. All of these factors can lead to a decrease in 3D positioning accuracy.

[0006] Existing methods attempt to improve monocular 3D detection performance through geometric constraints, depth-assisted supervision, Transformer detection frameworks, or visual foundation models. For example, some methods utilize camera projection relationships or object size priors to estimate depth; some methods enhance deep learning with LiDAR depth or pseudo-depth labels; and some methods introduce visual foundation models such as DINOv2 into monocular 3D detection tasks to obtain stronger visual representations. Although these methods have made some progress, significant shortcomings remain: on the one hand, the pre-training objective of visual foundation models is mainly geared towards general semantic understanding, and their frozen features are not naturally adapted to geometrically sensitive 3D localization tasks; on the other hand, directly fine-tuning large-scale visual foundation models with all parameters incurs high computational costs and may weaken the generalization ability of pre-trained models. Furthermore, simply relying on learned depth regression cannot fully utilize the physical projection relationships in camera imaging, resulting in unstable depth estimation in long-distance, occluded, and cross-domain scenes.

[0007] Therefore, how to adapt the basic visual model to monocular 3D object detection tasks while maintaining its generalization ability and deployment efficiency, and how to alleviate monocular depth ambiguity through physically interpretable geometric constraints, are urgent problems to be solved in current monocular 3D detection technology. Summary of the Invention

[0008] To address the aforementioned issues, this invention proposes a monocular 3D detection method, system, device, and storage medium. This method enables the detection of the 3D position, size, and orientation of traffic participants such as vehicles, pedestrians, and cyclists from a single road scene image. It can be applied to scenarios such as autonomous driving, intelligent traffic monitoring, assisted driving perception, and road scene understanding.

[0009] The technical solution adopted in this invention is as follows: A monocular 3D detection method, comprising: Acquire a monocular road scene image and the camera intrinsic parameters corresponding to the monocular road scene image; The monocular road scene image is input into the visual base model with frozen original parameters. The task-adapted visual features are obtained by using the trainable low-rank residual branch set in the transformer attention projection branch of the visual base model. The monocular road scene image is input into the original parameter frozen depth base model, dense depth representation is extracted, and the dense depth representation is encoded to obtain depth features; The visual features and the depth features are jointly input into the depth-guided transformer decoder to generate multiple depth-aware target query features based on multiple learnable target queries. Target-related information is then predicted based on the target query features. The target-related information includes target category, projection coordinate parameters of the target's 3D center on the image plane, target 3D size, target orientation, target physical height, projection height of the target on the image plane, and depth residual. Based on the target physical height, the projected height, and the vertical focal length in the camera intrinsic parameters, the geometric depth anchor point is calculated according to the pinhole camera projection relationship, and the geometric depth anchor point is added to the depth residual to obtain the final depth; Based on the final depth, the camera intrinsic parameters, and the projection coordinate parameters, the three-dimensional center coordinates of the target in the camera coordinate system are obtained by backprojection, and the three-dimensional bounding box of the target is constructed based on the three-dimensional center coordinates, the three-dimensional size, and the orientation.

[0010] Further, the monocular road scene image is input into a visual base model with frozen original parameters. Through a trainable low-rank residual branch set in the transformer attention projection branch of the visual base model, task-adapted visual features are obtained, including: Keeping the original parameters of the visual base model frozen, a low-rank residual branch is added to the projection branch of at least one transformer attention layer of the visual base model; the low-rank residual branch includes a trainable first matrix and a second matrix, the first matrix is ​​randomly initialized, and the second matrix is ​​initialized to a zero matrix; The monocular road scene image is input into the visual base model, and after processing by the transformer attention layer, the adapted visual features are output.

[0011] It should be noted that by adding a low-rank residual branch to the frozen visual base model, the visual features can be shifted towards a geometrically relevant feature space suitable for 3D localization with a very small number of trainable parameters, while maintaining the original general visual representation capabilities of the pre-trained model. Initializing the second matrix to a zero matrix ensures that the initial state of the model is completely consistent with the frozen model, avoiding random branches from disrupting the stability of the pre-trained features in the early stages of training, thereby guaranteeing a smooth start and efficient convergence of the training process.

[0012] Further, the monocular road scene image is input into a depth base model with frozen original parameters, dense depth representations are extracted, and the dense depth representations are encoded to obtain depth features, including: Keeping the original parameters of the depth base model frozen, the monocular road scene image is input into the depth base model to extract dense depth representations; the dense depth representations are encoded into depth features that can be processed by the depth guided transformer decoder.

[0013] It should be noted that freezing the basic depth model preserves its general depth perception capabilities learned on large-scale data, avoiding the degradation of depth representations due to limited data volume in the target task. Encoding and transforming dense depth representations to align their dimensions with visual features facilitates efficient interaction and complementarity of the two modalities in subsequent fusion operations.

[0014] Furthermore, the visual features and the depth features are jointly input into the depth-guided transformer decoder to generate multiple depth-aware target query features based on multiple learnable target queries, and target-related information is predicted based on the target query features, including: The visual features processed by the visual encoder and the depth features processed by the depth encoder are respectively input into the depth-guided transformer decoder; The depth-guided transformer decoder generates multiple depth-aware target query features based on multiple learnable target queries through attentional interactions between the target queries and the visual and depth features. Predict target-related information based on the target query features. The target-related information includes target category, projection coordinate parameters of the target's 3D center on the image plane, target 3D size, target orientation, target physical height, target projection height on the image plane, and depth residual.

[0015] It should be noted that fusing visual and depth features enables the model to utilize both appearance semantic information and geometric depth cues, compensating for the inherent deficiency of monocular images in lacking physical depth information. The cross-attention mechanism, using the fused features as keys and values, allows each target query to adaptively aggregate its own relevant visual-depth joint representations from the entire feature map, thereby generating target query features with depth perception capabilities and providing an information-rich feature foundation for subsequent attribute prediction.

[0016] Furthermore, the method for calculating the final depth includes: Input depth calculation parameters, including the target's physical height, the target's projected height in the image plane, and the depth residual; Based on the target's physical height, the projected height, and the vertical focal length in the camera's intrinsic parameters, the geometric depth anchor point is calculated according to the pinhole camera's projection relationship; the geometric depth anchor point is then compared with the depth residual to compensate for the deviation, thus obtaining the final depth.

[0017] It should be noted that the geometric depth anchor is calculated based on the classic pinhole camera projection model, providing a depth benchmark that conforms to the laws of physical imaging and has strong interpretability and generalization. The depth residual is predicted by a data-driven approach, which can flexibly compensate for complex error factors that are difficult to model using geometric models. The combination of the two takes into account both the robustness of physical constraints and the adaptability of the learning method, effectively improving the accuracy of target depth estimation in complex road scenarios.

[0018] Furthermore, the method for constructing the three-dimensional bounding box includes: Input the three-dimensional bounding box attribute parameters, which include the projection coordinates of the target's three-dimensional center on the image plane, the target's three-dimensional size, and the target's orientation; Based on the final depth, the camera intrinsic parameters, and the projection coordinates of the target's 3D center on the image plane, the projection coordinates are back-projected to obtain the 3D center coordinates of the target in the camera coordinate system. The three-dimensional bounding box of the target is constructed based on the three-dimensional center coordinates, the three-dimensional dimensions, and the orientation.

[0019] It should be noted that the final depth, as the core basis for calculating the 3D center coordinates, directly affects the positioning accuracy of the 3D bounding box in space. Combining the 3D dimensions and orientation can completely describe the geometric occupancy and attitude information of the target in 3D space, providing structured 3D perception results for downstream autonomous driving planning and control modules, facilitating subsequent decision-making algorithms to predict the target's motion and assess collision risks.

[0020] Furthermore, the training methods for the visual foundation model and the deep foundation model include: Acquire training images, camera intrinsic parameters, target 2D annotations, and target 3D annotations; The target depth of each target in the camera coordinate system is determined based on the three-dimensional annotation of each target, and the target depth is discretized to obtain the target-level depth label; A foreground depth label map is constructed based on the two-dimensional bounding boxes of each target. Pixels inside each two-dimensional bounding box are assigned a target-level depth label to the corresponding target. When multiple two-dimensional bounding boxes overlap, the pixels in the overlapping area are assigned a target-level depth label to the target with the smaller actual depth. The foreground depth map loss is calculated based on the foreground depth map predicted by the model and the foreground depth label map. The target depth loss is calculated based on the final depth predicted by the model, and backpropagation is performed by combining the two-dimensional detection loss, the three-dimensional detection loss, and the foreground depth map loss. Keep the original parameters of the visual and deep base models frozen, and update the trainable parameters of the low-rank residual branch and the downstream detection module.

[0021] It should be noted that the above training supervision method constructs a foreground depth map using target-level depth labels, eliminating the need for additional dense depth map annotations and reducing data annotation costs. Assigning overlapping regions to targets closer to the camera aligns with the occlusion logic of perspective projection, making depth map supervision more accurate and reasonable. Joint optimization of depth map loss and 3D detection loss provides direct geometric supervision signals to the low-rank adaptation module, promoting more effective encoding of depth-related information by the adapted visual features, thereby improving the model's ability to perceive target depth.

[0022] A monocular three-dimensional inspection system, comprising: The image and camera intrinsic parameter input module is configured to acquire a monocular road scene image and camera intrinsic parameters corresponding to the monocular road scene image; The visual base model feature extraction module is configured to input the monocular road scene image into the visual base model with frozen original parameters, and obtain task-adapted visual features through a trainable low-rank residual branch set in the transformer attention projection branch of the visual base model. The depth feature extraction module is configured to input the monocular road scene image into a depth base model with frozen original parameters, extract dense depth representations, and encode the dense depth representations to obtain depth features. The depth-guided transformer decoding module is configured to perform joint decoding based on multiple learnable target queries according to the visual features and the depth features, and generate multiple depth-aware target query features. The detection attribute prediction module is configured to predict target-related information based on the target query features. The target-related information includes target category, projection coordinate parameters of the target's three-dimensional center on the image plane, target's three-dimensional size, target orientation, target's physical height, target's projection height on the image plane, and depth residual. The geometric complementary depth estimation module is configured to calculate the geometric depth anchor point according to the pinhole camera projection relationship based on the target physical height, the projected height and the vertical focal length in the camera intrinsic parameters, and add the geometric depth anchor point to the depth residual to obtain the final depth; The 3D detection result output module is configured to back-project the target's 3D center coordinates in the camera coordinate system based on the final depth, the camera intrinsic parameters, and the projection coordinate parameters, and construct the target's 3D bounding box based on the 3D center coordinates, the 3D size, and the orientation.

[0023] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the monocular three-dimensional detection method.

[0024] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the monocular 3D detection method.

[0025] The beneficial effects of this invention are as follows: 1. This invention uses a single road scene image as input and extracts data in parallel using a visual baseline model and a depth baseline model, achieving 3D target detection without relying on multiple sensors or temporal information. Compared to the end-to-end approach of directly regressing depth, this invention utilizes a frozen visual baseline model to retain general visual representation capabilities and adds a low-rank adaptation module to achieve directional shift of visual features to geometrically related spaces with a small number of trainable parameters. This reduces the risk of overfitting and avoids the training overhead of fine-tuning a large model. Simultaneously, the dense depth representation provided by the depth baseline model supplements the fusion stage with explicit geometric cues. The fusion decoding of visual and depth features allows the target query features to simultaneously contain appearance semantics and spatial depth information. Based on this, geometric depth anchors are calculated based on the pinhole camera projection relationship and added to the depth residual to obtain the final depth. This ensures that the depth estimation results have physically interpretable baseline constraints and can flexibly compensate for complex factors such as target pose changes, occlusion truncation, image discretization, and network prediction bias through residual learning, effectively improving the accuracy and robustness of 3D target localization under monocular images.

[0026] 2. This invention employs a structure design of frozen parameters plus low-rank adaptation in the visual base model. Its core advantage lies in balancing task adaptability and training efficiency. The low-rank adaptation module is set in the query, key, and value projection branches of the attention layer. Low-rank decomposition significantly reduces the number of trainable parameters, enabling the model to converge stably even with limited target task data. Initializing the second matrix as a zero matrix ensures that the adaptation branch does not change the original model output during the initial training phase, guaranteeing the complete inheritance of pre-trained knowledge and a smooth start to the training process. Regarding deep feature extraction, the frozen deep base model provides reliable dense deep representations, which, after encoding, are fused with visual features in the channel or spatial dimensions. This results in geometrically complementary enhancements to visual semantic features, providing richer conditional information for target query generation. Consequently, the accuracy of the detection head in predicting target spatial attributes is improved without relying on additional annotation costs.

[0027] 3. The geometric complementary depth estimation mechanism employed in this invention achieves an organic combination of interpretability and flexibility by embedding a physical projection model into a data-driven framework. Specifically, the geometric depth anchor point is calculated based on the target's physical height, 2D projection height, and camera's vertical focal length according to the projection relationship. This anchor point itself possesses high physical reliability, providing an accurate depth benchmark, especially when the target's pose is regular and occlusion is minor. The joint prediction of depth residuals and depth uncertainty endows the model with adaptive compensation capabilities for unmodelable errors in complex scenes. During the training phase, the uncertainty-aware loss function allows the network to adjust gradient contributions based on the sample's own quality, reducing the interference of difficult samples or annotation noise on model training. Supervision using foreground depth maps constructed from target-level depth labels further provides structured constraints for depth prediction. When 2D bounding boxes overlap, the overlapping region is assigned to targets closer to the camera, conforming to the physical laws of perspective occlusion. This allows for more accurate supervision signals during depth feature extraction, resulting in more spatially consistent 3D detection results during the inference phase. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a monocular three-dimensional detection system according to Embodiment 1 of the present invention.

[0029] Figure 2 This is a schematic diagram of the low-rank adapter module structure in Embodiment 2 of the present invention.

[0030] Figure 3 This is a schematic diagram of the internal and external processing flow of the geometric complementary depth estimation module in Embodiment 2 of the present invention. Detailed Implementation

[0031] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments are now described. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0032] Example 1 This embodiment provides a monocular 3D detection method and system. By performing task-specific low-rank adaptation on the frozen visual base model and combining geometric depth anchor points based on camera projection relationship with learning-based residual correction, it can detect the 3D position, size, and orientation of traffic participants such as vehicles, pedestrians, and cyclists from a single road scene image. It can be applied to scenarios such as autonomous driving, intelligent traffic monitoring, assisted driving perception, and road scene understanding.

[0033] Specifically, the monocular three-dimensional detection method and system of this embodiment can be implemented in the following ways.

[0034] I. Overall Plan The monocular 3D detection method and system in this embodiment are designed for 3D target detection tasks in road scenes. By adapting the visual basic model and using geometric complementary depth estimation, the 3D positioning accuracy under monocular image conditions is improved.

[0035] Specifically, taking a single road scene image as input, visual features are extracted using a frozen visual baseline model. A low-rank adaptation module is then set in the attention layer of its transformer to shift general visual features towards a geometrically relevant feature space suitable for 3D localization. Simultaneously, a frozen depth baseline model is used to extract dense depth representations, supervised by target-level depth labels. The low-rank adapted visual and depth features are fused by a visual and depth feature fusion module and then input into the depth-guided transformer decoding module to obtain a depth-aware target query. The detection attribute prediction module predicts the target category, 2D bounding box, 3D center-related parameters, 3D size, orientation, 2D projection height, target physical height, depth residual, and depth uncertainty based on the target query. Finally, a geometrically complementary depth estimation module calculates geometric depth anchor points based on the pinhole camera projection relationship and fuses them with the depth residual to obtain the final depth. The 3D detection result output module generates and outputs the target 3D bounding box based on the final depth and other 3D attributes.

[0036] II. Specific Plan This embodiment provides a monocular three-dimensional detection method, including: Step 1: Obtain the monocular road scene image to be detected and the corresponding camera intrinsic parameters. The camera intrinsic parameters include at least the camera's focal length in the vertical direction of the image. f y .

[0037] Step 2: Input the monocular road scene image into the visual base model and keep the original parameters of the visual base model frozen; set a low-rank adaptation module in at least one transformer attention layer of the visual base model, and adapt the visual features through the trainable low-rank residual branch in the low-rank adaptation module to obtain visual features suitable for monocular 3D target localization.

[0038] Step 3: Input the monocular road scene image into the depth base model, extract the dense depth representation of the monocular road scene image, and encode the dense depth representation to obtain depth features.

[0039] Step four: fuse the visual features and the depth features, and input the resulting fused features into the depth-guided transformer decoding module to generate multiple depth-aware target query features.

[0040] Step 5: Input the depth-sensing target query features into the detection attribute prediction module to predict the target category, 2D bounding box, 3D center-related parameters, 3D size, orientation, and target physical height. h 3D The two-dimensional projection height of the target in the image plane v 2D Depth residual δz and depth uncertainty.

[0041] Step six, based on the target physical height h 3D The two-dimensional projection height v 2D and the focal length of the camera in the vertical direction of the image. f y The geometric depth anchor point is calculated according to the following formula:

[0042] The geometric depth anchor point z geo The final depth z of the target is obtained by adding the depth residual δz predicted by the detection attribute prediction module:

[0043] The geometric depth anchor point is used to provide a depth reference that conforms to the camera projection relationship, and the depth residual is used to compensate for depth errors caused by target pose changes, occlusion, truncation, image discretization, and network prediction bias.

[0044] Step 7: Determine the target's 3D center coordinates based on the final depth, camera intrinsic parameters, and 3D center-related parameters. Combine this with the 3D dimensions and orientation to generate the target's 3D bounding box, and output the target category and 3D bounding box parameters.

[0045] Where x, y, z represent the three-dimensional center coordinates of the target, h, w, l represent the height, width, and length of the target, respectively, and θ represents the target's orientation.

[0046] It should be noted that steps two and three can be executed in parallel, and the execution order of the two does not constitute a limitation on the technical solution of the present invention.

[0047] During the model training phase, a foreground depth map is constructed for supervision based on the target-level depth labels. When the 2D bounding boxes of different targets overlap, the overlapping region is assigned to the target closer to the camera, so as to supervise the depth feature extraction and depth prediction process using the target-level depth labels.

[0048] Accordingly, this embodiment also provides a monocular 3D detection system, including an image and camera intrinsic parameter input module, a visual basic model feature extraction module, a low-rank adaptation module, a depth feature extraction module, a visual and depth feature fusion module, a depth-guided transformer decoding module, a detection attribute prediction module, a geometric complementary depth estimation module, and a 3D detection result output module.

[0049] like Figure 1 As shown, the image and camera intrinsic parameter input modules are connected to the visual basic model feature extraction module and the depth feature extraction module, respectively; the low-rank adaptation module is set in at least one transformer attention layer of the visual basic model feature extraction module; the visual features and depth features after low-rank adaptation are jointly input into the visual and depth feature fusion module, and the fusion result is sequentially input into the depth-guided transformer decoding module and the detection attribute prediction module; the target physical height, 2D projection height and depth residual output by the detection attribute prediction module and the camera vertical focal length provided by the image and camera intrinsic parameter input module are jointly input into the geometric complementary depth estimation module; the final depth output by the geometric complementary depth estimation module and the target category, 3D center, 3D size and orientation output by the detection attribute prediction module are jointly input into the 3D detection result output module to generate and output the target category and 3D bounding box.

[0050] It should be noted that the low-rank adaptation module and the geometrically complementary depth estimation module work collaboratively through the aforementioned data link. The low-rank adaptation module adjusts the attention allocation of the visual base model, making the visual features more sensitive to target scale, spatial boundaries, and occlusion relationships. These visual features are fused and decoded to form a target query. The detection attribute prediction module jointly predicts the target's physical height, 2D projection height, depth residual, and depth uncertainty based on the same target query. The geometrically complementary depth estimation module then generates a physically constrained final depth based on the target's physical height, 2D projection height, camera vertical focal length, and depth residual. The training loss propagates back along this data link to constrain the low-rank adaptation module and achieve end-to-end collaborative optimization.

[0051] III. Low-rank adaptation module The low-rank adaptation module is one of the core technologies of this invention. Its basic idea is to add a low-rank residual branch to the attention projection layer of the frozen visual base model, instead of directly updating the parameters of the original large model.

[0052] For input features The original frozen projection weights are The low-rank branch includes two trainable matrices. and Then the adapted projection can be represented as:

[0053] in, To adapt the rank to the lower rank, Scaling factor This indicates random deactivation or an equivalent regular expression operation. and The dimension satisfies the low-rank constraint, and It is smaller than the input or output channel dimension, thus significantly reducing the number of trainable parameters.

[0054] In a preferred embodiment, the low-rank adaptation modules are respectively configured in the query projection, key projection, and value projection (Query, Key, Value) of the attention layer:

[0055]

[0056]

[0057] in, , , For the frozen pre-trained projected weights, ( , ), ( , ), ( , ) represents the corresponding trainable low-rank adaptation parameters. Through adaptation... And Key, can adjust attention allocation, making the model focus more on target areas and spatial relationships related to 3D localization; through adaptation This can improve the feature representation transmitted to downstream detection heads.

[0058] During training initialization, the low-rank matrix A can be initialized using Kaiming Uniform Initialization, and the low-rank matrix B can be initialized to a zero matrix, so that the initial output of the low-rank branch is zero. In this way, the initial state of the model is consistent with the frozen visual base model, avoiding the destruction of pre-trained features at the beginning of training.

[0059] During the inference deployment phase, low-rank branches can be merged back into the original projected weights:

[0060] The merged model can complete inference without additional branches, thus achieving task adaptability while maintaining high deployment efficiency.

[0061] IV. Geometric Complementary Depth Estimation Module The geometric complementary depth estimation module is positioned after the detection attribute prediction module, and it includes a geometric depth anchor point calculation unit and a depth residual fusion unit. For each target query, the detection attribute prediction module predicts the target's physical height. The two-dimensional projection height of the target on the image plane Depth residual and deep uncertainty Based on the pinhole camera projection model, the target depth satisfies an approximate relationship with the physical height, focal length, and projected height. Therefore, a geometric depth anchor point is constructed:

[0062] Meanwhile, the multilayer perceptron (MLP) in the attribute prediction module predicts depth residuals and depth uncertainty based on the target query feature h:

[0063] The final depth is:

[0064] in, Provides a physically interpretable depth benchmark. This is used to compensate for errors caused by changes in target pose, occlusion, truncation, image discretization, and network prediction bias. This structure allows the model to have both geometric constraints and retain the flexibility of learned regression.

[0065] V. Loss Function Design This embodiment adopts an end-to-end training method, and the overall loss includes two-dimensional detection loss, three-dimensional attribute loss, depth loss and foreground depth map loss.

[0066] The final depth output by the geometrically complementary depth estimation module Uncertainty-aware Laplacian depth loss can be used:

[0067] in, z gt To achieve true depth, This represents the uncertainty in the prediction. The loss can adaptively adjust the deep regression error weights based on the sample uncertainty.

[0068] For the depth branch, foreground depth map supervision can be constructed based on target-level depth labels. Specifically, pixels inside the 2D target bounding box are assigned discrete depth labels for that target; when multiple 2D bounding boxes overlap, the overlapping region can be assigned to the nearest target. This method does not require additional dense depth annotations.

[0069] The total loss can be written as:

[0070] in, This includes classification loss, 2D bounding box regression loss, and GIoU loss; Including losses related to 3D center, size, and orientation; This represents the loss in the foreground depth map.

[0071] In summary, the monocular 3D detection method and system of this embodiment have the following characteristics: 1. Improve the adaptability of the visual base model to monocular 3D detection tasks. By introducing a low-rank residual branch into the frozen visual base model through a low-rank adaptation module, general semantic features are transformed into geometrically relevant feature spaces, alleviating the task mismatch problem when the visual base model is directly transferred to monocular 3D detection tasks.

[0072] 2. Reduce training and deployment costs. The parameters of the original visual base model remain frozen, and only the low-rank adaptation parameters are trained; during inference, the low-rank branches can be merged back into the original projected weights, thereby maintaining an inference structure and parameter scale close to the baseline model.

[0073] 3. Improve the physical consistency of depth estimation. The geometric complementary depth estimation module constructs geometric depth anchors using the projection relationship of pinhole cameras and compensates for deviations in the real scene through learning-based residual correction, making depth estimation both interpretable and flexible.

[0074] 4. Enhanced robustness to occlusion and distant target detection. The low-rank adaptation module enables the model to focus on target regions with greater geometric significance, while the geometrically complementary depth estimation module provides stable depth constraints. The combination of the two helps improve the 3D localization accuracy of occluded, truncated, and distant targets.

[0075] 5. Improved cross-domain generalization ability. By preserving the pre-trained priors of the visual base model and introducing physical geometric constraints, this method exhibits lower depth error and stronger detection stability when transferring from KITTI to cross-dataset scenarios such as nuScenes.

[0076] 6. Reduced reliance on additional sensor supervision. This method can be trained using monocular images, camera intrinsics, and target-level depth labels, without relying on additional LiDAR dense depth annotations, making it suitable for low-cost monocular perception deployments.

[0077] Example 2 This embodiment is based on embodiment 1: This embodiment provides a monocular 3D detection method for detecting the 3D position, size, and orientation of traffic participants from monocular images of road scenes, as detailed below.

[0078] I. Overall Architecture Implementation The input consists of a monocular color image (RGB image) of the road scene, and the corresponding camera intrinsics are read. The input image is first fed into the frozen visual base model backbone network to obtain multi-layer visual features.

[0079] To adapt the frozen visual base model to the monocular 3D object detection task, a low-rank adaptation module is set in its transformer attention layer. Only the low-rank parameters in the low-rank adaptation module are trained, while the original parameters of the visual base model are kept frozen.

[0080] Simultaneously, the input image is fed into the frozen depth base model branch to obtain a dense depth representation. This depth representation, after encoding, is input together with the low-rank adapted visual features into the visual and depth feature fusion module. The fusion result is then input into the depth-guided transformer decoding module to generate multiple target queries. The detection attribute prediction module predicts the target category, 2D bounding box, 3D center, 3D size, orientation, target physical height, 2D projected height, depth residual, and depth uncertainty based on each target query.

[0081] Subsequently, the geometrically complementary depth estimation module calculates the final depth using the target's physical height, 2D projected height, and depth residual output by the detection attribute prediction module, as well as the vertical focal length from the camera's intrinsic parameters. The 3D detection result output module generates and outputs the 3D bounding box for each target based on the final depth, 3D center, 3D dimensions, and orientation.

[0082] II. Implementation of Low-Rank Adaptation Module like Figure 2 As shown, in a preferred embodiment, the low-rank adaptation module is set in the multi-head self-attention layer of the frozen visual base model. For input features... Original projection weights , , All remain frozen. For each projected weight, set two low-rank matrices. and ,in Map the input features to a low-rank space. Map the low-rank space back to the output channel. Only update during training. and .

[0083] The Query, Key, and Value after low-rank adaptation are as follows:

[0084]

[0085]

[0086] Then calculate according to the standard attention formula:

[0087] This design adjusts attention allocation and feature transformation using a small number of trainable parameters, enabling the frozen visual base model to better focus on 3D localization-related information such as the vehicle body, spatial boundaries, occlusion relationships, and scale changes.

[0088] During deployment, the low-rank matrix can be combined with the frozen projection weights:

[0089]

[0090]

[0091] After merging, no additional low-rank branches are needed, and the network structure during the deployment phase is consistent with that of a normal projection layer.

[0092] III. Implementation of the Geometric Complementary Depth Estimation Module like Figure 3 As shown, in a preferred embodiment, the detection attribute prediction module precedes the geometric complementary depth estimation module; the two are independent functional modules connected sequentially. The geometric complementary depth estimation module includes a geometric depth anchor point calculation unit and a depth residual fusion unit. The detection attribute prediction module predicts the physical height of an object from the target query. Two-dimensional projection height Depth residual and uncertainty According to the vertical focal length in the camera's intrinsic parameters. Calculate the geometric depth anchor point:

[0093] This anchor point embodies the physical relationship between the object's true height, the image projection height, and the camera's focal length. However, considering the differences in target pose, occlusion, truncation, annotation noise, and network prediction bias in real-world road scenes, using only this anchor point may not be accurate enough. Therefore, further... Predict depth residuals based on target query features And obtain the final depth:

[0094] This approach combines analytical geometric estimation with data-driven correction, avoiding both the lack of physical constraints in pure regression and the difficulty of adapting purely geometric methods to complex real-world scenarios.

[0095] IV. Experiment Setup (1) Dataset selection: The KITTI dataset is a widely used benchmark dataset for road scenes in the field of 3D object detection for autonomous driving. It contains 7,481 training images and 7,518 test images, and provides 2D bounding boxes, 3D bounding boxes, class information, and camera calibration information for targets such as vehicles, pedestrians, and cyclists. Following the common partitioning method in monocular 3D object detection, the training set is divided into 3,712 training images and 3,769 validation images. This method is primarily evaluated on the KITTI validation set for the Car class to verify its effectiveness in monocular 3D object detection tasks in road scenes.

[0096] The nuScenes dataset was used to further evaluate the model's cross-dataset generalization ability. The experiments employed a subset of the front-view cameras from nuScenes and converted them to the KITTI coordinate system and annotation format according to existing cross-dataset evaluation methods. The model was trained solely on the KITTI dataset without additional fine-tuning on the nuScenes dataset, directly testing its generalization performance under different camera configurations, road layouts, and scene distributions.

[0097] (2) Evaluation indicators: On the KITTI dataset, Average Precision (AP) was used as the primary evaluation metric, calculating both the average precision for bird's-eye view (AP_BEV) and the average precision for 3D object detection (AP_3D). The evaluation followed the official KITTI R40 protocol, and results were reported according to three difficulty levels: Easy, Moderate, and Hard. The Moderate AP_3D for the Car category is a commonly used benchmark in monocular 3D object detection.

[0098] In the cross-dataset evaluation from KITTI to nuScenes, distance matching was used to calculate the mean average precision (mAP), depth mean absolute error (DME), and match rate. Specifically, the mean precision (AP) was calculated based on the matching relationship between the predicted bounding box and the ground truth bounding box at the center distance on the ground plane, using four distance thresholds: 0.5 m, 1.0 m, 2.0 m, and 4.0 m. The final mAP was the average of the APs across these thresholds. Furthermore, detection results were statistically analyzed for three distance ranges: 0-20 m, 20-40 m, and above 40 m, to assess the model's generalization ability and depth estimation stability across different depth ranges.

[0099] (3) Implementation details: The experiments were conducted on a single NVIDIA RTX A6000 GPU based on the PyTorch framework, with a batch size of 8. Network optimization used the AdamW optimizer, with an initial learning rate of 2×10⁻⁶. -4 The weight decay is set to 1×10. -4 The training process employs a cosine annealing learning rate scheduling strategy with warmup, and the total training period is 225 epochs.

[0100] In terms of model structure, the frozen visual foundation model DINOv2 is used as the backbone network, and low-rank adaptation modules are set in the query projection, key projection, and value projection of its Transformer Attention Layer. In the main experiment, the low-rank adaptation rank r is set to 64, and the impact of different rank values ​​on performance is further compared in the ablation experiment. At the same time, dense depth representations are extracted using the frozen depth foundation model Depth Anything V2 branch, and foreground depth maps are constructed for supervision through target-level depth labels. The depth range is set to

[10] . -3 The depth is calculated as

[60] m and discretized into 80 depth intervals. Data augmentation employs random flipping and random pruning. Finally, the detection attribute prediction module and the geometric complementary depth estimation module are connected in sequence. The geometric complementary depth estimation module obtains the final depth based on the predicted target physical height, 2D projection height, depth residual, and camera vertical focal length.

[0101] (4) Experimental results and conclusions: Experimental results show that, compared to the baseline scheme without a low-rank adaptation module and a geometrically complementary depth estimation module, our method achieves overall improvements in AP_BEV and AP_3D metrics on the KITTI validation set, and demonstrates better stability in 3D localization for medium and hard difficulty targets, distant targets, and occluded targets. Ablation experiments demonstrate that the low-rank adaptation module enhances geometrically relevant feature representation while keeping the original parameters of the visual base model frozen, and the geometrically complementary depth estimation module reduces depth estimation bias by utilizing projective geometric constraints; the two modules, when used together, have synergistic gains. Cross-dataset testing further shows that our method maintains good matching rate and depth stability under different camera configurations and road scene distributions, exhibiting good cross-domain generalization ability.

[0102] In summary, the monocular three-dimensional detection method of the present invention has the following characteristics: 1. Low-rank adaptation module for the visual base model for monocular 3D object detection: A trainable low-rank residual branch is set in the transformer attention layer of the frozen visual base model to adapt general visual features to the monocular 3D object detection task. Specifically, low-rank residual updates are added to query projection, key projection, and value projection, and geometrically relevant features are learned through a small number of trainable parameters.

[0103] 2. A geometrically complementary depth estimation method based on projected geometric anchor points and learned residual correction: This method calculates geometric depth anchor points based on the predicted target physical height, 2D projected height, and camera vertical focal length. These anchor points are then added to the learned depth residuals to obtain the final depth. This approach combines camera projection physical constraints with data-driven depth correction to alleviate monocular depth ambiguity.

[0104] 3. A monocular 3D target detection framework that combines low-rank adaptation and geometric complementary depth estimation: The overall process of the "low-rank adaptation module + geometric complementary depth estimation module" is as follows: First, the low-rank adaptation module makes the frozen visual base model output geometrically sensitive features that are more suitable for 3D localization. Then, the geometric complementary depth estimation module generates a physically consistent final depth. Finally, the target category and 3D bounding box are output.

[0105] 4. The system architecture for monocular 3D target detection in road scenes includes an image and camera intrinsic parameter input module, a visual basic model feature extraction module, a low-rank adaptation module, a depth feature extraction module, a visual and depth feature fusion module, a depth-guided transformer decoding module, a detection attribute prediction module, a geometric complementary depth estimation module, and a 3D detection result output module, which are used to output the target category and 3D bounding box (x, y, z, h, w, l, θ) from the monocular road scene image.

[0106] Example 3 This embodiment is based on embodiment 1: This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the monocular 3D detection method of Embodiment 1. The computer program can be in the form of source code, object code, executable file, or some intermediate form.

[0107] It should be noted that the aforementioned computer equipment executes the program instructions stored in the memory through the processor, which can fully realize each processing step of the monocular 3D detection method. It has corresponding functional modules and processing capabilities and can be deployed in the on-board computing platform of autonomous vehicles or cloud processing system to provide efficient 3D target detection services for practical application scenarios.

[0108] Example 4 This embodiment is based on embodiment 1: This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the monocular 3D detection method of Embodiment 1. The computer program can be in the form of source code, object code, executable file, or some intermediate form. The storage medium includes any entity or device capable of carrying computer program code, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0109] It should be noted that the aforementioned computer-readable storage medium can permanently store the program that implements the monocular 3D detection method in an executable form, which facilitates deployment and migration on various computing devices, and also facilitates algorithm version management and update maintenance between different devices.

[0110] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technical or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0111] It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. Furthermore, the terms "first," "second," and "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

Claims

1. A monocular three-dimensional detection method, characterized in that, include: Acquire a monocular road scene image and the camera intrinsic parameters corresponding to the monocular road scene image; The monocular road scene image is input into the visual base model with frozen original parameters. The task-adapted visual features are obtained by using the trainable low-rank residual branch set in the transformer attention projection branch of the visual base model. The monocular road scene image is input into the original parameter frozen depth base model, dense depth representation is extracted, and the dense depth representation is encoded to obtain depth features; The visual features and the depth features are jointly input into the depth-guided transformer decoder to generate multiple depth-aware target query features based on multiple learnable target queries. Target-related information is then predicted based on the target query features. The target-related information includes target category, projection coordinate parameters of the target's 3D center on the image plane, target 3D size, target orientation, target physical height, projection height of the target on the image plane, and depth residual. Based on the target physical height, the projected height, and the vertical focal length in the camera intrinsic parameters, the geometric depth anchor point is calculated according to the pinhole camera projection relationship, and the geometric depth anchor point is added to the depth residual to obtain the final depth; Based on the final depth, the camera intrinsic parameters, and the projection coordinate parameters, the three-dimensional center coordinates of the target in the camera coordinate system are obtained by backprojection, and the three-dimensional bounding box of the target is constructed based on the three-dimensional center coordinates, the three-dimensional size, and the orientation.

2. The monocular three-dimensional detection method according to claim 1, characterized in that, The monocular road scene image is input into a visual base model with frozen original parameters. Through a trainable low-rank residual branch set in the transformer attention projection branch of the visual base model, task-adapted visual features are obtained, including: Keeping the original parameters of the visual base model frozen, a low-rank residual branch is added to the projection branch of at least one transformer attention layer of the visual base model; the low-rank residual branch includes a trainable first matrix and a second matrix, the first matrix is ​​randomly initialized, and the second matrix is ​​initialized to a zero matrix; The monocular road scene image is input into the visual base model, and after processing by the transformer attention layer, the adapted visual features are output.

3. The monocular three-dimensional detection method according to claim 1, characterized in that, The monocular road scene image is input into a depth baseline model with frozen original parameters. Dense depth representations are extracted, and the dense depth representations are encoded to obtain depth features, including: Keeping the original parameters of the depth base model frozen, the monocular road scene image is input into the depth base model to extract dense depth representations; the dense depth representations are encoded into depth features that can be processed by the depth guided transformer decoder.

4. The monocular three-dimensional detection method according to claim 1, characterized in that, The visual features and the depth features are jointly input into the depth-guided transformer decoder to generate multiple depth-aware target query features based on multiple learnable target queries, and target-related information is predicted based on the target query features, including: The visual features processed by the visual encoder and the depth features processed by the depth encoder are respectively input into the depth-guided transformer decoder; The depth-guided transformer decoder generates multiple depth-aware target query features based on multiple learnable target queries through attentional interactions between the target queries and the visual and depth features. Predict target-related information based on the target query features. The target-related information includes target category, projection coordinate parameters of the target's 3D center on the image plane, target 3D size, target orientation, target physical height, target projection height on the image plane, and depth residual.

5. The monocular three-dimensional detection method according to claim 1, characterized in that, The method for calculating the final depth includes: Input depth calculation parameters, including the target's physical height, the target's projected height in the image plane, and the depth residual; Based on the target's physical height, the projected height, and the vertical focal length in the camera's intrinsic parameters, the geometric depth anchor point is calculated according to the pinhole camera's projection relationship; the geometric depth anchor point is then compared with the depth residual to compensate for the deviation, thus obtaining the final depth.

6. The monocular three-dimensional detection method according to claim 1, characterized in that, The method for constructing the three-dimensional bounding box includes: Input the three-dimensional bounding box attribute parameters, which include the projection coordinates of the target's three-dimensional center on the image plane, the target's three-dimensional size, and the target's orientation; Based on the final depth, the camera intrinsic parameters, and the projection coordinates of the target's 3D center on the image plane, the projection coordinates are back-projected to obtain the 3D center coordinates of the target in the camera coordinate system. The three-dimensional bounding box of the target is constructed based on the three-dimensional center coordinates, the three-dimensional dimensions, and the orientation.

7. The monocular three-dimensional detection method according to claim 1, characterized in that, The training methods for the visual foundation model and the deep foundation model include: Acquire training images, camera intrinsic parameters, target 2D annotations, and target 3D annotations; The target depth of each target in the camera coordinate system is determined based on the three-dimensional annotation of each target, and the target depth is discretized to obtain the target-level depth label; A foreground depth label map is constructed based on the two-dimensional bounding boxes of each target. Pixels inside each two-dimensional bounding box are assigned a target-level depth label to the corresponding target. When multiple two-dimensional bounding boxes overlap, the pixels in the overlapping area are assigned a target-level depth label to the target with the smaller actual depth. The foreground depth map loss is calculated based on the foreground depth map predicted by the model and the foreground depth label map. The target depth loss is calculated based on the final depth predicted by the model, and backpropagation is performed by combining the two-dimensional detection loss, the three-dimensional detection loss, and the foreground depth map loss. Keep the original parameters of the visual and deep base models frozen, and update the trainable parameters of the low-rank residual branch and the downstream detection module.

8. A monocular three-dimensional detection system, characterized in that, include: The image and camera intrinsic parameter input module is configured to acquire a monocular road scene image and camera intrinsic parameters corresponding to the monocular road scene image; The visual base model feature extraction module is configured to input the monocular road scene image into the visual base model with frozen original parameters, and obtain task-adapted visual features through a trainable low-rank residual branch set in the transformer attention projection branch of the visual base model. The depth feature extraction module is configured to input the monocular road scene image into a depth base model with frozen original parameters, extract dense depth representations, and encode the dense depth representations to obtain depth features. The depth-guided transformer decoding module is configured to perform joint decoding based on multiple learnable target queries according to the visual features and the depth features, and generate multiple depth-aware target query features. The detection attribute prediction module is configured to predict target-related information based on the target query features. The target-related information includes target category, projection coordinate parameters of the target's three-dimensional center on the image plane, target's three-dimensional size, target orientation, target's physical height, target's projection height on the image plane, and depth residual. The geometric complementary depth estimation module is configured to calculate the geometric depth anchor point according to the pinhole camera projection relationship based on the target physical height, the projected height and the vertical focal length in the camera intrinsic parameters, and add the geometric depth anchor point to the depth residual to obtain the final depth; The 3D detection result output module is configured to back-project the target's 3D center coordinates in the camera coordinate system based on the final depth, the camera intrinsic parameters, and the projection coordinate parameters, and construct the target's 3D bounding box based on the 3D center coordinates, the 3D size, and the orientation.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the monocular three-dimensional detection method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the monocular three-dimensional detection method according to any one of claims 1-7.

Images