A monocular three-dimensional target detection method based on an enhanced depth predictor

Abstract

The application discloses a monocular three-dimensional target detection method based on an enhanced depth predictor, and comprises the following steps: acquiring an image to be detected; constructing a monocular three-dimensional target detection model for detecting a single target in the image respectively; training the monocular three-dimensional target detection model to obtain a trained monocular three-dimensional target detection model; and inputting the image to be detected into the trained monocular three-dimensional target detection model to realize detection of the single target in the image. The monocular three-dimensional target detection method based on the enhanced depth predictor effectively solves the problems of misalignment between backbone features and depth tasks and spatial inconsistency of depth features. The application realizes integration of visual perception and geometric estimation, significantly improves the quality and reliability of depth information, and provides an effective solution for constructing a robust three-dimensional vision system independent of expensive sensors.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, and relates to a monocular 3D target detection method based on an enhanced depth predictor. Background Technology

[0002] Monocular 3D object detection plays an irreplaceable role in key scenarios such as autonomous driving, intelligent monitoring, and robot navigation due to its high cost-effectiveness, strong deployment flexibility, and ability to provide rich semantic information such as color, texture, and contour. However, traditional monocular 3D object detection methods have significant limitations in depth information estimation: their depth perception mechanisms are relatively simple, and the modeling of scene geometry is insufficient, resulting in inadequate depth estimation accuracy, which severely restricts the improvement of overall 3D detection performance.

[0003] With the continuous advancement of deep learning technology, deep learning-based monocular 3D object detection methods have achieved effective extraction of image and depth features through the design of dedicated network architectures, significantly improving detection accuracy. However, existing methods still face challenges such as limited depth feature extraction capabilities, significant influence of depth errors on 3D detection results, and long model training cycles. These technical bottlenecks collectively restrict further improvements in the performance of monocular 3D object detection, necessitating the development of novel detection technologies to overcome these limitations. Summary of the Invention

[0004] To address the above problems, the technical solution adopted by this invention is: a monocular 3D target detection method based on an enhanced depth predictor, comprising the following steps:

[0005] Acquire the image to be detected; A monocular 3D target detection model is constructed to detect individual targets in an image. The monocular 3D target detection model is trained to obtain a trained monocular 3D target detection model. The image to be detected is input into a trained monocular 3D object detection model to detect a single object in the image.

[0006] Furthermore, the monocular 3D target detection model includes: Visual feature extraction module: used to extract multi-scale feature maps of monocular images, wherein the multi-scale feature maps include multiple feature maps with progressively decreasing detail information; the detail information includes at least target edges and texture features; Visual feature encoder: Based on the multi-scale feature map of the monocular image in the image transmitted by the visual feature extraction module, the encoder uses multi-layer Transformer blocks to perform global semantic modeling of visual features, extracts high-order appearance and context information, and provides strong semantic support for 3D detection. Enhanced Depth Predictor Module: Used to extract depth-sensing features from monocular images and predict the 3D position of objects; Depth information refinement module: Based on the depth perception features transmitted by the enhanced depth predictor module and the multi-scale feature map transmitted by the visual feature encoding module, it combines visual features to perform feature aggregation and context modeling in the depth dimension, corrects inaccurate or low-confidence depth predictions, and provides reliable 3D feature alignment capability for view transformation. Depth Feature Encoder: Used to process the depth features transmitted by the depth refinement module, and to perform global geometric modeling of the depth features using a lightweight Transformer, thereby mining long-range depth dependencies between foreground regions and providing robust non-local spatial structure information for 3D detection. Decoding module guided by depth information: Based on the features transmitted by the depth feature encoding module and the visual features transmitted by the visual feature encoder, the module utilizes scene-level depth and visual information and a multi-stage attention mechanism to gradually integrate geometric and semantic information into the learnable object query, thereby achieving more accurate monocular 3D object detection. The detection head network is used to decode high-level, abstract features transmitted by the depth-information-guided decoding module into physical properties that can be used to construct 3D bounding boxes, generate 3D detection boxes for target objects, and ultimately achieve monocular target detection in images.

[0007] Furthermore, the depth-information-guided decoding module includes: The first Transformer decoder block: Based on the features transmitted by the depth feature encoding module and the visual features transmitted by the visual feature encoder, it focuses on the basic interaction between depth information and object query, allowing each object query to initially capture spatial clues guided by depth. Second Transformer Decoder Block: Used to integrate the relationships between the first integrated objects based on the features output by the first Transformer Decoder Block; The third Transformer decoder block: Based on the relationships between objects output by the second Transformer decoder block, it performs deep fusion of depth information and visual information to form a comprehensive understanding of the 3D scene.

[0008] Furthermore, the first Transformer decoder block, the second Transformer decoder block, and the third Transformer decoder block have the same structure; The first Transformer decoder block includes a deep cross-attention layer, an inter-query self-attention layer, a visual cross-attention layer, and a feedforward neural network; Deep cross-attention layer: used to enable each object query to perceive regions related to its own depth, achieving depth-guided feature extraction; Inter-query self-attention layer: This layer is used to transmit features based on the depth cross-attention layer, enabling different object queries to exchange information with each other, which helps the model understand the relative position and depth relationship between objects. Visual cross-attention layer: used to collect rich visual semantic information based on the features transmitted by the self-attention layer between queries, add visual context to each object query, and use it for category prediction and bounding box fine-tuning; Through a feedforward neural network, the features transmitted by the visual cross-attention layer undergo nonlinear transformation and further processing to generate the final feature representation for each object query.

[0009] Furthermore, the depth information refinement module includes: Feature enhancement submodule: Performs non-linear feature enhancement on the visual features output by the visual feature encoding module to improve its expressive power and provide richer and more discriminative cues for the optimization of depth information; Detail optimization submodule: The enhanced features output by the feature enhancement submodule and the depth features output by the enhanced depth predictor are concatenated using two cascaded residual blocks; each residual block contains two 3×3 convolutional layers with a ReLU activation function in between.

[0010] Furthermore, both the deep feature encoding module and the visual feature encoding module include a Transformer block, which is used to allow each feature point to interact with all other feature points within the feature map based on a self-attention mechanism.

[0011] Furthermore, the Transformer block includes: Self-attention layer: used to calculate the correlation between each element in the sequence and other elements, and to assign different weights to each element; Normalization layer: The elements of the output of the self-attention layer are assigned different weights for normalization, so that the mean and variance of each sample remain consistent between layers. Feedforward network layer: Used to standardize the output samples of the normalization layer, perform nonlinear transformation, and extract higher-level feature representations.

[0012] Furthermore, the enhanced depth predictor module includes: Large kernel convolution module: Used to significantly increase the effective receptive field based on large kernel convolution, enabling the model to capture a wider range of local contextual information; Global Feature Enhancement Module: Based on the features processed by the large kernel convolution module, it enhances the features related to depth prediction, suppresses noise interference in irrelevant regions, and improves the stability of the model in complex scenarios such as occlusion or blurring, thereby comprehensively improving the network's ability to perceive the distance relationship of objects and its overall robustness. Average pooling layer: Based on the enhanced local features transmitted by the global feature enhancement module, image pooling is performed to extract low-resolution spatial structure information, providing spatial anchors for subsequent feature fusion; Furthermore, the visual feature extraction module includes a residual network, ResNet-50.

[0013] A monocular 3D target detection device based on an enhanced depth predictor, comprising: Acquisition module: Used to acquire the image to be detected; Module: Used to build a monocular 3D object detection model, which is used to detect individual objects in an image. Training module: Used to train the monocular 3D object detection model to obtain a trained monocular 3D object detection model; Implementation module: This module is used to input the image to be detected into the trained monocular 3D target detection model to detect a single target in the image.

[0014] The present invention provides a monocular 3D target detection method based on an enhanced depth predictor, which has the following advantages: (1) The enhanced depth predictor module directly extracts depth features from the original image, avoiding reliance on intermediate features of the backbone. It performs downsampling through three levels and extracts multi-level depth features by fusing cross-stage features with a large kernel convolution module. The module directly uses the image data itself to extract depth-related features, bypassing the secondary processing of backbone derived features, thereby reducing the possibility of misaligned learning during model training. The module performs downsampling through multi-stage convolution operations to preserve spatial information and enhance hierarchical feature expression. (2) Through the depth information refinement module, the input visual features are first nonlinearly enhanced through feature enhancement to improve their expressive power, and then concatenated with the depth features. The concatenated features are then fed into two residual blocks for depth information correction. This module takes depth estimation and image features as input, and through a network containing multiple convolutions, it gradually fuses visual information and expands the receptive field, ultimately outputting a more accurate and spatially consistent refined depth map. During training, Gaussian noise is added to the ground truth bounding box, depth value, and class label, and the model learns how to "de-noise" to restore the original annotation, thereby enhancing the model's ability to model the relationship between target localization and classification, accelerating the model's convergence speed, and ultimately improving detection performance. This invention effectively solves the problem of misalignment between backbone features and depth tasks, and spatial inconsistency of depth features, through a monocular 3D target detection model based on an enhanced depth predictor.

[0015] (3) During training, Gaussian noise is applied to the real labels to simulate various errors that may occur during model inference, allowing the monocular 3D object detection model to learn some general learning abilities, effectively reducing the instability of bipartite graph matching. Gaussian noise is added to the ground truth bounding box, depth value and category label during training, and the monocular 3D object detection model learns how to "de-noise" and restore the original labels, thereby enhancing the monocular 3D object detection model's ability to model the relationship between object localization and classification, accelerating the model's convergence speed, and ultimately improving detection performance.

[0016] This invention effectively solves the problems of misalignment between backbone features and depth tasks, and spatial inconsistency of depth features, through a monocular 3D target detection model based on an enhanced depth predictor.

[0017] This invention re-examines the depth estimation problem in monocular 3D detection from a mechanistic perspective, proposing a coupled optimization framework for feature extraction, information fusion, and training strategies. (An enhanced depth predictor is used for feature extraction, and a depth information refinement module is used to fuse visual and depth features; the modules are relatively independent, hence the coupled optimization framework.) This framework significantly improves the quality and reliability of depth information by integrating visual perception and geometric estimation, providing an effective solution for building robust 3D vision systems that do not rely on expensive sensors. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an overall flowchart of a monocular 3D target detection method based on an enhanced depth predictor according to the present invention; Figure 2 This is a diagram of the overall network framework. Figure 3 This is a schematic diagram of the network structure of the depth information refinement module of the present invention; Figure 4 This is a schematic diagram of the network structure of the enhanced depth predictor module of the present invention; Figure 5 The diagram shows the results of monocular 3D target detection on the KITTI dataset using the present invention; (a) original image I and processing result, (b) original image II and processing result, and (c) original image III and processing result. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Figure 1 This is an overall flowchart of a monocular 3D target detection method based on an enhanced depth predictor according to the present invention; A monocular 3D target detection method based on an enhanced depth predictor includes the following steps: S1: Acquire the image to be detected; S2: Construct a monocular 3D target detection model to detect individual targets in the image; S3: Train the monocular 3D object detection model to obtain a trained monocular 3D object detection model; obtain the KITTI dataset for monocular 3D object detection; divide the downloaded training set into a training set and a validation set according to a preset ratio, and use the default settings for the test set; train the monocular 3D object detection model based on the training set. S4: Input the image to be detected into the trained monocular 3D target detection model to detect a single target in the image.

[0023] Steps S1 / S2 / S3 / S4 are executed sequentially; Figure 2 This is a diagram of the overall network framework. The monocular 3D target detection model includes: Visual feature extraction module: used to extract multi-scale feature maps of monocular images, wherein the multi-scale feature maps include multiple feature maps with progressively decreasing detail information; the detail information includes at least target edges and texture features; Visual feature encoder: Based on the multi-scale feature map of the monocular image in the image transmitted by the visual feature extraction module, the encoder uses multi-layer Transformer blocks to perform global semantic modeling of visual features, extracts high-order appearance and context information, and provides strong semantic support for 3D detection. Enhanced Depth Predictor Module: Used to extract depth-sensing features from monocular images and predict the 3D position of objects; Depth information refinement module: Based on the depth perception features transmitted by the enhanced depth predictor module and the multi-scale feature map transmitted by the visual feature encoding module, it combines visual features to perform feature aggregation and context modeling in the depth dimension, corrects inaccurate or low-confidence depth predictions, and provides reliable 3D feature alignment capability for view transformation. Depth Feature Encoder: Used to process the depth features transmitted by the depth refinement module, and to perform global geometric modeling of the depth features using a single-layer Transformer to mine the long-range depth dependencies between foreground regions, providing robust non-local spatial structure information for 3D detection; Decoding module guided by depth information: Based on the features transmitted by the depth feature encoding module and the visual features transmitted by the visual feature encoder, the module utilizes scene-level depth and visual information and a multi-stage attention mechanism to gradually integrate geometric and semantic information into the learnable object query, thereby achieving more accurate monocular 3D object detection. The detection head network is used to decode high-level, abstract features transmitted by the depth-information-guided decoding module into physical properties that can be used to construct 3D bounding boxes, generate 3D detection boxes for target objects, and ultimately achieve monocular target detection in images.

[0024] Furthermore, the depth-information-guided decoding module includes: The first Transformer decoder block: Based on the features transmitted by the depth feature encoding module and the visual features transmitted by the visual feature encoder, it focuses on the basic interaction between depth information and object query, allowing each object query to initially capture spatial clues guided by depth. Second Transformer Decoder Block: Used to integrate the relationships between the first integrated objects based on the features output by the first Transformer Decoder Block; The third Transformer decoder block: Based on the relationships between objects output by the second Transformer decoder block, it performs deep fusion of depth information and visual information to form a comprehensive understanding of the 3D scene.

[0025] Furthermore, the first Transformer decoder block, the second Transformer decoder block, and the third Transformer decoder block have the same structure; The first Transformer decoder block includes a deep cross-attention layer, an inter-query self-attention layer, a visual cross-attention layer, and a feedforward neural network; Deep cross-attention layer: used to enable each object query to perceive regions related to its own depth, achieving depth-guided feature extraction; Inter-query self-attention layer: This layer is used to transmit features based on the depth cross-attention layer, enabling different object queries to exchange information with each other, which helps the model understand the relative position and depth relationship between objects. Visual cross-attention layer: used to collect rich visual semantic information based on the features transmitted by the self-attention layer between queries, add visual context to each object query, and use it for category prediction and bounding box fine-tuning; Through a feedforward neural network, the features transmitted by the visual cross-attention layer undergo nonlinear transformation and further processing to generate the final feature representation for each object query.

[0026] Figure 3 This is a schematic diagram of the network structure of the depth information refinement module of the present invention; Furthermore, the depth information refinement module includes: Feature enhancement submodule: Performs non-linear feature enhancement on the visual features output by the visual feature encoding module to improve its expressive power and provide richer and more discriminative cues for the optimization of depth information; Detail optimization submodule: The enhanced features output by the feature enhancement submodule and the depth features output by the enhanced depth predictor are concatenated using two cascaded residual blocks; each residual block contains two 3×3 convolutional layers with a ReLU activation function in between.

[0027] Furthermore, both the deep feature encoding module and the visual feature encoding module include a Transformer block, which is used to allow each feature point to interact with all other feature points within the feature map based on a self-attention mechanism.

[0028] Further: The Transformer block includes: Self-attention layer: used to calculate the correlation between each element in the sequence and other elements, and to assign different weights to each element; Normalization layer: The elements of the output of the self-attention layer are assigned different weights for normalization, so that the mean and variance of each sample remain consistent between layers. Feedforward network layer: Used to standardize the output samples of the normalization layer, perform nonlinear transformation, and extract higher-level feature representations.

[0029] Figure 4 This is a schematic diagram of the network structure of the enhanced depth predictor module of the present invention; Furthermore, the enhanced depth predictor module includes: Large kernel convolution module: Used to significantly increase the effective receptive field based on large kernel convolution, enabling the model to capture a wider range of local contextual information; Global Feature Enhancement Module: Based on the features processed by the large kernel convolution module, it enhances the features related to depth prediction, suppresses noise interference in irrelevant regions, and improves the stability of the model in complex scenarios such as occlusion or blurring, thereby comprehensively improving the network's ability to perceive the distance relationship of objects and its overall robustness. Average pooling layer: Based on the enhanced local features transmitted by the global feature enhancement module, image pooling is performed to extract low-resolution spatial structure information, providing spatial anchors for subsequent feature fusion; Furthermore, the visual feature extraction module includes a residual network, ResNet-50.

[0030] A monocular 3D target detection device based on an enhanced depth predictor, comprising: Acquisition module: Used to acquire the image to be detected; Module: Used to build a monocular 3D object detection model, which is used to detect individual objects in an image. Training module: Used to train the monocular 3D object detection model to obtain a trained monocular 3D object detection model; Implementation module: This module is used to input the image to be detected into the trained monocular 3D target detection model to detect a single target in the image.

[0031] Example 1 A monocular 3D target detection method based on an enhanced depth predictor includes the following steps: S1: Acquire the image to be detected; S2: Construct a monocular 3D target detection model to detect individual targets in the image; S3: Train the monocular 3D object detection model to obtain the trained monocular 3D object detection model; the images in this example are from the dataset publicly available on the KITTI website, including 7,481 training images and 7,518 test images. Following the preset rules, 3,769 validation images were split from the training set as the validation set. S4: Input the image to be detected into the trained monocular 3D target detection model to detect a single target in the image.

[0032] The monocular 3D target model includes an enhanced depth predictor module, a visual feature extraction module, a depth information refinement module, a depth feature encoder, a visual feature encoding module, a depth information-guided decoding module, and a detection head network; Step 200: The enhanced depth predictor module processes the image from three dimensions: input processing, hierarchical feature extraction, and information fusion. First, the original image is pooled and then fed into the initial feature extraction module. Convolution is used for preliminary dimensionality reduction and channel transformation to extract shallow features such as basic edges and textures. The output of the initial feature extraction module is concatenated with the pooled image features along the channel dimension to generate the first-stage fusion feature. This fusion feature is then subjected to downsampling and local multi-scale feature enhancement, and subsequently concatenated again with the pooled image features along the channel dimension to form the second-stage fusion feature.

[0033] The enhanced deep predictor module built in the specific example includes an average pooling layer, an initial feature extraction module, a concatenation layer, a downsampling module, a large kernel convolution module, and a global feature enhancement module; the average pooling layer, the initial feature extraction module, the concatenation layer, the downsampling module, the large kernel convolution module, and the global feature enhancement module are connected sequentially. The average pooling layer extracts low-resolution spatial structure information by downsampling the original image, providing "spatial anchors" for subsequent feature fusion; The initial feature extraction module performs preliminary dimensionality reduction and channel transformation on the original image through convolution, extracting shallow features such as basic edges and textures. The concatenation layer greatly reduces the spatial information loss caused by downsampling by concatenating convolutional features with pooled features along the channel dimension.

[0034] The downsampling module downsamples the preceding fused features through convolution, reducing the resolution while increasing the number of channels, compressing the spatial dimension, and improving the semantic abstraction level.

[0035] The large kernel convolution module implements 7x7 convolution using depthwise separable convolution, which significantly reduces the amount of computation and parameters with almost no loss of expressive power. At the same time, it increases the receptive field to capture a wider range of contextual information. Combined with residual connections, it greatly enhances multi-scale local features.

[0036] The global feature enhancement module enhances features relevant to depth prediction, suppresses noise interference from irrelevant regions, and improves the model's stability in complex scenarios such as occlusion or blurring, thereby comprehensively improving the network's ability to perceive the distance relationship of objects and its overall robustness.

[0037] This example demonstrates how an enhanced depth predictor module can be directly constructed after the input image. Through multiple downsampling operations, feature enhancement and fusion are performed to obtain feature maps at different scales, significantly improving the ability to extract depth features. Specifically, the enhanced depth predictor module is as follows: Figure 3As shown, the original image (H×W×3) is directly input into this module. Initial feature extraction is performed on the image. The original image is converted into a feature map of H / 2×W / 2×C1 through a 3x3 convolution with a stride of 2, performing preliminary dimensionality reduction and channel transformation. This convolutional feature is concatenated with the image features after average pooling along the channel dimension to obtain the fused feature. A 3x3 convolution (with a stride of 2) is then performed on this fused feature, followed by the large kernel convolution module. First, a depthwise separable convolution (7x7) is used to expand the receptive field without significantly increasing the computational cost, enhancing local multi-scale features. Specifically, the depthwise separable convolution (7x7) is implemented by stacking three 3x3 depthwise convolutions and then adding a 1x1 pointwise convolution. The first 3x3 depthwise convolution (each channel is convolved separately) results in a receptive field of 3x3. The second 3x3 depthwise convolution expands the receptive field to 5x5. The third 3x3 depthwise convolution expands the receptive field to 7x7. Finally, a 1x1 convolution is added to map the input channels to the output channels, achieving cross-channel fusion. To ensure feature nonlinearity and training stability, batch normalization and activation functions are added after each depthwise convolution. The feature output is then residually connected to the input to enhance feature transfer and reuse capabilities. After the output from the large-kernel convolution module, the global feature enhancement module strengthens depth-related features and suppresses noise, improving the model's depth perception capability in complex scenes. It is then concatenated again with the features from the original image after average pooling along the channel dimension to compensate for spatial information loss caused by downsampling. After concatenation, downsampling is performed again, processed by the large-kernel convolution module and then by the global feature enhancement module. This process is repeated twice to gradually reduce the resolution to H / 16×W / 16×C, achieving spatial dimension compression and semantic level enhancement, ensuring that the output features contain both local details and global structure.

[0038] Step 201: Extract visual features from monocular images using the visual feature extraction module, gradually abstracting from low-level features of the original image to high-level features. When extracting features, residual connections are used to overcome the training bottleneck of deep networks, and the Bottleneck structure is used to balance efficiency and performance.

[0039] Specifically, this paper uses ResNet50 as the backbone network for extracting visual features. The feature extraction process is mainly divided into five stages. First, initial convolutions are performed on the image to reduce its size and initially extract basic low-level visual features such as edges and colors. The next four stages use stacked residual blocks to progressively downsample and increase the number of channels, achieving feature abstraction from low to high levels. Each residual block uses a structure of "1x1 convolution for dimensionality reduction -> 3x3 convolution for feature extraction -> 1x1 convolution for dimensionality increase." This reduces computational cost while preserving features and mitigating gradient vanishing through residual connections, ensuring the effective transfer of deep features.

[0040] Step 202: Through the depth information refinement module, the feature enhancement part performs non-linear enhancement on the input visual features to improve their expressive power and provide richer and more discriminative cues for the optimization of depth information. The data is then fed into two residual blocks (each containing two convolutional layers) in series. A layer-polarized feature interaction mechanism promotes the progressive fusion and complementarity of deep and visual features. This process, while preserving the inherent distribution patterns of deep features, leverages refined visual information to achieve high-precision optimization and representation enhancement of deep features.

[0041] In a specific embodiment, the constructed depth information refinement module includes a feature processing module composed of convolutional layers, splicing layers, residual connection layers, activation function layers, etc. This example maps visual features to depth features and refines the depth features in stages through multi-layer convolution, specifically including: Feature extraction is performed using 5x5 depthwise separable convolutions, followed by batch normalization, channel upscaling, activation functions, and channel downscaling to enhance the nonlinearity of visual features.

[0042] The interaction between visual and depth features is achieved through two residual blocks. The first residual block mainly smooths local noise, corrects abrupt changes in depth, and preserves object edges. The second residual block mainly corrects the depth of occluded areas and uses visual context to correct systematic errors. The residual connection mainly constrains physical continuity to ensure that changes in depth values ​​conform to physical laws.

[0043] In this example, the depth information refinement module is as follows: Figure 4As shown, after the depth features are extracted, this module refines them based on existing visual features, improving local noise and errors in occluded areas to make the depth features more accurate. Specifically, this module has two inputs: the output of the enhanced depth predictor module and visual features from the penultimate layer of the autonomous backbone network. First, a 5x5 depthwise separable convolution is used for feature extraction. Then, batch normalization, channel upscaling, activation functions, and channel downscaling are applied to map the visual features onto the depth features, while simultaneously learning channel interactions. Next, a first 3x3 convolutional layer suppresses noise and captures the correspondence between depth values ​​and visual gradients. A ReLU activation function further enhances the nonlinear feature representation. After a second 3x3 convolutional layer, the receptive field is increased, and the interaction between depth and visual features is modeled to achieve visually guided depth correction, further enhancing noise suppression. Activation functions are then used to further enhance the nonlinear feature representation, followed by residual connections to increase physical continuity constraints. Next, we move to the second residual block, which also consists of two convolutions. The receptive field is further expanded. The first convolution performs contextual modeling of the occluded area, correcting most of the occlusion errors. The last convolution has an even larger receptive field (9x9), which can observe both the occluded and background areas. Combined with residual connections, it specifically handles the problem of residual abrupt changes, ensuring that the depth values ​​are both accurate and physically consistent under visual guidance.

[0044] Step 203: Through the visual feature and depth feature encoding module, the input visual information is processed in depth to mine high-level semantic information, capture complex appearance features, and establish the relationship between distant objects in the image through the self-attention mechanism of Transformer; the input depth information is processed in a targeted manner to construct 3D geometric cognition, establish cross-regional depth connections and provide spatial prior information through the self-attention mechanism, such as implicit occlusion relationships, drivable areas, obstacle boundaries and other spatial cues that are difficult to reliably obtain through pure vision.

[0045] In specific examples, the constructed visual feature encoder and deep feature encoder are mainly composed of Transformer blocks, which are mainly composed of self-attention layers, feedforward networks, and normalization layers.

[0046] The Transformer block includes: Self-attention layer: used to calculate the correlation between each element in the sequence and other elements, and to assign different weights to each element; Normalization layer: The elements of the output of the self-attention layer are assigned different weights for normalization, so that the mean and variance of each sample remain consistent between layers. Feedforward network layer: Used to standardize the output samples of the normalization layer, perform nonlinear transformation, and extract higher-level feature representations.

[0047] In this example, the visual encoder uses three Transformer blocks because RGB images contain rich and complex appearance information such as texture, color, lighting, and semantics, requiring a deeper network to fully model them. The deep feature encoder uses one Transformer block because depth information is relatively simpler in structure and sparser in information compared to visual features. These two encoders are also decoupled, allowing each branch to focus on its own modal characteristics. The depth branch learns geometric structure and spatial layout, while the visual branch learns texture, semantics, object categories, etc. Ultimately, these two complementary perspectives can be combined in the decoder module to achieve stronger scene understanding capabilities.

[0048] Step 204: A depth-guided decoder module, leveraging a depth cross-attention mechanism, fuses depth and visual information, solving the core challenge of inferring 3D structure from 2D images. This design not only improves the accuracy and robustness of 3D object detection but also provides a general modeling paradigm for multimodal perception tasks. Incorporating a denoising task during training bypasses the Hungarian matching algorithm, providing stable supervision signals and allowing the model to learn how to correct from an initial position close to the ground truth to a precise location, accelerating model convergence and training stability.

[0049] Specifically, the Transformer block consists of a deep cross-attention layer, a query-to-query self-attention layer, a visual cross-attention layer, and a feedforward network.

[0050] By using a deep cross-attention layer, object queries can adaptively capture spatial cues in deep features, achieving deep-guided scene-level understanding. By modeling the relationships between different object queries using a self-attention layer between queries, we can understand the spatial interactions between objects; Semantic information is collected from visual features through a visual cross-attention layer, which supplements the limitations of deep information.

[0051] The expressive power of the model is enhanced by using a feedforward network to perform nonlinear feature transformation and fusion.

[0052] By forcing the model to learn to recover the original target from the perturbation through the denoising task, the model can gain a deeper understanding of the context, significantly improve the accuracy of localization and classification, generalization ability and convergence speed, and effectively enhance the robustness of the model in complex scenarios.

[0053] In this example, the depth-guided decoder module uses three Transformer decoder blocks: the first Transformer decoder block, the second Transformer decoder block, and the third Transformer decoder block. The first Transformer decoding block focuses on the basic interaction between depth information and object queries, allowing each object query to initially capture spatial clues guided by depth. The second Transformer decoding block begins to integrate the relationships between objects and further fuses depth and visual information; The third Transformer decoding block is responsible for the deep fusion of depth and visual information to form a comprehensive understanding of the 3D scene.

[0054] The first Transformer decoder block, the second Transformer decoder block, and the third Transformer decoder block are connected sequentially; This example introduces a denoising task. By applying a controlled perturbation to the ground truth bounding box as an explicit query input, it effectively bypasses the discreteness and instability inherent in the traditional DETR framework, which relies on Hungarian matching for prediction and ground truth assignment. Specifically, this design provides the model with an auxiliary supervision path with a clear optimization objective: each noisy query naturally corresponds to a unique ground truth value, eliminating the need for a dynamic and gradient-sensitive bipartite graph matching process. As a result, the model can focus on learning the relative offset from approximate initial location to the precise target location, significantly improving the accuracy and robustness of location regression. Simultaneously, this stable and semantically consistent supervision signal effectively alleviates the optimization objective conflict caused by drastic jumps in matching results between training iterations, accelerating model convergence and enhancing overall stability during training.

[0055] Step 205: The detection head network transforms abstract features into specific task attributes. By assigning tasks to sub-heads, it adapts to the learning needs of different attributes, handles attribute specificities, and ensures reasonable predictions. Specifically, in this example, the detection head network distributes the encoder's output to different detection heads to output prediction results for different attributes, such as category, 2D location, object center point, depth, 3D size, and orientation.

[0056] The monocular 3D object detection model constructed in this embodiment, through enhanced depth predictor, depth information refinement module, and denoising training, achieves significantly higher detection accuracy than current advanced monocular 3D object detection models. After training, by inputting a test image, the optimal detection model obtained in this embodiment determines the location and category of the target in the image. Furthermore, during testing, by inputting any image, the monocular 3D object detection result can be directly obtained based on the trained model parameters, such as... Figure 5 As shown.

[0057] Figure 5 The diagram shows the results of monocular 3D target detection on the KITTI dataset using the present invention; (a) original image I and processing result, (b) original image II and processing result, and (c) original image III and processing result.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A monocular 3D target detection method based on an enhanced depth predictor, characterized in that, Including the following steps: Acquire the image to be detected; A monocular 3D target detection model is constructed to detect individual targets in an image. The monocular 3D target detection model is trained to obtain a trained monocular 3D target detection model. The image to be detected is input into a trained monocular 3D object detection model to detect a single object in the image.

2. The monocular 3D target detection method based on an enhanced depth predictor according to claim 1, characterized in that, The monocular 3D target detection model includes: Visual feature extraction module: used to extract multi-scale feature maps of monocular images, wherein the multi-scale feature maps include multiple feature maps with progressively decreasing detail information; the detail information includes at least target edges and texture features; Visual feature encoder: Based on the multi-scale feature map of the monocular image in the image transmitted by the visual feature extraction module, the encoder uses multi-layer Transformer blocks to perform global semantic modeling of visual features, extracts high-order appearance and context information, and provides strong semantic support for 3D detection. Enhanced Depth Predictor Module: Used to extract depth-sensing features from monocular images and predict the 3D position of objects; Depth information refinement module: Based on the depth perception features transmitted by the enhanced depth predictor module and the multi-scale feature map transmitted by the visual feature encoding module, it combines visual features to perform feature aggregation and context modeling in the depth dimension, corrects inaccurate or low-confidence depth predictions, and provides reliable 3D feature alignment capability for view transformation. Depth Feature Encoder: Used to process the depth features transmitted by the depth refinement module, and to perform global geometric modeling of the depth features using a lightweight Transformer, thereby mining long-range depth dependencies between foreground regions and providing robust non-local spatial structure information for 3D detection. Decoding module guided by depth information: Based on the features transmitted by the depth feature encoding module and the visual features transmitted by the visual feature encoder, the module utilizes scene-level depth and visual information and a multi-stage attention mechanism to gradually integrate geometric and semantic information into the learnable object query, thereby achieving more accurate monocular 3D object detection. The detection head network is used to decode high-level, abstract features transmitted by the depth-information-guided decoding module into physical properties that can be used to construct 3D bounding boxes, generate 3D detection boxes for target objects, and ultimately achieve monocular target detection in images.

3. A monocular 3D target detection method based on an enhanced depth predictor according to claim 1, characterized in that, The depth-information-guided decoding module includes: The first Transformer decoder block: Based on the features transmitted by the depth feature encoding module and the visual features transmitted by the visual feature encoder, it focuses on the basic interaction between depth information and object query, allowing each object query to initially capture spatial clues guided by depth. Second Transformer Decoder Block: Used to integrate the relationships between the first integrated objects based on the features output by the first Transformer Decoder Block; The third Transformer decoder block: Based on the integrated relationships between objects output by the second Transformer decoder block, it performs deep fusion of depth information and visual information to form a comprehensive understanding of the 3D scene.

4. A monocular 3D target detection method based on an enhanced depth predictor according to claim 3, characterized in that: The first Transformer decoder block, the second Transformer decoder block, and the third Transformer decoder block have the same structure; The first Transformer decoder block includes a deep cross-attention layer, an inter-query self-attention layer, a visual cross-attention layer, and a feedforward neural network; Deep cross-attention layer: used to enable each object query to perceive regions related to its own depth, achieving depth-guided feature extraction; Inter-query self-attention layer: This layer is used to transmit features based on the depth cross-attention layer, enabling different object queries to exchange information with each other, which helps the model understand the relative position and depth relationship between objects. Visual cross-attention layer: used to collect rich visual semantic information based on the features transmitted by the self-attention layer between queries, add visual context to each object query, and use it for category prediction and bounding box fine-tuning; Through a feedforward neural network, the features transmitted by the visual cross-attention layer undergo nonlinear transformation and further processing to generate the final feature representation for each object query.

5. The monocular 3D target detection method based on an enhanced depth predictor according to claim 2, characterized in that, The depth information refinement module includes: Feature enhancement submodule: Performs non-linear feature enhancement on the visual features output by the visual feature encoding module to improve its expressive power and provide richer and more discriminative cues for the optimization of depth information; Detail optimization submodule: The enhanced features output by the feature enhancement submodule and the depth features output by the enhanced depth predictor are concatenated using two cascaded residual blocks; each residual block contains two 3×3 convolutional layers with a ReLU activation function in between.

6. The monocular 3D target detection method based on an enhanced depth predictor according to claim 2, characterized in that, Both the deep feature encoding module and the visual feature encoding module include a Transformer block, which is used to allow each feature point to interact with all other feature points within the feature map based on a self-attention mechanism.

7. The monocular 3D target detection method based on an enhanced depth predictor according to claim 6, characterized in that: The Transformer block includes: Self-attention layer: used to calculate the correlation between each element in the sequence and other elements, and to assign different weights to each element; Normalization layer: The elements of the output of the self-attention layer are assigned different weights for normalization, so that the mean and variance of each sample remain consistent between layers. Feedforward network layer: Used to standardize the output samples of the normalization layer, perform nonlinear transformation, and extract higher-level feature representations.

8. The monocular 3D target detection method based on an enhanced depth predictor according to claim 2, characterized in that, The enhanced depth predictor module includes: Large kernel convolution module: Used to significantly increase the effective receptive field based on large kernel convolution, enabling the model to capture a wider range of local contextual information; Global Feature Enhancement Module: Based on the features processed by the large kernel convolution module, it enhances the features related to depth prediction, suppresses noise interference in irrelevant regions, and improves the stability of the model in complex scenarios such as occlusion or blurring, thereby comprehensively improving the network's ability to perceive the distance relationship of objects and its overall robustness. Average pooling layer: Based on the enhanced local features transmitted by the global feature enhancement module, image pooling is performed to extract low-resolution spatial structure information, providing spatial anchors for subsequent feature fusion.

9. The monocular three-dimensional target detection method based on accurate depth information according to claim 2, characterized in that, The visual feature extraction module includes a residual network, ResNet-50.

10. A monocular 3D target detection device based on an enhanced depth predictor, characterized in that, include: Acquisition module: Used to acquire the image to be detected; Module: Used to build a monocular 3D object detection model, which is used to detect individual objects in an image. Training module: Used to train the monocular 3D object detection model to obtain a trained monocular 3D object detection model; Implementation module: This module is used to input the image to be detected into the trained monocular 3D target detection model to detect a single target in the image.

Images