Monocular 3D detection method, device and storage medium

Abstract

The application provides a monocular 3D detection method, device and storage medium. The monocular 3D detection method comprises: performing depth estimation on an input single two-dimensional image to generate a depth map fused with semantic information; projecting and fusing the depth map and the original two-dimensional image into an aerial view coordinate system to generate an aerial view feature map of a current frame; mapping historical aerial view features to a current frame coordinate system, performing weighted fusion on the aerial view feature map of the current frame and the historical aerial view features, and outputting a time-series enhanced fused feature map; and decoding the fused feature map to output three-dimensional detection information of each target in a scene. The application can improve depth quality and time-series stability under a low-cost sensor configuration and simplify a detection process.

Description

Technical Field

[0001] This application belongs to the technical field of environmental perception, and relates to a monocular 3D detection method, device and storage medium. Background Technology

[0002] In recent years, improving environmental perception capabilities has become particularly crucial in autonomous driving technology. Three-dimensional object detection, as a core component of autonomous driving perception systems, directly impacts vehicle driving safety and decision-making.

[0003] While traditional two-dimensional target detection technologies are relatively mature, they cannot provide information on the target's distance, size, and orientation in three-dimensional space, making them insufficient for the complex environment perception requirements of autonomous driving. Therefore, researchers have turned to using three-dimensional sensing data for target detection.

[0004] Currently, the most commonly used 3D perception solutions in the industry mainly fall into two categories: LiDAR point clouds and camera images.

[0005] (1) LiDAR-based solutions can acquire high-precision dense point clouds, but the sensors are expensive and their performance drops sharply in severe weather conditions such as rain and snow, which limits their application in mass-produced vehicles.

[0006] (2) Camera-based solutions are low-cost and stable under various lighting conditions, so using vehicle-mounted cameras to achieve three-dimensional environmental perception has become an important research direction.

[0007] However, 3D object detection using a monocular camera faces an inherent challenge: how to recover lost depth information from a single 2D image. Summary of the Invention

[0008] This application provides a monocular 3D detection method, device, and storage medium to address the problem of improving the detection capability of monocular vision in three-dimensional environment perception.

[0009] In a first aspect, this application provides a monocular 3D detection method, comprising: performing depth estimation on an input single two-dimensional image to generate a depth map fused with semantic information, including using a pre-trained monocular depth estimation model to perform depth estimation on the input single two-dimensional image to generate a depth map fused with semantic information; wherein the monocular depth estimation model is a student depth estimation model trained through a teacher-student learning framework; projecting and fusing the depth map and the original two-dimensional image onto a bird's-eye view coordinate system to generate a bird's-eye view feature map of the current frame, including combining the depth map with the original two-dimensional image, performing coordinate back projection and feature scattering, transforming and fusing them into a unified bird's-eye view feature space to generate a bird's-eye view feature map of the current frame; mapping historical bird's-eye view features onto the current frame coordinate system, performing weighted fusion of the bird's-eye view feature map of the current frame and the historical bird's-eye view features, and outputting a temporally enhanced fused feature map; decoding the fused feature map to output the three-dimensional detection information of each target in the scene.

[0010] In one implementation of the first aspect, the pre-training step of the monocular depth estimation model includes: using a pre-trained visual base model as a teacher model to perform depth inference on a large number of unlabeled images to generate pseudo-depth labels; introducing a semantic feature-assisted loss function during training to force the student model to learn a depth representation consistent with scene semantics, thereby improving the depth prediction accuracy and cross-scene robustness in textureless regions; and iteratively training the monocular depth estimation model with semantic enhancement using the pseudo-labels provided by the teacher model and the depth representation.

[0011] In one implementation of the first aspect, the step of iteratively training a semantically enhanced monocular depth estimation model using pseudo-labels provided by the teacher model and the depth representation includes: setting a semantic similarity loss, calculating and minimizing the similarity difference between the high-dimensional semantic features extracted by the teacher model and the depth features of the intermediate layer of the student model in the semantic embedding space; the semantic similarity loss forces the depth estimation of the student model not only to be numerically close to the pseudo-labels given by the teacher model, but also to maintain consistency with the perspective of the teacher model in terms of semantic structure, thereby filling the information gaps in the depth map in areas where texture is missing.

[0012] In one implementation of the first aspect, the step of combining the depth map with the original two-dimensional image, transforming and fusing them into a unified bird's-eye view feature space through coordinate back projection and feature scattering to generate a bird's-eye view feature map for the current frame includes: back-projecting the depth value of each pixel in the depth map to a three-dimensional coordinate system according to the camera intrinsic parameter model to generate pseudo-point cloud data; simultaneously, scattering the two-dimensional visual features of the two-dimensional image onto the bird's-eye view plane according to their corresponding spatial positions; and constructing the bird's-eye view feature map for the current frame based on the aligned and fused pseudo-point cloud data and two-dimensional visual features on the bird's-eye view plane.

[0013] In one implementation of the first aspect, the step of constructing a bird's-eye view feature map of the current frame based on the aligned and fused pseudo-point cloud data and two-dimensional visual features on the bird's-eye view plane includes: establishing a fixed-size bird's-eye view grid on a horizontal plane with the vehicle as the origin; and sorting each pseudo-point in the pseudo-point cloud data according to its... The planar coordinates fall into the corresponding bird's-eye view grid cells and are fused and synchronized with the visual features of the original two-dimensional image to form a rasterized bird's-eye view feature map.

[0014] In one implementation of the first aspect, each pseudopoint in the pseudo-point cloud data is sorted according to its... The steps of merging planar coordinates into corresponding bird's-eye view grid cells and combining them with the visual features of the original image to form a rasterized bird's-eye view feature map include: extracting multi-scale two-dimensional features from the original two-dimensional image using a convolutional neural network; accumulating and projecting the feature vector of each pixel onto the corresponding bird's-eye view grid cell after determining which bird's-eye view grid cell each pixel falls into; each bird's-eye view grid cell merging the feature vectors of 3D geometry and 2D texture at that location; and forming a rasterized bird's-eye view feature map through the accumulation of all pixels.

[0015] In one implementation of the first aspect, the steps of mapping historical bird's-eye view features to the current frame coordinate system, performing weighted fusion of the current frame's bird's-eye view feature map and the historical bird's-eye view features, and outputting a temporally enhanced fused feature map include: performing motion compensation alignment on the historical bird's-eye view features based on vehicle motion estimation, mapping the bird's-eye view features of past frames to the current frame coordinate system; performing weighted fusion of the current frame's bird's-eye view feature map and the historical bird's-eye view features, adaptively adjusting the fusion weights of the two, and outputting a temporally enhanced fused feature map; the fusion weights are generated by a learning gating mechanism or attention mechanism to suppress and filter out noise and interference in the current frame's bird's-eye view feature map and the historical bird's-eye view features.

[0016] In one implementation of the first aspect, the step of performing motion compensation alignment on historical bird's-eye view features based on vehicle motion estimation and mapping the bird's-eye view features of past frames to the coordinate system of the current frame includes: acquiring the vehicle's self-motion information between adjacent frames; and mapping the previous frame's bird's-eye view features to the coordinate system of the current frame. The historical bird's-eye view features are translated and rotated to the current frame coordinate system according to the estimated motion; for historical bird's-eye view features of earlier frames, they can be iteratively accumulated and aligned to... At any given moment, several aligned historical bird's-eye view feature images are obtained; through vehicle motion compensation, the historical bird's-eye view features at different times are spatially aligned and overlapped under static conditions.

[0017] In one implementation of the first aspect, the step of weighted fusion of the current frame's bird's-eye view feature map and the historical bird's-eye view features, and adaptively adjusting the fusion weights of the two, includes: inputting the current frame's bird's-eye view feature map and the aligned historical bird's-eye view features from the previous frame into a lightweight gating network; the gating network consists of several convolutional layers and a sigmoid activation function, outputting a gating matrix with the same size as the current bird's-eye view feature map; substituting the gating matrix into a preset function to calculate the fused feature map; when the gating matrix at a certain position is close to 1, it indicates that the current frame feature is reliable at that location and dominates. Weights are required; a gating matrix close to 0 emphasizes the use of information from the previous frame; if multi-frame fusion is used, it can be expanded to a weighted summation of features from multiple frames or a recursive application of gating units; the gating matrix is ​​a nonlinear function learned that can automatically detect where monocular depth uncertainty occurs based on input features, including: for areas with high noise in pseudo-point clouds, the gating network tends to reduce the weight of the current frame features and rely more on stable information from historical frames; conversely, for newly appearing objects or significant changes in the current frame, the gating network will assign higher weights to the current features; the gating subnetwork consists of 2–3 layers of 3×3 convolutions and a sigmoid activation function, and the output is G_t of the same size as the bird's-eye view features; the total loss = detection loss + first weight·affinity loss + second weight·consistency loss + third weight·semantic loss; the historical bird's-eye view features are first motion compensated before participating in gating fusion.

[0018] In one implementation of the first aspect, the step of decoding the fused feature map and outputting the 3D detection information of each target in the scene includes: inputting the fused feature map into a decoder based on a Transformer architecture; interacting with the bird's-eye view features in the fused feature map through the Transformer decoder to generate an encoding of each potential target in the scene; and predicting the 3D bounding box parameters, object category, and motion speed of each target through the encoding; wherein the 3D bounding box parameters include position, size, and orientation.

[0019] Secondly, this application provides a monocular 3D detection device, comprising: an image acquisition module for acquiring monocular camera images of a scene in front of a vehicle; a processor (such as an onboard computing unit), electrically connected to the image acquisition module; and a memory storing instruction code that causes the processor to execute the method. The processor specifically includes the following functional units: a semantic depth generation module configured to perform depth estimation on an input single two-dimensional image and generate a depth map fused with semantic information, including using a pre-trained monocular depth estimation model to perform depth estimation on the input single two-dimensional image and generate a depth map fused with semantic information; wherein the monocular depth estimation model is trained using a teacher-student learning framework. A student depth estimation model; a coordinate transformation module, configured to project and fuse the depth map and the original 2D image into a bird's-eye view coordinate system to generate a bird's-eye view feature map for the current frame, including combining the depth map with the original 2D image, performing coordinate back projection and feature scattering, transforming and fusing them into a unified bird's-eye view feature space to generate a bird's-eye view feature map for the current frame; a temporal fusion module, configured to map historical bird's-eye view features onto the current frame coordinate system, perform weighted fusion of the bird's-eye view feature map for the current frame and the historical bird's-eye view features, and output a temporally enhanced fused feature map; and a unified detection module, configured to decode the fused feature map and output 3D detection information of each target in the scene.

[0020] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described thereon.

[0021] As described above, the monocular 3D detection method, apparatus, and storage medium of this application have the following beneficial effects:

[0022] This application provides a method and apparatus for detecting point clouds in a vehicle-mounted forward-looking monocular camera, aiming to achieve accurate, robust, and real-time 3D target detection using only a low-cost monocular camera. Specifically, this application can solve the following technical problems:

[0023] (1) This application improves the accuracy and robustness of depth estimation: overcomes the problems of low accuracy and poor stability of existing monocular depth estimation in the case of no texture, long distance and dynamic objects, and generates pseudo point clouds that are more accurate in measurement and more coherent in semantics, providing high-quality input for subsequent 3D detection.

[0024] (2) This application guarantees the stability of temporal fusion: suppressing the cumulative amplification effect of monocular vision inherent noise (such as scale drift and depth flicker) in temporal fusion, avoiding the propagation and accumulation of errors, and realizing a stable and consistent environmental representation under multi-frame information fusion.

[0025] (3) This application provides an end-to-end unified architecture: it provides an efficient and unified perception framework, breaks down the barriers between depth estimation and target detection modules, achieves full integration at the feature level, replaces the problems of poor integration between modules and suboptimal performance in traditional serial design, and comprehensively improves detection accuracy and reliability. Attached Figure Description

[0026] Figure 1 The diagram shown illustrates an application scenario of the monocular 3D detection method described in this application embodiment.

[0027] Figure 2 The diagram shown is a flowchart illustrating the principle of the monocular 3D detection method described in this application embodiment.

[0028] Figure 3 The diagram shown is a model training flowchart of the monocular 3D detection method described in this application embodiment.

[0029] Figure 4 The diagram shown illustrates the semantic depth principle of the monocular 3D detection method described in this application embodiment.

[0030] Figure 5 The diagram shown is a bird's-eye view coordinate transformation flowchart of the monocular 3D detection method described in this application embodiment.

[0031] Figure 6 The flowchart shown is a temporal fusion process of the monocular 3D detection method described in the embodiments of this application.

[0032] Figure 7 The diagram shown illustrates the temporal fusion principle of the monocular 3D detection method described in this application embodiment.

[0033] Figure 8 The diagram shown is a unified detection flowchart of the monocular 3D detection method described in the embodiments of this application.

[0034] Figure 9 The diagram shown is a schematic diagram of the monocular 3D detection device described in the embodiments of this application.

[0035] Figure 10 The diagram shown illustrates the application principle of the monocular 3D detection device described in this application embodiment.

[0036] Component designation explanation

[0037] 9 Monocular 3D Inspection Device 91 Image acquisition module 92 processor 921 Semantic deep generation module 922 Coordinate Transformation Module 923 Timing Fusion Module 924 Unified detection module 93 memory 100 vehicle-mounted device 110 Image acquisition module 120 Data processing unit 121 Semantic deep generation module 122 BEV conversion module 123 Timing Fusion Module 124 Unified detection module 130 Memory Detailed Implementation

[0038] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0039] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0040] Schemes for recovering lost depth information from a single 2D image can employ a two-step approach: first, 2D object detection is performed, and then the depth of each object is estimated using empirical models or simple geometric assumptions. However, this indirect method struggles to meet security requirements in terms of accuracy. Therefore, deep learning is used to predict a dense depth map from a monocular image, and then the depth map is back-projected into a 3D pseudo-point cloud based on camera intrinsics. Finally, existing point cloud object detection algorithms are applied to this pseudo-point cloud. While theoretically feasible, this approach still faces several bottlenecks in practice, exhibiting the following significant drawbacks:

[0041] (1) Insufficient accuracy and stability of pseudo-point cloud: Monocular depth estimation model has inherent problems.

[0042] (a) Scale uncertainty: Since only relative depth can be predicted rather than absolute distance, the depth result often has an unknown scale factor, which requires additional calibration; in dynamic scenes, the lack of absolute scale information can easily lead to scale drift.

[0043] (b) In textureless areas (such as monochrome walls or flat roads) or on distant objects, the depth estimation error increases dramatically, resulting in generated pseudo point clouds that are noisy, sparse, and even distorted.

[0044] (c) When dealing with moving objects, depth estimation using self-supervised training often produces significant depth artifacts due to pixel matching errors. These inaccurate and unstable pseudo-point clouds, as input for subsequent detection, limit the performance of the entire perception system from the source.

[0045] (2) The fusion effect of time series information is not good:

[0046] To obtain a more stable and complete environmental representation, researchers have attempted to fuse perceptual results from multiple frames of images. For example, image information from multiple time points is converted into BEV (Bird's-Eye View) features without being specifically designed for monocular depth noise patterns. Such methods alleviate the missed detection problem caused by the sparsity of single-frame point clouds to some extent. However, since traditional fusion mechanisms (such as direct stitching or weighted averaging) are not designed for the specific noise patterns introduced by monocular depth estimation, these errors may be propagated and amplified when fusing BEV features containing scale shifts or depth flicker noise, resulting in false targets ("ghosting") or incorrect object trajectories, thus limiting the improvement in stability.

[0047] (3) The perception process modules are fragmented and lack unified optimization:

[0048] Existing monocular perception systems often treat depth estimation and object detection as independent, sequential modules, lacking an end-to-end joint optimization design. For example, while depth feature convolutions enhance detection, the feature map extraction and fusion steps within these methods are relatively independent. This modular design ignores the inherent connections between the various stages, resulting in suboptimal overall feature representation and limiting the potential for improving detection reliability and accuracy. Furthermore, most traditional methods still employ classic detection heads to output candidate bounding boxes, which may not fully utilize global information for monocular vision tasks.

[0049] The following embodiments of this application provide a monocular 3D detection method, device, and storage medium, including but not limited to applications in autonomous driving scenarios for vehicles. The following description will use this application scenario as an example.

[0050] Please see Figure 1 The image shows a schematic diagram illustrating an application scenario of the monocular 3D detection method described in this application embodiment. Figure 1 As shown, in autonomous driving, the vehicle's monocular camera acquires a single two-dimensional image, and the vehicle's main control device executes the monocular 3D detection method of this application to obtain three-dimensional detection information based on the single two-dimensional image.

[0051] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0052] Please see Figure 2 The diagram shows the principle flowchart of the monocular 3D detection method described in the embodiments of this application. Figure 2 As shown, this embodiment provides a monocular 3D detection method, which specifically includes the following steps:

[0053] S21, perform depth estimation on the input single two-dimensional image to generate a depth map fused with semantic information, including using a pre-trained monocular depth estimation model to perform depth estimation on the input single two-dimensional image to generate a depth map fused with semantic information; wherein, the monocular depth estimation model is a student depth estimation model trained through a teacher-student learning framework.

[0054] In one embodiment, the step of performing depth estimation on an input single two-dimensional image to generate a depth map incorporating semantic information includes:

[0055] A pre-trained monocular depth estimation model is used to estimate the depth of a single input 2D image, generating a depth map that incorporates semantic information; wherein, the monocular depth estimation model is a student depth estimation model trained through a teacher-student learning framework.

[0056] Please see Figure 3 The flowchart shown is a model training flowchart of the monocular 3D detection method described in this application embodiment. The pre-training steps of the monocular depth estimation model, namely training a student depth estimation model using a teacher-student learning framework, include:

[0057] S211 uses a pre-trained visual base model as a teacher model to perform depth inference on massive amounts of unlabeled images to generate pseudo-depth labels.

[0058] Specifically, a pre-trained Vision Foundation Model is used as the teacher model to perform depth inference on massive amounts of unlabeled images to generate pseudo-depth labels.

[0059] S212 introduces a loss function aided by semantic features during training, forcing the student model to learn a depth representation consistent with scene semantics, thereby improving the accuracy of depth prediction in textureless regions and cross-scene robustness.

[0060] S213, using the pseudo-labels provided by the teacher model and the depth representation (i.e., limited real depth data supervision), a semantically enhanced monocular depth estimation model is iteratively trained. This monocular depth estimation model is then used to infer the image from the monocular camera in front of the vehicle, outputting a dense depth map of the corresponding scene.

[0061] Please see Figure 4 This is a semantic depth principle diagram of the monocular 3D detection method described in the embodiments of this application. Figure 4 As shown, a schematic diagram of the semantic-assisted teacher-student learning framework is given, demonstrating the training process of the teacher model and the student model, as well as the working mechanism of affine invariant depth loss, consistency loss and semantic similarity loss.

[0062] In this embodiment, the training of the student monocular depth estimation model employs a hybrid strategy of supervised training with labeled data and distillation of unlabeled data. The teacher model is a pre-trained model with strong visual representation capabilities, such as using the DINOv2 self-supervised visual Transformer as the feature extraction backbone, connected to a DPT (Dense Prediction Transformer) depth regression head to form the teacher network. First, a limited number of labeled image data (each containing an RGB image and its corresponding precise depth map) are obtained using several public datasets (such as KITTI, NYU Depth, etc.); simultaneously, a much larger amount of unlabeled image data (from vehicle camera sequences in autonomous driving scenarios, etc.) is collected. Figure 4 As shown, the teacher model is fine-tuned and trained on labeled data to extract deep feature representations that are highly sensitive to scene semantics and geometry. Then, the teacher model infers on unlabeled images, generating corresponding pseudo-depth labels to guide the student model's learning.

[0063] During the training of the student model, three main loss functions are set:

[0064] (a) Affine-invariant depth loss, used to measure the difference between student-predicted depth and ground truth / pseudo-ground truth depth, but filters out the effects of scale and bias, making the model focus on relative depth relationships;

[0065] (b) Unlabeled consistency loss, which predicts the original unlabeled image and its version after strong enhancement transformation (such as color jitter, Gaussian blur, CutMix, etc.) separately, encourages the student model to keep the outputs of the two consistent, thereby improving the robustness to various visual perturbations.

[0066] (c) Semantic similarity loss, which is one of the core innovations of this application, calculates and minimizes the similarity difference between the high-dimensional semantic features extracted by the teacher model and the deep features of the intermediate layer of the student model in the semantic embedding space. This loss forces the depth estimation of the student model to not only be numerically close to the pseudo-labels given by the teacher, but also to be consistent with the teacher's perspective in terms of semantic structure, thereby filling the information gaps in the depth map in areas where texture is missing.

[0067] The student model parameters are iteratively updated by combining the above losses, and finally a semantically enhanced monocular deep student model is trained.

[0068] During the inference phase, the semantic depth generation module inputs each frame of RGB image captured by the vehicle's forward-facing camera into the student model, quickly outputting a depth map of the corresponding scene. This depth map, after being corrected for absolute scale and calibrated / verified in the target scene, closely approximates the physical distance and provides more consistent and reasonable predictions of depth for different categories of areas such as road surfaces, vehicles, and pedestrians.

[0069] S22, the depth map and the original two-dimensional image are projected and fused into the bird's-eye view coordinate system to generate the bird's-eye view feature map of the current frame. This includes combining the depth map with the original two-dimensional image, performing coordinate back projection and feature scattering, transforming and fusing them into a unified bird's-eye view feature space to generate the bird's-eye view feature map of the current frame.

[0070] In one embodiment, the step of fusing the depth map and the original two-dimensional image projection into a bird's-eye view coordinate system to generate a bird's-eye view feature map of the current frame includes:

[0071] The depth map is combined with the original two-dimensional image, and after coordinate back projection and feature scattering, it is transformed and fused into a unified bird's-eye view feature space to generate the bird's-eye view feature map of the current frame.

[0072] Specifically, the original two-dimensional image is the original front-view RGB (Red, Green, Blue) image. The depth map obtained in step S21 is combined with the original front-view RGB image, and after operations such as coordinate back projection and feature platting, it is transformed and fused into a unified bird's-eye view (BEV) feature space.

[0073] Please see Figure 5 The diagram shows a bird's-eye view coordinate transformation flowchart of the monocular 3D detection method described in this application embodiment. Figure 5 As shown, the steps of combining the depth map with the original two-dimensional image, transforming and fusing them into a unified bird's-eye view feature space through coordinate back projection and feature scattering to generate the bird's-eye view feature map of the current frame include:

[0074] S221, Based on the camera intrinsic parameter model, the depth value of each pixel in the depth map is back-projected onto the three-dimensional coordinate system to generate dense pseudo-point cloud data.

[0075] S222, Simultaneously, the two-dimensional visual features of the two-dimensional image are scattered onto the bird's-eye view plane according to their corresponding spatial positions.

[0076] S223, construct the bird's-eye view feature map of the current frame based on the aligned and fused pseudo-point cloud data and two-dimensional visual features on the bird's-eye view plane.

[0077] Specifically, a BEV feature map is constructed by aligning and fusing 3D geometric information (pseudo-point cloud data) with 2D texture features (two-dimensional visual features) in the BEV space, which serves as the input representation for subsequent modules. This BEV feature map is naturally organized in a vehicle-centric planar coordinate system, facilitating temporal fusion and target detection.

[0078] In practical applications, after obtaining the depth map, the processor enters the BEV transformation module to generate bird's-eye view features. Figure 1 (S2).

[0079] First, based on the camera intrinsic parameter matrix (focal length) Main point (etc.) and the depth value of each pixel, in the image coordinate system of the pixel Back projection onto the three-dimensional coordinate system in front of the vehicle The formula for calculating the middle part is:

[0080] ,in This is the value of the depth map at that pixel. Performing the above coordinate transformation on the entire depth map yields the set of pseudo-point clouds corresponding to the current frame. Unlike LiDAR point clouds, pseudo-point clouds are mapped from dense pixels of the camera and can theoretically be dense. However, due to monocular depth errors, pseudo-point clouds may contain outliers or holes.

[0081] To leverage mature point cloud detection architectures, this application further represents pseudo-point clouds as rasterized bird's-eye view feature maps. Specifically, a fixed-size bird's-eye view grid is established on a horizontal plane with the vehicle as the origin, for example, in front... The area is divided into grid cells of a certain resolution (e.g., 0.1 meters / pixel). Then, each pseudo-point is assigned according to its... Planar coordinates are placed into the corresponding BEV grid cells and fused synchronously with the visual features of the original image. For example, a feature splatting method can be used: multi-scale two-dimensional features are extracted from the camera's RGB image using CNNs such as ResNet. Knowing which BEV grid each pixel falls into, the feature vector of that pixel is accumulated / projected onto the corresponding BEV grid position. After accumulating all points, a C-channel BEV feature map is obtained (where each grid cell contains a feature vector that fuses the 3D geometry and 2D texture at that location). Simultaneously, the height information of the pseudo-point cloud can be... Encoding is performed as an additional height dimension or multiple raster planes to enrich the BEV features' ability to represent 3D structures. Through the above process, depth and image information in the pixel coordinate system are successfully transformed into bird's-eye view planar features centered on the vehicle, providing a unified representation for multi-frame spatiotemporal alignment and detection.

[0082] Example of BEV working range: forward 60–80 m, lateral ±30–40 m; grid resolution 0.1–0.5 m / cell; the spread aggregation can use weighted accumulation and normalization, or use max pooling.

[0083] S23, the historical bird's-eye view features are mapped onto the current frame coordinate system, and the bird's-eye view feature map of the current frame and the historical bird's-eye view features are weighted and fused to output a temporally enhanced fused feature map. Thus, the BEV feature map generated in the current frame is fused with historical BEV features from several previous moments, improving the completeness and stability of the environment representation.

[0084] Please see Figure 6 The diagram shows the temporal fusion flowchart of the monocular 3D detection method described in this application embodiment. Figure 6 As shown, the steps of mapping historical bird's-eye view features to the current frame coordinate system, performing weighted fusion of the current frame's bird's-eye view feature map and the historical bird's-eye view features, and outputting a temporally enhanced fused feature map include:

[0085] S231, perform motion compensation alignment on historical bird's-eye view features based on vehicle motion estimation (such as inertial navigation IMU and wheel odometer information), and map the bird's-eye view features of past frames to the coordinate system of the current frame.

[0086] S232, perform weighted fusion of the bird's-eye view feature map of the current frame and the historical bird's-eye view features, adaptively adjust the fusion weight of the two, and output a temporally enhanced fusion feature map.

[0087] In one embodiment, the fusion weights are generated by a learning gating mechanism or attention mechanism to suppress and filter out noise and interference in the bird's-eye view feature map of the current frame and the historical bird's-eye view features.

[0088] Specifically, this application introduces a gating / attention fusion mechanism to weight and fuse the current frame and aligned historical features, adaptively adjusting their respective proportions. The fusion weights are generated by a learning-based gating or attention mechanism, capable of suppressing and filtering out noise and inconsistencies in the features. This dynamic fusion strategy effectively reduces the accumulation of monocular depth noise over time, resulting in output fused BEV features with stronger temporal coherence and robustness.

[0089] Please see Figure 7 The diagram shows the temporal fusion principle of the monocular 3D detection method described in this application embodiment. Figure 7 As shown, a schematic diagram of the noise-resistant temporal BEV fusion module is presented, depicting the vehicle motion compensation alignment process and the principle of the gating mechanism for fusing current frame and historical features.

[0090] like Figure 7 As shown, a feature fusion method based on a gating mechanism is designed to address the noise patterns unique to monocular pseudo-point clouds. Assume we maintain a sliding time window, storing the most recent... BEV characteristics of frames (including the current frame) and the preceding frames When the new frame Upon arrival, perform the following steps:

[0091] (1) Motion-compensated alignment:

[0092] Acquire vehicle self-motion information between adjacent frames, such as attitude changes (yaw angle) calculated from the onboard IMU and odometer. Displacement (etc.). Previous frame The BEV features are translated and rotated to the current frame coordinate system according to the estimated motion. For earlier frames, the alignment can be iteratively accumulated to... Time. This step yields several aligned historical BEV feature maps ( Figure 4 middle Due to vehicle motion compensation, features at different times can be aligned and overlap in space even when the vehicle is stationary.

[0093] (2) Gated fusion:

[0094] BEV feature map of the current frame Aligned with the features of the previous frame Input a lightweight gated network. This network can consist of several convolutional layers and a sigmoid activation function, and output a gate matrix with the same size as the feature map. (Element values ​​are between 0 and 1). Then follow the formula. Calculate the fused feature map (where (Represents element-wise product). Intuitively, when a certain position... When the value is close to 1, it indicates that the feature of the current frame is reliable at that point and has a major weight. A value close to 0 emphasizes utilizing information from the previous frame. If multi-frame fusion is used, it can be expanded to weighted summation of features from multiple frames or recursively applying gating units. It's worth noting that this gating matrix is ​​a learned nonlinear function that can automatically detect where monocular depth uncertainty occurs based on input features: for example, for regions with high pseudo-point cloud noise, the gating network tends to reduce the weight of current frame features, relying more on stable information from historical frames; conversely, for newly appearing objects or significant changes in the current frame, the gating network assigns higher weights to current features. This adaptive fusion effectively suppresses the propagation of monocular noise, resulting in a fusion outcome... It combines historical robustness with current fresh information. Experiments show that, compared with simple direct averaging or splicing, gated fusion significantly reduces the false alarm rate in the detection of fast-moving objects and distant targets, achieving temporal consistency in scene understanding.

[0095] (3) The gated subnet can be composed of 2–3 layers of 3×3 convolutions and sigmoid functions, with the output being G_t of the same size as the BEV features; the total loss can be written as:

[0096] L_total = L_det + λ_1·L_aff + λ_2·L_con + λ_3·L_sem

[0097] Where L_det represents the detection loss, λ_1 represents the first weight, L_aff represents the affinity loss, λ_2 represents the second weight, L_con represents the consistency loss, λ_3 represents the third weight, L_sem represents the semantic loss, and λ_i∈[0.1, 2.0]. The temporal window N∈[2, 5], and historical features are first subjected to SE(2) motion compensation before participating in gating fusion.

[0098] In other implementations, temporal fusion mechanisms can also employ attention mechanisms. For example, a Transformer Encoder can be introduced, using current frame features as the query and historical features from multiple frames as keys and values, to obtain the fusion result through self-attention computation. These methods can also learn weights from historical information, but the computational cost is relatively high. Using gated convolutions offers better real-time performance while maintaining overall performance.

[0099] After temporal fusion, a fused and enhanced BEV feature map is obtained, which incorporates information from consecutive time steps, significantly improving robustness and completeness. This fused feature map will then be used for unified detection in S24.

[0100] S24, decode the fused feature map and output the three-dimensional detection information of each target in the scene.

[0101] Please see Figure 8 The diagram shows a unified detection flowchart of the monocular 3D detection method described in the embodiments of this application. Figure 8 As shown, the steps for decoding the fused feature map and outputting the 3D detection information of each target in the scene include:

[0102] S241, the fused feature map is input into a decoder based on the Transformer architecture.

[0103] S242, the Transformer decoder interacts with the bird's-eye view features in the fused feature map to generate encodings for each potential target in the scene.

[0104] Specifically, the decoder employs an end-to-end query-key-value attention mechanism (e.g., drawing inspiration from the DETR or Mask2Former paradigm): it initializes several learnable target query vectors, interacts with BEV features through the Transformer decoder, and generates encoded representations of each potential target in the environment.

[0105] S243, the three-dimensional bounding box parameters, object category, and motion speed of each target are predicted through the encoding; wherein, the three-dimensional bounding box parameters include position, size, and orientation.

[0106] Specifically, a Transformer decoder is used to directly infer information about the 3D target. The Transformer decoder was originally used in 2D object detection (such as the DETR model), and this application improves upon it for 3D BEV features. Its structure is as follows: First, a fixed number of... Each query vector represents a potential target to be detected. These query vectors are iteratively updated on the BEV feature map through multiple layers of self-attention and cross-attention, gradually focusing on different regions of interest. Specifically, each layer of the decoder performs the following operations: (a) self-attention between queries, ensuring that different queries avoid focusing on the same target or competing with each other; (b) cross-attention between query and BEV feature key-value pairs, extracting the local description of the target from the BEV features and assigning it to the query vector; (c) feedforward network update. After the layer decoder, the output is Each query vector corresponds to the detected... There are 10 candidate targets. Each query vector is then decoded by a dedicated feedforward network to extract target parameters, such as the coordinates of the center point on the BEV plane. Target size Orientation angle ,speed And category confidence, etc. For queries that do not match the real target, they will be predicted as "no target" (background class), thus naturally achieving one-to-one prediction and matching with the real target, avoiding NMS post-processing. The decoder can use the number of queries M∈[100, 300] and the number of layers L∈[3, 6]; Hungarian matching is used, and the classification / box / orientation / velocity loss is weighted by αcls, αbox, αdir, αvel∈[0.1, 2.0] respectively.

[0107] It is worth mentioning that the Transformer decoder head of this application can utilize the idea of ​​Mask Transformer (such as Mask2Former) to treat 3D detection as a problem of parallel pixel segmentation and bounding box regression. By introducing learnable BEV feature masks and multi-task training, the accuracy and stability of detection can be further improved. This is one of the optimization transformation schemes of this application, and will not be elaborated here.

[0108] After decoding, the system outputs a list of 3D bounding boxes and attribute information for all targets detected in the current frame. These detection results can be directly sent to the downstream path planning and control system for autonomous driving decision-making.

[0109] It should be noted that the hardware and libraries (such as a certain brand's SoC and inference library) mentioned in the embodiments are optional implementations and do not limit the scope of protection of this application; the parameters and grid resolution can be adjusted according to the scenario.

[0110] Therefore, compared with the traditional detection head design that uses pixel-by-pixel prediction followed by NMS screening, this unified decoding module can make full use of global feature relationships to produce the final 3D detection box set in one go, simplifying the detection process and improving accuracy.

[0111] This application achieves higher depth estimation accuracy and robustness. By introducing a semantically assisted teacher-student learning framework, the pseudo-point clouds generated by this application outperform traditional monocular methods in both depth measurement accuracy and structural consistency. In challenging scenarios such as textureless and long-distance scenes, semantic priors effectively compensate for the perceptual blind spots of depth models, significantly reducing depth errors and scale drift. It provides high-quality 3D point cloud input from the source, raising the overall performance ceiling of the system.

[0112] This application improves temporal continuity and stability. The designed noise-resistant temporal fusion mechanism can intelligently suppress noise such as flicker and drift introduced by monocular depth estimation, effectively ensuring the consistency of detection results across frames. The output 3D bounding box changes smoothly over time, significantly reducing false detections and missed detections of both static and dynamic objects. It has certain advantages in detection stability compared to techniques using simple feature stitching and fusion.

[0113] This application's unified architecture achieves high performance by integrating depth estimation, feature fusion, and object detection, and utilizes advanced architectures such as the Transformer as a unified decoder for global modeling of multi-source information. Verification shows that this unified framework exhibits competitive performance on publicly available datasets; it achieves real-time inference speed on embedded platforms (specifically related to resolution, batch size, and power budget), significantly narrowing the gap with expensive LiDAR-dependent solutions. Furthermore, end-to-end joint optimization of each module avoids error propagation in traditional serial processes, resulting in excellent overall performance.

[0114] This application offers low cost and high efficiency, meeting real-time requirements by requiring only a single camera as a sensor, making it significantly cheaper than multi-sensor solutions. Through teacher-student learning, knowledge from a large teacher model is distilled into a compact and efficient student model. Combined with optimization techniques such as TensorRT to accelerate inference, the entire complex perception model can achieve near real-time inference speeds on automotive-grade embedded hardware (e.g., on mainstream automotive-grade embedded platforms with GPU / AI acceleration), meeting the timeliness requirements of practical autonomous driving deployments.

[0115] The scope of protection of the monocular 3D detection method described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0116] This application also provides a monocular 3D detection device, which can implement the monocular 3D detection method described in this application. However, the implementation device of the monocular 3D detection method described in this application includes, but is not limited to, the structure of the monocular 3D detection device listed in this embodiment. All structural modifications and substitutions of the prior art made based on the principles of this application are included within the protection scope of this application.

[0117] Please see Figure 9 The diagram shows the structural principle of the monocular 3D detection device described in the embodiments of this application. Figure 9 As shown, this embodiment provides a monocular 3D detection device 9, including: an image acquisition module 91, a processor 92, and a memory 93.

[0118] The image acquisition module 91 is used to acquire monocular camera images of the scene in front of the vehicle.

[0119] The processor 92 (such as an on-board computing unit) is electrically connected to the image acquisition module.

[0120] The memory 93 stores instruction code that causes the processor to execute the monocular 3D detection method.

[0121] The processor 92 specifically includes the following functional units:

[0122] The semantic depth generation module 921 is configured to perform depth estimation on the input single two-dimensional image and generate a depth map fused with semantic information. This includes using a pre-trained monocular depth estimation model to perform depth estimation on the input single two-dimensional image and generate a depth map fused with semantic information. The monocular depth estimation model is a student depth estimation model trained through a teacher-student learning framework.

[0123] The coordinate transformation module 922 is configured to project and fuse the depth map and the original two-dimensional image into the bird's-eye view coordinate system to generate a bird's-eye view feature map of the current frame. This includes combining the depth map and the original two-dimensional image, performing coordinate back projection and feature scattering, transforming and fusing them into a unified bird's-eye view feature space to generate a bird's-eye view feature map of the current frame.

[0124] The temporal fusion module 923 is configured to map historical bird's-eye view features to the current frame coordinate system, perform weighted fusion of the bird's-eye view feature map of the current frame and the historical bird's-eye view features, and output a temporally enhanced fused feature map.

[0125] The unified detection module 924 is configured to decode the fused feature map and output the three-dimensional detection information of each target in the scene.

[0126] The processor 92 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0127] The aforementioned memory 93 may include random access memory (RAM), and may also include non-volatile memory and non-volatile flash memory. For example, it may be DRAM (Dynamic Random Access Memory), NAND Flash, NOR Flash, EEPROM (Electrically Erasable Programmable Read Only Memory), etc.

[0128] Please see Figure 10 The image shown is a schematic diagram illustrating the application principle of the monocular 3D detection device described in this application embodiment. Figure 10As shown, in a specific embodiment, the monocular 3D detection device is an in-vehicle device 100 applied to a vehicle. The in-vehicle device 100 includes a monocular camera (110), a data processing unit (120), and memory (130), etc. In actual implementation, the data processing unit 120 can be an embedded computing platform equipped with GPU acceleration, such as a mainstream automotive-grade embedded platform with GPU / AI acceleration. The memory 130 pre-stores files such as depth estimation student models and Transformer model weights. When the vehicle is driving, the camera 110 continuously acquires images of the road conditions ahead, which are processed in real time by the processing unit 120 according to the above steps. The output 3D detection results are sent to the vehicle's decision control module (downstream system D) via CAN bus or Ethernet. The solution of this application has low requirements for computing resources and can be deployed in an embedded environment to meet the needs of in-vehicle deployment.

[0129] Therefore, this application provides a low-cost, high-performance vehicle-mounted monocular 3D target detection solution through innovative teacher-student learning depth estimation, gated temporal fusion, and Transformer target decoding. This solution effectively solves the problems of inaccurate depth, unstable detection, and module disconnect in existing technologies, and has achieved good results in actual road tests.

[0130] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0131] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0132] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not exceed the scope.

[0133] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed, implements the monocular 3D detection method described above.

[0134] The monocular 3D detection method includes: performing depth estimation on an input single 2D image to generate a depth map fused with semantic information; projecting and fusing the depth map and the original 2D image onto a bird's-eye view coordinate system to generate a bird's-eye view feature map for the current frame; mapping historical bird's-eye view features onto the current frame coordinate system, performing weighted fusion of the bird's-eye view feature map for the current frame and the historical bird's-eye view features to output a temporally enhanced fused feature map; and decoding the fused feature map to output 3D detection information for each target in the scene.

[0135] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. This available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0136] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0137] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A monocular 3D detection method, characterized in that, include: Depth estimation is performed on an input single 2D image to generate a depth map incorporating semantic information. This includes using a pre-trained monocular depth estimation model to estimate the depth of the input single 2D image and generate a depth map incorporating semantic information. The monocular depth estimation model is trained using a teacher-student learning framework to train a student depth estimation model. The pre-training of the monocular depth estimation model includes: using a pre-trained visual baseline model as the teacher model to perform depth inference on a large number of unlabeled images to generate pseudo-depth labels; introducing a semantic feature-assisted loss function during training to force the student model to learn a depth representation consistent with scene semantics, thereby improving the accuracy of depth prediction in textureless regions and cross-scene robustness; and iteratively training the model using the pseudo-labels provided by the teacher model and the depth representation to obtain a semantically enhanced monocular depth estimation model. The depth map and the original two-dimensional image are projected and fused into the bird's-eye view coordinate system to generate the bird's-eye view feature map of the current frame. This includes combining the depth map with the original two-dimensional image, performing coordinate back projection and feature scattering, transforming and fusing them into a unified bird's-eye view feature space to generate the bird's-eye view feature map of the current frame. The process involves mapping historical bird's-eye view features to the current frame coordinate system, performing weighted fusion of the current frame's bird's-eye view feature map and the historical bird's-eye view features, and outputting a temporally enhanced fused feature map. This includes: performing motion compensation alignment on the historical bird's-eye view features based on vehicle motion estimation; mapping the bird's-eye view features of past frames to the current frame coordinate system; performing weighted fusion of the current frame's bird's-eye view feature map and the historical bird's-eye view features, adaptively adjusting the fusion weights of the two, and outputting a temporally enhanced fused feature map. The fusion weights are generated by a learning gating mechanism or attention mechanism to suppress and filter noise and interference in the current frame's bird's-eye view feature map and the historical bird's-eye view features. The fused feature map is decoded to output the 3D detection information of each target in the scene.

2. The method according to claim 1, characterized in that, The steps of iteratively training a semantically enhanced monocular depth estimation model using pseudo-labels provided by the teacher model and the depth representation include: A semantic similarity loss is set up, which calculates and minimizes the similarity difference between the high-dimensional semantic features extracted by the teacher model and the deep features of the intermediate layer of the student model in the semantic embedding space. The semantic similarity loss forces the depth estimation of the student model to not only be numerically close to the pseudo-label given by the teacher model, but also to be consistent with the perspective of the teacher model in terms of semantic structure, thereby filling the information gaps in the depth map in the texture-deficient areas.

3. The method according to claim 1, characterized in that, The steps of combining the depth map with the original two-dimensional image, performing coordinate back projection and feature scattering, transforming and fusing them into a unified bird's-eye view feature space to generate the bird's-eye view feature map of the current frame include: Based on the camera intrinsic parameter model, the depth value of each pixel in the depth map is back-projected onto the three-dimensional coordinate system to generate pseudo point cloud data; At the same time, the two-dimensional visual features of the two-dimensional image are scattered onto the bird's-eye view plane according to their corresponding spatial positions; Based on the aligned and fused pseudo-point cloud data and 2D visual features on the bird's-eye view plane, construct the bird's-eye view feature map of the current frame.

4. The method according to claim 3, characterized in that, The steps for constructing a bird's-eye view feature map for the current frame based on the aligned and fused pseudo-point cloud data and 2D visual features on the bird's-eye view plane include: Establish a fixed-size bird's-eye view grid on a horizontal plane with the vehicle as the origin; Each pseudopoint in the pseudo point cloud data is placed into the corresponding bird's-eye view grid cell according to its (X,Y) plane coordinates, and then fused and synchronized with the visual features of the original two-dimensional image to form a rasterized bird's-eye view feature map.

5. The method according to claim 4, characterized in that, The steps of placing each pseudo-point in the pseudo-point cloud data into the corresponding bird's-eye view grid cell according to its (X,Y) plane coordinates, and fusing and synchronizing it with the visual features of the original image to form a rasterized bird's-eye view feature map include: Multi-scale two-dimensional features are extracted from the original two-dimensional image using a convolutional neural network. After determining which bird's-eye view grid each pixel falls into, the feature vector of that pixel is accumulated and projected onto the corresponding bird's-eye view grid cell. Each bird's-eye view grid cell integrates the feature vectors of 3D geometry and 2D texture at that location. The accumulated results of all pixels form a rasterized bird's-eye view feature map.

6. The method according to claim 1, characterized in that, The steps of performing motion compensation and alignment on historical bird's-eye view features based on vehicle motion estimation, and mapping bird's-eye view features from past frames to the current frame coordinate system, include: Acquire the vehicle's self-motion information between adjacent frames; translate and rotate the historical bird's-eye view features of the previous frame at time t-1 according to the estimated motion to the coordinate system of the current frame; for historical bird's-eye view features of earlier frames, iteratively accumulate and align them to time t to obtain several aligned historical bird's-eye view feature maps; through vehicle motion compensation, ensure that historical bird's-eye view features at different times are spatially aligned and overlapped in a static environment.

7. The method according to claim 1, characterized in that, The step of weightedly fusing the bird's-eye view feature map of the current frame and the historical bird's-eye view features, and adaptively adjusting the fusion weights of the two, includes: The current frame's bird's-eye view feature map and the aligned historical bird's-eye view feature map from the previous frame are input into a lightweight gating network. The gating network consists of several convolutional layers and a sigmoid activation function, and outputs a gating matrix with the same size as the current frame's bird's-eye view feature map. The gating matrix is ​​then substituted into a preset function to calculate the fused feature map. When the gating matrix at a certain position is close to 1, it indicates that the current frame feature is reliable at that location and has a major weight; when the gating matrix is ​​close to 0, it emphasizes the use of information provided by the previous frame. If multi-frame fusion is used, it can be extended to a weighted summation of features from multiple frames or a recursive application of gating units. The gating matrix is ​​a nonlinear function obtained through learning, which can automatically detect where monocular depth uncertainty occurs based on the input features. This includes: for areas with high noise in pseudo-point clouds, the gating network tends to reduce the weight of the current frame features and rely more on stable information from historical frames; conversely, for newly appearing objects or significant changes in the current frame, the gating network assigns higher weights to the current features.

8. A monocular 3D detection device, characterized in that, include: The image acquisition module is used to acquire images of the scene in front of the vehicle from a monocular camera. The processor is electrically connected to the image acquisition module; A memory storing instruction code that causes a processor to perform the method according to any one of claims 1 to 7; The processor specifically includes the following functional units: A semantic depth generation module is configured to perform depth estimation on an input single 2D image and generate a depth map incorporating semantic information. This includes using a pre-trained monocular depth estimation model to perform depth estimation on the input single 2D image and generate a depth map incorporating semantic information. The monocular depth estimation model is trained using a teacher-student learning framework to train a student depth estimation model. The pre-training of the monocular depth estimation model includes: using a pre-trained visual baseline model as the teacher model to perform depth inference on a large number of unlabeled images to generate pseudo-depth labels; introducing a semantic feature-assisted loss function during training to force the student model to learn a depth representation consistent with scene semantics, thereby improving the accuracy of depth prediction in textureless regions and cross-scene robustness; and iteratively training the model using the pseudo-labels provided by the teacher model and the depth representation to obtain a semantically enhanced monocular depth estimation model. The coordinate transformation module is configured to project and fuse the depth map and the original two-dimensional image into the bird's-eye view coordinate system to generate the bird's-eye view feature map of the current frame. This includes combining the depth map and the original two-dimensional image, performing coordinate back projection and feature scattering, transforming and fusing them into a unified bird's-eye view feature space to generate the bird's-eye view feature map of the current frame. The temporal fusion module is configured to map historical bird's-eye view features to the current frame coordinate system, perform weighted fusion of the current frame's bird's-eye view feature map and the historical bird's-eye view features, and output a temporally enhanced fused feature map. This includes: performing motion compensation alignment on historical bird's-eye view features based on vehicle motion estimation; mapping past frame bird's-eye view features to the current frame coordinate system; performing weighted fusion of the current frame's bird's-eye view feature map and the historical bird's-eye view features; adaptively adjusting the fusion weights of the two; and outputting a temporally enhanced fused feature map. The fusion weights are generated by a learning gating mechanism or attention mechanism to suppress and filter noise and interference in the current frame's bird's-eye view feature map and the historical bird's-eye view features. The unified detection module is configured to decode the fused feature map and output the 3D detection information of each target in the scene.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the method described in any one of claims 1 to 7.

Images