Target detection method, apparatus, device, medium, and product

Abstract

The application provides a target detection method, device, equipment, medium and product, relates to the technical field of 3D detection, and includes the following steps: inputting an RGB image into a visual backbone network to obtain a plurality of visual features, the visual backbone network adopts a hybrid strategy of local window attention and global modeling to perform feature extraction on the RGB image; performing feature reconstruction and fusion on the plurality of visual features to obtain fused features; inputting the fused features into a depth network to obtain a depth map; inputting the fused features and the depth map into a depth perception decoder to obtain first decoding information; the first decoding information includes the category, orientation, 3D and 2D information, depth and depth weight of a first detected target. According to the 3D and 2D information, depth and depth weight, and dense sampling depth corresponding to the depth map, depth fusion is performed to obtain first target depth; and according to the first target depth, category, orientation, 3D and 2D information, a three-dimensional detection result is obtained.

Description

Technical Field

[0001] This application relates to the field of 3D inspection technology, and in particular to a target inspection method, apparatus, equipment, medium and product. Background Technology

[0002] The core task of monocular 3D target detection is to recover the 3D bounding box of the target from a single RGB image, including its 3D center position, size, and orientation. Compared with LiDAR, multi-view cameras, and fusion perception systems, monocular solutions have lower hardware costs, are more flexible in installation and modification, and are suitable for large-scale deployment. Therefore, they have high application value in vehicle-to-everything (V2X) systems, roadside perception, and intelligent transportation systems.

[0003] From a technological evolution perspective, existing monocular 3D detection methods can be broadly categorized into three types. The first type is direct regression schemes based on 2D center points, keypoints, or local features. Their advantages include a lightweight structure and ease of deployment, but depth and size regression are highly dependent on local visual features, leading to significant performance degradation in distant or occluded scenes. The second type is spatial enhancement schemes using depth as an intermediary. These schemes first predict pixel-by-pixel depth or depth distribution, then enhance 2D features to 3D space or a bird's-eye view for detection. These methods offer strong spatial representation capabilities, but depth estimation errors propagate continuously during subsequent enhancement processes, and typically require heavy networks or additional projection operators. The third type is end-to-end schemes based on Detection Transformer (DETR) or Transformer. These schemes model category, location, and geometric attributes uniformly through query vectors, resulting in a simpler overall framework. However, standard global self-attention is complex under high-resolution inputs, and roadside deployment is costly.

[0004] In roadside applications, there are some unique challenges that differ from those of vehicle-mounted cameras. Because cameras are often mounted at higher positions to cover a wider field of view, the number of distant targets in the image is greater, their size is smaller, and their occlusion relationships are more complex. Furthermore, the installation height, viewing angle, and calibration parameters of cameras at different intersections vary significantly, resulting in noticeable differences in the projection shape of similar targets in different scenarios. For monocular 3D tasks, depth errors are further amplified in 3D center and size estimation; therefore, without explicit depth modeling and geometric consistency constraints, it is difficult to obtain stable 3D localization relying solely on 2D detection results.

[0005] To alleviate these problems, existing research has attempted to introduce strategies such as multi-scale features, depth-guided attention, explicit geometric relationships, or category prior sizes. However, overall, current technologies still suffer from insufficient real-time performance and accuracy in object detection. Summary of the Invention

[0006] This application provides a target detection method, apparatus, device, medium, and product, which solves the problems of low timeliness and accuracy in current target detection methods.

[0007] Firstly, to achieve the above objectives, embodiments of this application provide a target detection method, comprising:

[0008] An RGB image is input into a visual backbone network to obtain multiple visual features output by the visual backbone network. The visual backbone network uses a hybrid strategy of local window attention and global modeling to extract features from the RGB image.

[0009] The visual features are reconstructed and fused to obtain fused features;

[0010] The fused features are input into a deep network to obtain a depth map output by the deep network;

[0011] The fused features and the depth map are input to a depth-aware decoder to obtain first decoding information output by the depth-aware decoder; the first decoding information includes the category, orientation, 3D information, 2D information, depth information, and depth weight information of the first detected target; wherein, the first detected target is the target in the RGB image;

[0012] The first target depth of the first detected target is obtained by performing depth fusion based on the 3D information, the 2D information, the depth information, the depth weight information, and the dense sampling depth corresponding to the depth map;

[0013] Based on the first target depth, the category, the orientation, the 3D information, and the 2D information, the three-dimensional detection result of the first detected target is obtained.

[0014] Specifically, an RGB image is input into a visual backbone network to obtain multiple visual features output by the visual backbone network, including:

[0015] The RGB image is divided into multiple spatially non-overlapping local windows;

[0016] In the multiple first network layers of the visual backbone network, local modeling is performed on each of the local windows;

[0017] In the multiple second network layers of the visual backbone network, the RGB image is globally modeled;

[0018] Based on the results of the local modeling and the global modeling, multiple visual features are obtained.

[0019] The process involves reconstructing and fusing multiple visual features to obtain fused features corresponding to the multiple visual features, including:

[0020] Multiple visual features are mapped to the same channel dimension via multi-scale projection.

[0021] The mapped visual features are reconstructed to obtain a multi-scale feature pyramid;

[0022] The feature maps in the multi-scale feature pyramid are fused to obtain the fused features.

[0023] The process of fusing features from each feature map in the multi-scale feature pyramid to obtain the fused features includes:

[0024] The P3 feature map in the multi-scale feature pyramid is downsampled to the main scale, and the P5 feature map in the multi-scale feature pyramid is upsampled to the main scale, wherein the main scale is the scale of the P4 feature map in the multi-scale feature pyramid;

[0025] The downsampled P3 feature map, the upsampled P5 feature map, and the P4 feature map are adaptively weighted and fused to obtain the fused feature.

[0026] The process of inputting the fused features into a deep network to obtain a depth map output by the deep network includes:

[0027] The deep network predicts the depth distribution of each pixel location in the fused feature;

[0028] Normalize the depth values ​​in the depth distribution;

[0029] Based on the probability of each depth in the depth distribution, the normalized depth values ​​are weighted and summed.

[0030] The depth map is obtained based on the summation results corresponding to each pixel position.

[0031] The method further includes:

[0032] The dense sampling depth is obtained by predicting the depth of the three-dimensional center position of the first detected target through Gaussian sampling in the depth map.

[0033] The method for obtaining the first target depth of the first detected target by performing depth fusion based on the 3D information, the 2D information, the depth information, the depth weight information, and the dense sampling depth corresponding to the depth map includes:

[0034] The geometric depth estimate of the target is determined based on the size in the 3D information, the 2D information, and the camera intrinsic parameters corresponding to the RGB image; the camera is a device that captures the RGB image.

[0035] Based on the depth weight information, the dense sampling depth, the geometric depth estimation, and the depth information are fused to obtain the first target depth.

[0036] The visual backbone network, the depth map network, and the depth-aware decoder constitute the object detection model, and the method further includes:

[0037] For each pixel location in the training image, Gaussian soft distribution labels are constructed for multiple depth intervals adjacent to the true depth of the pixel location;

[0038] During the training process of the target detection model, the depth map network supervises the prediction process of the depth values ​​at each pixel location using the Gaussian distributed soft labels.

[0039] The method further includes:

[0040] During the training of the target detection model, based on the 2D information in the second decoding information, the second target depth of the second detected target, and the camera intrinsic parameters, the center of the second detected target is back-projected from the image plane to the three-dimensional camera coordinate system to obtain the predicted three-dimensional center of the second detected target; wherein, the second detected target is the target in the training image, the second decoding information is the decoding result of the second detected target; and the camera intrinsic parameters are the intrinsic parameters of the camera that captured the training image.

[0041] Based on the predicted 3D center, the true 3D center of the second detection target, and the pre-constructed geometric consistency loss function, determine the consistency loss corresponding to the second detection target;

[0042] The target detection model is jointly trained based on the consistency loss and the loss corresponding to each item in the second decoded information.

[0043] Secondly, to achieve the above objectives, embodiments of this application provide a target detection device, comprising:

[0044] The first processing module is used to input an RGB image into a visual backbone network and obtain multiple visual features output by the visual backbone network. The visual backbone network uses a hybrid strategy of local window attention and global modeling to extract features from the RGB image.

[0045] The second processing module is used to perform feature reconstruction and feature fusion on multiple visual features to obtain fused features;

[0046] The third processing module is used to input the fused features into the deep network to obtain the depth map output by the deep network;

[0047] The fourth processing module is used to input the fused features and the depth map into the depth-aware decoder to obtain the first decoding information output by the depth-aware decoder; the first decoding information includes the category, orientation, 3D information, 2D information, depth information and depth weight information of the first detected target; wherein, the first detected target is the target in the RGB image;

[0048] The fifth processing module is used to perform depth fusion based on the 3D information, the 2D information, the depth information, the depth weight information, and the dense sampling depth corresponding to the depth map to obtain the first target depth of the first detected target;

[0049] The detection module is used to obtain the three-dimensional detection result of the first target based on the first target depth, the category, the orientation, the 3D information and the 2D information.

[0050] Thirdly, to achieve the above objectives, embodiments of this application provide a target detection device, including a transceiver, a processor, a memory, and a program stored in the memory and executable on the processor; the transceiver is used to send and receive data under the control of the processor, and the processor implements the target detection method as described in the first aspect when executing the program.

[0051] Fourthly, to achieve the above objectives, embodiments of this application provide a readable storage medium having a program or instructions stored thereon, which, when executed by a processor, implement the target detection method as described in the first aspect.

[0052] Fifthly, to achieve the above objectives, embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the target detection method as described in the first aspect.

[0053] The beneficial effects of the above technical solution in this application are as follows:

[0054] In the target detection method of this application, an RGB image is input into a visual backbone network to obtain multiple visual features output by the visual backbone network. The visual backbone network employs a hybrid strategy of local window attention and global modeling to extract features from the RGB image, thereby reducing computational complexity and memory overhead. Subsequently, feature reconstruction and feature fusion are performed on the multiple visual features to obtain fused features. The fused features are input into a deep network to obtain a depth map output by the deep network. The fused features and the depth map are input into a depth-aware decoder to obtain first decoding information output by the depth-aware decoder. The first decoding information includes the category, orientation, 3D information, 2D information, depth information, and depth weight information of a first detected target. The first detected target is a target in the RGB image. In this way, depth information can be used as prior information for decoding, solving the problem of the coarseness of current depth supervision methods. Finally, depth fusion is performed based on the 3D information, the 2D information, the depth information, the depth weight information, and the dense sampling depth corresponding to the depth map to obtain the first target depth of the first detected target. This solves the problem of a single depth source and lack of target-level adaptation, improving robustness. Finally, based on the first target depth, the category, the orientation, the 3D information, and the 2D information, the three-dimensional detection result of the first detected target is obtained. Ultimately, real-time and accurate target detection is achieved, solving the problems of low timeliness and accuracy in current detection methods. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the target detection method according to an embodiment of this application;

[0056] Figure 2 This is a schematic diagram of the overall framework of the target detection method according to an embodiment of this application;

[0057] Figure 3 This is a schematic diagram of the window attention backbone in the embodiments of this application;

[0058] Figure 4 This is a comparative diagram of Gaussian soft labels and traditional labels in the embodiments of this application;

[0059] Figure 5 This is a schematic diagram of the deep perception decoding and query vector-level learnable deep fusion in the embodiments of this application;

[0060] Figure 6 This is a schematic diagram of geometric consistency supervision in the embodiments of this application;

[0061] Figure 7 This is a schematic diagram of the target detection device according to an embodiment of this application;

[0062] Figure 8 This is a schematic diagram of the target detection device according to an embodiment of this application. Detailed Implementation

[0063] To make the technical problems, technical solutions and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0064] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0065] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0066] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0067] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean that B is determined solely based on A, but can also be determined based on A and / or other information.

[0068] Before describing the embodiments of this application, the relevant technical points will first be described by way of example:

[0069] Current object detection methods generally face the following common problems: First, it is difficult to balance backbone network efficiency with global semantic modeling; second, after continuous depth is forcibly discretized into single-label classifications, the training signal is not smooth enough; third, there is a lack of target-level collaboration between different depth sources, making it impossible to dynamically select more reliable depth paths based on the scene; fourth, 2D localization and 3D spatial constraints are usually scattered in different loss terms and have not yet formed a tight closed loop. Specifically:

[0070] 1) Insufficient encoding efficiency. Standard ViT or Transformer based on global self-attention requires processing a large number of labeled units in high-resolution images, resulting in high computational complexity and memory overhead, which is not conducive to stable real-time deployment on roadside servers or edge devices.

[0071] 2) The depth supervision method is relatively crude. Existing methods often use one-hot discrete classification or single scalar regression as depth supervision, which is insufficient for representing continuous depth, especially near the depth boundary, where quantization error and gradient discontinuity are easily introduced.

[0072] 3) The depth source is singular and lacks target-level adaptation. Some methods rely solely on query vectors to regress depth, while others rely solely on geometric formulas or dense depth maps. They lack a mechanism for dynamically selecting the depth source based on the target, thus resulting in insufficient robustness in scenes with low texture, occlusion, truncation, or drastic scale changes.

[0073] 4) Insufficient geometric involvement in the decoding stage. Many methods treat depth information as a post-processing supplement or only add constraints at the loss level, resulting in the query vector still mainly relying on two-dimensional appearance feature updates during the decoding process, making it difficult to obtain depth cues in a timely manner.

[0074] 5) Lack of closed-loop optimization in 2D and 3D. 2D bounding box, center offset, depth and 3D center are often modeled separately, lacking a mechanism to form a unified back-projection constraint through the camera imaging model. Therefore, even if the 2D projection position is reasonable, inconsistencies may occur in 3D space.

[0075] 6) Not good for distant small targets. Due to the small pixel ratio and weak geometric information of the target, if there is a lack of multi-scale depth supervision, central region weighting, and object-level loss reduction, small targets are easily dominated by large targets during training, resulting in a decline in detection and localization performance.

[0076] In view of this, embodiments of this application provide a target detection method, specifically a monocular 3D target detection method for real-time roadside deployment scenarios. This method addresses key issues of existing monocular 3D detection methods, such as high computational overhead under high-resolution input, unstable localization of small targets at long distances, significant discrete errors in depth estimation, and a lack of unified constraints between 2D detection and 3D localization. Specifically, existing solutions fall into two categories: one focuses on 2D appearance modeling but has limited depth recovery capabilities; the other focuses on geometric or bird's-eye view (BEV) enhancement but suffers from a heavy network structure and long inference links, making stable deployment on roadside servers or edge devices difficult.

[0077] For roadside cameras, the scale of targets within the same frame varies greatly. Nearby vehicles and distant small targets differ significantly in pixel ratio, and the projected shape of targets changes noticeably due to variations in camera mounting height, pitch angle, and focal length. Standard global Transformers need to process a large number of labeled units on such high-resolution images. The computational complexity of self-attention increases quadratically with the number of labeled units, easily leading to significant computational and memory pressure. Simply reducing the input resolution further weakens the processing of distant targets and fine-grained geometric cues.

[0078] Furthermore, the uniquely heated depth discrete supervision commonly used in existing methods only maps the true depth to a single discrete depth interval, which will produce quantization errors and discontinuous gradients. At the same time, most methods rely only on single path depth estimation and fail to jointly utilize complementary information such as regression depth, geometric depth and dense depth map sampling depth, which makes the prediction of 3D center, size and orientation prone to jitter in scenes with occlusion, low texture or large scale changes.

[0079] To address the aforementioned issues, the target detection method in this application proposes an end-to-end monocular 3D detection scheme consisting of "efficient window attention encoding, multi-scale depth modeling, depth perception decoding, query vector-level multi-source depth fusion, and 3D back-projection geometric closed-loop constraint." This scheme enables the model to maintain good real-time performance on high-resolution roadside images while enhancing depth continuity, robustness of target-level depth estimation, and consistency of 3D localization.

[0080] The implementation process of the target detection method provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0081] like Figure 1 As shown, the target detection method includes:

[0082] Step 101: Input the RGB image into the visual backbone network to obtain multiple visual features output by the visual backbone network. The visual backbone network uses a hybrid strategy of local window attention and global modeling to extract features from the RGB image.

[0083] The aforementioned visual backbone network employs a hybrid strategy of local window attention in non-critical layers and global modeling in critical output layers to extract visual features from RGB images. The local window attention mechanism in non-critical layers reduces complexity, while the global modeling in critical output layers avoids the problem of insufficient cross-window information interaction caused by pure local window modeling. This hybrid strategy maintains overall structural awareness while reducing complexity, achieving a balance between deployment efficiency and semantic expression.

[0084] It should be noted that before step 101 above, the method further includes: preprocessing the RGB image, wherein the preprocessing includes at least one of the following: scaling, cropping, and flipping.

[0085] Step 102: Perform feature reconstruction and feature fusion on the multiple visual features to obtain fused features;

[0086] In step 102 above, feature reconstruction can involve downsampling the visual features to obtain a feature pyramid, for example, using the YOLO Feature Pyramid Network (FPN) algorithm for feature reconstruction. Feature fusion refers to transforming the reconstructed features to a uniform scale before fusing them.

[0087] Step 103: Input the fused features into the deep network to obtain the depth map output by the deep network;

[0088] In the above steps, a depth map is obtained using a deep network, which makes the depth information obtained based on the depth map continuous. This can reduce the depth discretization error and gradient discontinuity problem, improve the smoothness of depth estimation, and solve the problem of insufficient robustness in scenes with low texture, occlusion, truncation, or drastic scale changes.

[0089] Step 104: Input the fused features and the depth map into the depth-aware decoder to obtain the first decoding information output by the depth-aware decoder; the first decoding information includes the category, orientation, 3D information, 2D information, depth information and depth weight information of the first detected target; wherein, the first detected target is the target in the RGB image;

[0090] The aforementioned 2D information includes, for example, a two-dimensional prediction box; the aforementioned 3D information includes, for example, a three-dimensional center offset and 3D dimensions, specifically, the normalized offset of the three-dimensional center relative to the center of the two-dimensional prediction box; the aforementioned depth weight information can also be referred to as depth uncertainty.

[0091] In the above steps, decoding based on the depth map can use depth information as prior information for decoding, thereby enhancing the stability of 3D localization in complex scenes.

[0092] After multiple layers of decoder iteration, each query vector outputs results such as category, 2D predicted bounding box, 3D center offset, query vector regression depth, depth uncertainty, 3D size, and orientation. The 3D center offset is preferably a normalized offset relative to the center of the 2D bounding box, ensuring stable representation even if the 3D center projection falls outside the 2D bounding box. The core of the decoding stage is to first introduce local depth prior at the query vector level, and then use the query vector to uniformly complete target attribute prediction, thereby enhancing the stability of 3D localization in complex scenes.

[0093] Step 105: Perform depth fusion based on the 3D information, the 2D information, the depth information, the depth weight information, and the dense sampling depth corresponding to the depth map to obtain the first target depth of the first detected target;

[0094] Through the multi-source deep fusion process described above, the robustness of the depth is improved, enabling the dynamic selection of more reliable depth sources based on the target state, thereby enhancing robustness in complex scenarios.

[0095] Step 106: Obtain the three-dimensional detection result of the first target based on the first target depth, the category, the orientation, the 3D information, and the 2D information.

[0096] In the target detection method of this application, an RGB image is input into a visual backbone network to obtain multiple visual features output by the visual backbone network. The visual backbone network employs a hybrid strategy of local window attention and global modeling to extract features from the RGB image, thereby reducing computational complexity and memory overhead. Subsequently, feature reconstruction and feature fusion are performed on the multiple visual features to obtain fused features. The fused features are input into a deep network to obtain a depth map output by the deep network. The fused features and the depth map are input into a depth-aware decoder to obtain first decoding information output by the depth-aware decoder. The first decoding information includes the category, orientation, 3D information, 2D information, depth information, and depth weight information of a first detected target. The first detected target is a target in the RGB image. In this way, depth information can be used as prior information for decoding, solving the problem of the coarseness of current depth supervision methods. Finally, depth fusion is performed based on the 3D information, the 2D information, the depth information, the depth weight information, and the dense sampling depth corresponding to the depth map to obtain the first target depth of the first detected target. This solves the problem of a single depth source and lack of target-level adaptation, improving robustness. Finally, based on the first target depth, the category, the orientation, the 3D information, and the 2D information, the three-dimensional detection result of the first detected target is obtained. Ultimately, real-time and accurate target detection is achieved, solving the problems of low timeliness and accuracy in current detection methods.

[0097] As an optional implementation, step 101 involves inputting an RGB image into a visual backbone network to obtain multiple visual features output by the visual backbone network, including:

[0098] Sub-step A1: Divide the RGB image into multiple spatially non-overlapping local windows;

[0099] like Figure 3 As shown, an RGB image can be divided into K (e.g., Figure 3 The diagram shows 16 spatially non-overlapping local windows, where each window contains M marker units, satisfying N=K*M, where N is the number of marker units.

[0100] Sub-step A2: In the multiple first network layers of the visual backbone network, perform local modeling on each of the local windows;

[0101] It should be noted that for the sequence of labeled units formed after image patch embedding of the input RGB image, if standard global self-attention is used, each labeled unit needs to interact with all labeled units in pairs, and its computational complexity is proportional to the square of the number of labeled units N. When the roadside image resolution is high, N will increase rapidly, leading to a significant increase in inference latency and memory usage. In the above sub-step A1, the RGB image is divided into K spatially non-overlapping local windows, and the above sub-step A2 only performs self-attention independently within each local window; therefore, the total complexity of window attention can be uniformly expressed as O(K·M²), which, compared to O(N²) of standard global self-attention, allows the complexity to increase approximately linearly with the total number of labeled units N when the window size is fixed, making it more suitable for high-resolution roadside image scenarios.

[0102] Sub-step A3: Globally model the RGB image within multiple second network layers of the visual backbone network; wherein, the second network layers are, for example, key output layers in the visual backbone network. Global modeling within these key output layers avoids the problem of insufficient cross-window information interaction caused by purely local window modeling.

[0103] The above sub-steps A2 and A3 achieve efficient local modeling in most layers of the visual backbone network, and maintain long-range context and overall structural awareness through global interaction in key layers, thus balancing deployment efficiency and semantic expression.

[0104] Sub-step A4: Based on the results of the local modeling and the global modeling, obtain multiple visual features.

[0105] In the above-mentioned optional implementation, local modeling is performed on each of the local windows in multiple first network layers of the visual backbone network; and global modeling is performed on the RGB image in multiple second network layers of the visual backbone network; visual features are obtained based on the local modeling results and the global modeling results. This solves the problem that the Transformer based on global self-attention needs to process a large number of labeling units in high-resolution images, resulting in high computational complexity and memory overhead, which is not conducive to stable real-time deployment on roadside servers or edge devices, and improves coding efficiency.

[0106] As an optional implementation, step 102 involves reconstructing and fusing features from the multiple visual features to obtain fused features corresponding to the multiple visual features, including:

[0107] Sub-step B1: Map the multiple visual features to the same channel dimension through multi-scale projection;

[0108] Sub-step B2: Reconstruct the mapped visual features to obtain a multi-scale feature pyramid; here, using... Figure 2 For example, multi-scale feature pyramids include P3, P4, and P5 pyramid features.

[0109] Sub-step B3: Perform feature fusion on each feature map in the multi-scale feature pyramid to obtain the fused features.

[0110] As a specific implementation, sub-step B3 above involves fusing features from each feature map in the multi-scale feature pyramid to obtain the fused features, including:

[0111] Sub-step B21: Downsample the P3 feature map in the multi-scale feature pyramid to the main scale, and upsample the P5 feature map in the multi-scale feature pyramid to the main scale, wherein the main scale is the scale of the P4 feature map in the multi-scale feature pyramid;

[0112] Sub-step B22: Adaptively weightedly fuse the downsampled P3 feature map, the upsampled P5 feature map, and the P4 feature map to obtain the fused feature.

[0113] For example, the specific implementation described above involves downsampling the P3 pyramid features to the main scale, upsampling the P5 pyramid features to the main scale, leaving the P4 pyramid features unchanged, and then adaptively weighting and combining the three features using a gating fusion module. This allows subsequent depth branches to simultaneously utilize high-resolution details, mid-scale main structure, and low-resolution context, thus providing multi-scale semantic and detailed information for subsequent detection and depth estimation.

[0114] As an optional implementation, step 103 involves inputting the fused features into a deep network to obtain a depth map output by the deep network, including:

[0115] Sub-step C1: The deep network predicts the depth distribution of each pixel location in the fused feature; and obtains the unnormalized score for each pixel location;

[0116] Sub-step C2: Normalize the depth values ​​in the depth distribution; where the depth values ​​are the unnormalized scores of each pixel position; where sub-step C2 can be based on the softmax layer of the depth network for normalization.

[0117] Sub-step C3: Based on the probability of each depth in the depth distribution, perform a weighted summation of the normalized depth values; this allows for the recovery of continuous depth values, rather than simply selecting the category with the highest response. Figure 4 For example, the left side shows traditional hard labels, which only assign the true depth to the nearest single depth interval; the right side shows Gaussian soft labels obtained using this implementation method, which construct a continuous distribution across depth intervals near the true depth, allowing multiple adjacent depth intervals to participate in supervision simultaneously, which is more in line with the characteristics of continuous depth estimation.

[0118] Sub-step C4: Obtain the depth map based on the summation results corresponding to each pixel position.

[0119] Specifically, sub-step C4 can involve constructing a weighted distribution centered on the 3D center projection, giving pixels closer to the true 3D center higher supervision weights. Simultaneously, object-level reduction is used instead of simple full-pixel averaging to prevent small targets from being ignored during training due to their limited pixel count. The depth branch ultimately outputs three types of results: depth distribution prediction, probability-weighted continuous depth map, and depth location encoding. These are used for subsequent dense depth supervision, continuous depth recovery, and depth prior injection in the decoding stage, respectively.

[0120] Furthermore, as an optional implementation, the method also includes:

[0121] The dense sampling depth is obtained by predicting the depth of the three-dimensional center position of the first detected target through Gaussian sampling in the depth map.

[0122] Building upon the aforementioned implementation methods, the depth-aware decoder can, for example, employ a query vector-driven end-to-end detection mechanism. Each query vector, in addition to containing a learnable representation, corresponds to a reference point indicating the reference position of the current query vector in the image plane. Unlike traditional decoding methods that rely solely on appearance features, this approach samples the local depth prior at the corresponding position from the depth position encoding based on the reference point in each decoder layer, and injects this depth prior into the query vector update process through a linear mapping.

[0123] Therefore, while performing self-attention modeling of relationships between targets, the query vector also extracts appearance information from the P3, P4, and P5 multi-scale visual features obtained through feature reconstruction via deformable cross-attention, and receives geometric cues from local depth priors at the same layer. This mechanism ensures that depth information is no longer merely a post-processing supplement, but continuously participates in 2D localization and 3D attribute prediction during the query vector iteration process.

[0124] After multiple layers of decoder iterations, each query vector outputs results including category, 2D predicted bounding box, 3D center offset, query vector regression depth, depth uncertainty, 3D size, and orientation. The 3D center offset is preferably a normalized offset relative to the center of the 2D bounding box, ensuring stable representation even if the 3D center projection falls outside the 2D bounding box. In other words, the core of the decoding stage is to first introduce local depth priors at the query vector level, and then use the query vectors to uniformly predict target attributes, thereby enhancing the stability of 3D localization in complex scenes.

[0125] As an optional implementation, step 105, performing depth fusion based on the 3D information, the 2D information, the depth information, the depth weight information, and the dense sampling depth corresponding to the depth map to obtain the first target depth of the first detected target, includes:

[0126] Sub-step D1: Determine the geometric depth estimate of the target based on the size in the 3D information, the 2D information, and the camera intrinsic parameters corresponding to the RGB image; the camera is a device that captures the RGB image;

[0127] The above sub-steps can be understood as: geometric depth estimation (which can also be described as geometric analytical depth z). geo The value is derived from the target height (i.e., the size in 3D information), the 2D frame height (i.e., 2D information), and the camera focal length (i.e., camera intrinsic parameters).

[0128] Sub-step D2: Based on the depth weight information, perform depth fusion on the dense sampling depth, the geometric depth estimation, and the depth information to obtain the first target depth.

[0129] The aforementioned sub-step D2 specifically predicts an additional set of 3D fusion weight vectors for each query vector. After Softmax normalization, these weights correspond to three paths: regression depth, geometric depth, and dense sampling depth. The first target depth is obtained through depth fusion based on these three sources. This ensures that the first target depth is adaptively generated from the current query vector representation, rather than being a manually preset constant. Therefore, the model can dynamically favor more reliable depth sources based on the target's texture quality, scale, occlusion level, viewpoint changes, and geometric stability. For example, when the target texture is clear, more reliance can be placed on depth information; when the geometric relationships are more reliable, the weight of geometric depth estimation can be increased; and when the local quality of the dense depth map is higher, the weight of dense sampling depth can be increased. In this way, the contribution ratio of multi-source depth can be determined by the query vector itself, thereby achieving target-level, adaptive, and learnable depth fusion.

[0130] It should be noted that the above-mentioned dense sampling depth can also be described as dense sampling depth, which is obtained by sampling and predicting the depth of the three-dimensional center position on a probability-weighted continuous depth map, and is suitable for using pixel-level depth map information; the above-mentioned geometric depth estimation is suitable for targets with clear geometric relationships and stable scale; the above-mentioned depth information can also be called regression depth, which is directly predicted from the features of the query vector itself, and is suitable for scenarios with strong semantic representation and clear target appearance.

[0131] Based on the above sub-step D2, it can be seen that the geometric depth estimation is no longer used as a fixed empirical value, but is obtained by combining the category prior size and the size residual prediction. Therefore, it can participate in the training update and adapt to changes in target category, scale and viewpoint.

[0132] As an optional implementation, the visual backbone network, the depth map network, and the depth-aware decoder constitute an object detection model. Furthermore, the method further includes:

[0133] Step E1: For each pixel location in the training image, construct Gaussian soft distribution labels for multiple depth intervals adjacent to the true depth of the pixel location;

[0134] Step E1 above does not roughly map the true depth to a single discrete depth interval. Instead, it constructs Gaussian soft-distribution labels around the true depth for multiple adjacent depth intervals, allowing nearby depth intervals to simultaneously obtain continuous training signals. Compared to the traditional hard-label approach that only assigns labels to the nearest single depth interval, this method can alleviate the quantization error caused by hard depth discretization and enhance the supervised smoothness near the boundaries of depth intervals.

[0135] Step E2: Using the Gaussian distributed soft labels, the depth map network supervises the prediction process of the depth values ​​at each pixel location during the training of the target detection model.

[0136] Furthermore, as an optional implementation, the method also includes:

[0137] Step F1: During the training of the target detection model, based on the 2D information in the second decoding information, the second target depth of the second detected target, and the camera intrinsic parameters, the center of the second detected target is back-projected from the image plane to the three-dimensional camera coordinate system to obtain the predicted three-dimensional center of the second detected target; wherein, the second detected target is the target in the training image, the second decoding information is the decoding result of the second detected target; wherein, the camera intrinsic parameters are the intrinsic parameters of the camera that captured the training image;

[0138] It should be noted that the second target depth is the depth information of the second detected target output by the depth-aware decoder, the depth obtained by fusing the dense sampling depth corresponding to the second detected target and the geometric depth estimation.

[0139] In other words, after obtaining the two-dimensional projection center and fusion depth (second target depth) of the second detection target, the center of the second detection target is back-projected from the image plane to the three-dimensional camera coordinate system using the camera intrinsic parameter matrix K, and the back-projection result is used together with the real three-dimensional center to construct geometric consistency supervision.

[0140] Step F2: Based on the predicted 3D center, the true 3D center of the second detection target, and the pre-constructed geometric consistency loss function, determine the consistency loss corresponding to the second detection target; wherein, this step can be implemented based on the following formula:

[0141]

[0142] Where Lgeo represents the geometric consistency loss, This represents the fusion depth z of the two-dimensional projection center (u, v). fuse The predicted 3D center, C, obtained by back projection gt The true 3D center is represented by K; the camera intrinsic parameter matrix is ​​represented by u and v, which represent the horizontal and vertical coordinates of the center of the second detected target in the image plane, respectively.

[0143] It should be noted that the purpose of this loss term is not to supervise depth separately again, but to unify the 2D bounding box, center offset, fused depth, and camera imaging model into a single 3D closed loop. In this way, by forming a closed loop through forward and backward projection, the 2D localization results and 3D coordinates together satisfy the camera imaging relationship, thereby enhancing the 3D localization stability of distant and small targets.

[0144] Step F3: Jointly train the target detection model based on the consistency loss and the loss corresponding to each item in the second decoded information.

[0145] The target detection method in the embodiments of this application takes a single RGB image and corresponding camera intrinsic parameters as input, and outputs the target's category, 2D bounding box, 3D center, size, and orientation. Below, in conjunction with... Figure 2 The steps included in one embodiment of the method are described below.

[0146] First, feature extraction of the DINO network backbone (Window Attention VIT) DINOv2 Backbone;

[0147] It should be noted that before the aforementioned "first" step, the method also includes: preprocessing the input image by scaling, cropping, and flipping, and simultaneously updating the camera intrinsic parameters; and constructing supervisory signals such as 2D bounding boxes, 3D center projections, depth, 3D dimensions, and calibration information. Based on this, the "first" step can be: inputting the preprocessed image into a window that focuses on the visual backbone, using local window attention to reduce complexity in most layers, retaining the global modeling path in the key output layer, and outputting multiple visual features. In other words, the aforementioned "first" step is an efficient visual encoding based on window attention; specifically:

[0148] A hybrid strategy of "local window attention in non-critical layers + global modeling in critical output layers" is adopted in the visual backbone. The input image is processed by image patch embedding to form a sequence of labeled units. If standard global self-attention is used, each labeled unit needs to interact with all other labeled units, and its computational complexity is proportional to the square of the number of labeled units N. When the roadside image resolution is high, N increases rapidly, leading to a significant increase in inference latency and memory usage. Therefore, to reduce complexity, the "first" step divides the feature map into K spatially non-overlapping local windows, each containing M labeled units, satisfying N = K·M. In most layers, self-attention is performed independently only within each window; therefore, the total complexity of window attention can be uniformly expressed as O(K·M²), equivalently written as O(N·M) or O(N² / K). Compared to the standard global self-attention's O(N²), this structure allows the complexity to grow approximately linearly with the total number of labeled units N when the window size is fixed, making it more suitable for high-resolution roadside image scenarios.

[0149] Furthermore, to avoid insufficient cross-window information interaction caused by purely local window modeling, the backbone network in this step retains global modeling paths in several key output layers and uses these layers as multi-scale feature extraction points. In other words, the backbone network pursues efficient local modeling in most layers, while maintaining long-range context and overall structural awareness through global interaction in key layers, thus balancing deployment efficiency and semantic expression.

[0150] Second, the Multi-Scale Projector / Neck is used to align the multiple intermediate layer features output from the backbone to a unified channel dimension via a multi-scale projection mapping module, and reconstruct them into P3, P4, and P5 pyramid features. The high-resolution layers retain more local details, while the low-resolution layers retain stronger global semantics, which is beneficial for subsequent deep modeling and 3D object detection. The relationship between window attention and key layer global modeling is as follows: Figure 3 As shown.

[0151] Third, a multi-scale depth branch is constructed to fuse P3, P4, and P5 features at a unified scale to predict the unnormalized score of the pixel-by-pixel depth distribution. Specifically, the multi-scale depth branch receives features from P3, P4, and P5, transforms them to a unified scale, and then fuses them. Specifically, P3 is downsampled to the main scale, P5 is upsampled to the main scale, and P4 remains unchanged. Then, a gated fusion module adaptively weights and combines the three features. This design allows the depth branch to simultaneously utilize high-resolution details, mid-scale main structure, and low-resolution context.

[0152] Fourth, the depth branch is trained under Gaussian soft distribution supervision, and a continuous dense depth map is recovered through probability weighting, while simultaneously generating depth location codes; this step can be performed by... Figure 2 The implementation of Depth Map Network (DDN), Depth Map Coding, etc.

[0153] It should be noted that the above steps do not roughly map the true depth to a single discrete depth interval, but rather around the true depth d. gt Construct Gaussian soft distribution labels q for multiple adjacent depth intervals k This method enables continuous training signals to be obtained simultaneously for similar depth intervals. Compared to the traditional hard-label approach that assigns labels only to the nearest single depth interval, this method can alleviate the quantization error caused by hard depth discretization and enhance the supervised smoothness near the depth interval boundaries. Specifically:

[0154]

[0155] In the formula, q k Represents the true depth d gt The soft label value d in the k-th depth interval gt d represents the true depth of the target. k Let represent the physical depth corresponding to the center of the k-th depth interval, σ represent the standard deviation of the Gaussian distribution, and Z represent the normalization factor, which is used to ensure that the sum of the soft labels in each depth interval is 1.

[0156] During the inference phase, the depth branch outputs an unnormalized depth distribution score for each pixel location. After Softmax normalization, the physical depth values ​​corresponding to each depth interval are summed using probability weighting to recover continuous depth values, rather than simply taking the largest response category. Figure 4 As shown, the left side represents traditional hard labels, which only assign the true depth to the nearest single depth interval; the right side represents Gaussian soft labels used in this invention, which construct a continuous distribution across depth intervals near the true depth, allowing multiple adjacent depth intervals to participate in supervision simultaneously, which is more in line with the characteristics of continuous depth estimation.

[0157] Furthermore, the aforementioned "fourth" step constructs a weighted distribution centered on the 3D center projection within the target supervision region, giving pixels closer to the true 3D center higher supervision weights. Simultaneously, object-level reduction is employed instead of simple full-pixel averaging to prevent small targets from being ignored during training due to their limited pixel count. The depth branch ultimately outputs three types of results: depth distribution prediction, probability-weighted continuous depth map, and depth location encoding. These are used for dense depth supervision, continuous depth recovery, and depth prior injection in the decoding stages, respectively.

[0158] Fifth, in the Transformer decoder, local depth priors are sampled from the depth position encoding using the query vector reference point as an index, and injected into the query vector update process. This allows the query vector to simultaneously utilize appearance features and local depth information during self-attention and cross-attention iterations, outputting results such as category, 2D bounding box, 3D center offset, depth, size, and orientation.

[0159] The aforementioned "fifth" step implements 3D localization based on depth-aware decoding. The decoder employs a query vector-driven end-to-end detection mechanism. Each query vector, in addition to containing a learnable representation, corresponds to a reference point indicating the reference position of the current query vector in the image plane. Unlike traditional decoding methods that rely solely on appearance features, in this approach, each decoder layer samples the local depth prior at the corresponding position from the depth position encoding based on the reference point, and injects this depth prior into the query vector update process through a linear mapping.

[0160] Therefore, while performing self-attention modeling of relationships between targets, the query vector also acquires appearance information from multi-scale visual features at P3, P4, and P5 through deformable cross-attention, and receives geometric cues from local depth priors at the same layer. This mechanism ensures that depth information is not merely a post-processing supplement, but continuously participates in 2D localization and 3D attribute prediction during the query vector iteration process.

[0161] After multiple layers of decoder iteration, each query vector outputs results such as category, 2D predicted bounding box, 3D center offset, query vector regression depth, depth uncertainty, 3D size, and orientation. The 3D center offset is preferably a normalized offset relative to the center of the 2D bounding box, ensuring stable representation even if the 3D center projection falls outside the 2D bounding box. The core of the decoding stage is to first introduce local depth prior at the query vector level, and then use the query vector to uniformly complete target attribute prediction, thereby enhancing the stability of 3D localization in complex scenes.

[0162] Sixth, for each query vector, three paths are constructed simultaneously: regression depth, geometric analytical depth, and dense depth map sampling depth. These three paths are then adaptively combined using the learnable fusion weights predicted by the query vector itself to obtain the target-level fusion depth. Figure 2(The final depth in the middle).

[0163] The sixth step described above achieves target depth estimation based on query vector-level learnable deep fusion weights. To avoid the failure of single-path depth prediction in certain scenarios, this step constructs three depth sources simultaneously for each query vector. The first is the regression depth z. reg The first type is directly predicted from the features of the query vector itself, which is suitable for scenarios with strong semantic representation and clear target appearance; the second type is the geometric analytical depth z. geo The first method, derived from the target height, the 2D bounding box height, and the camera focal length, is suitable for targets with clear geometric relationships and stable scale. The third method is the dense sampling depth z. map The depth of the 3D center position is obtained by sampling on a probability-weighted continuous depth map, which is suitable for utilizing pixel-level depth map information.

[0164] In other words, the geometric resolution depth is no longer used as a fixed empirical value, but is obtained by combining the category prior size and the size residual prediction. Therefore, it can participate in training updates and adapt to changes in target category, scale, and viewpoint.

[0165] Furthermore, an additional set of 3D fusion weight vectors is predicted for each query vector, which, after Softmax normalization, correspond to the three paths: regression depth, geometric depth, and dense sampling depth, respectively. The final fusion depth can be expressed as:

[0166]

[0167] In the formula, z fuse Indicates the final fusion depth, z reg z represents the depth obtained by direct regression from the query vector. geo The z-axis represents the geometric depth obtained from the analysis of the target size, the height of the 2D bounding box, and the camera intrinsic parameters. map This represents the depth sampled from a dense depth map; , and These represent the normalized fusion weights corresponding to the three depth sources.

[0168] The fusion weights are adaptively generated based on the current query vector representation, rather than being manually preset constants. Therefore, the model can dynamically favor more reliable depth sources based on the target's texture quality, scale, occlusion level, viewpoint changes, and geometric stability. For example, when the target texture is clear, it can rely more on z... reg z can be improved when the geometric relationship is more reliable. geo Weights can be increased when the local quality of the dense depth map is higher. mapWeighting. The innovation of this mechanism lies not in simple summation, but in the fact that the contribution ratio of multi-source depth is determined by the query vector itself, thereby achieving target-level, adaptive, and learnable deep fusion. A related illustration is shown below. Figure 5 As shown.

[0169] Seventh, based on the 2D projection center, fusion depth, and camera intrinsic parameters, the target center is back-projected from the image plane to the 3D camera coordinate system to obtain the predicted 3D center; then, a consistency loss is constructed with the real 3D center, thus forming an optimization closed loop composed of 2D position, depth, and camera geometry.

[0170] Eighth, the classification loss, 2D bounding box loss, center offset loss, depth loss, size loss, orientation loss, dense depth map loss, and geometric consistency loss are combined for joint training to output the final 3D detection result.

[0171] In other words, after obtaining the two-dimensional projection center (u, v) and the fusion depth z fuse Then, the target center can be back-projected from the image plane to the 3D camera coordinate system using the camera intrinsic parameter matrix K, and the back-projection result can be used together with the real 3D center to construct a geometric consistency supervision.

[0172]

[0173] In the formula, Lgeo represents the geometric consistency loss. This represents the fusion depth z of the two-dimensional projection center (u, v). fuse The predicted 3D center, C, obtained by back projection gt The target center is represented by ; K represents the camera intrinsic parameter matrix, and u and v represent the horizontal and vertical coordinates of the target center in the image plane, respectively.

[0174] The purpose of this loss term is not to supervise depth separately again, but to unify the 2D bounding box, center offset, fused depth, and camera imaging model into a single 3D closed loop. For example... Figure 6 As shown, the upper part illustrates the geometric inconsistency problem between traditional independent 2D center supervision and independent 3D center supervision, especially the problem that the supervision amount of distant targets is too small and is ignored; the lower part illustrates the joint geometric consistency supervision adopted by the present invention, which forms a closed loop through forward projection and backward projection, so that the 2D positioning results and 3D coordinates together satisfy the camera imaging relationship, thereby enhancing the 3D positioning stability of distant and small targets.

[0175] The object detection method described in this application has two aspects. First, it relates to an efficient visual coding structure based on window attention. This structure uses local window attention for most layers in the Vision Transformer backbone, while retaining the global modeling path for key output layers. This effectively reduces the computational cost of self-attention in high-resolution images while preserving global contextual representation capabilities, thus balancing computational efficiency and global semantic representation under high-resolution input. This is more suitable for the real-time deployment requirements of roadside perception. Second, it relates to a depth branch based on multi-scale fusion and soft distribution supervision. This branch aligns P3, P4, and P5 features to a unified scale and performs gated fusion. It uses Gaussian soft distribution supervision instead of one-hot supervision for pixel-by-pixel depth and recovers continuous depth through probability weighting. The depth supervision is more continuous and stable. By replacing one-hot hard discrete classification with Gaussian soft distribution supervision and using weighted summation to recover continuous depth, this invention can alleviate depth discretization errors and gradient discontinuities, improving the smoothness of depth estimation. The third aspect involves a query vector-level learnable multi-source deep fusion mechanism. This mechanism simultaneously constructs three paths for each query vector: regression depth, geometric analytical depth, and dense depth map sampling depth. Fusion weights are adaptively generated based on the query vector representation, and the three paths are combined using a target-level weighted combination. This improves the robustness of deep fusion. Specifically, by using query vector-level learnable weights to adaptively combine regression depth, geometric depth, and dense sampling depth at the target level, a more reliable depth source can be dynamically selected based on the target state, enhancing robustness in complex scenarios. The fourth aspect involves a geometric consistency constraint based on a closed loop of 2D projection and 3D backprojection. This geometric consistency is based on recovering the predicted 3D center using 2D center projection, fusion depth, and camera intrinsics, and constructing a consistency loss with the true 3D center, enabling joint optimization of 2D and 3D localization under a unified geometric model. Specifically, by constructing a backprojection closed loop constraint using 2D center projection, fusion depth, and camera intrinsics, the 2D localization results can be kept consistent with the 3D spatial coordinates, reducing 3D center drift and scale mismatch problems.

[0176] The aforementioned method organizes efficient visual encoding, multi-scale depth modeling, depth-aware decoding, query vector-level deep fusion, and geometric consistency constraints into a unified training and inference process, applicable to roadside perception, vehicle-to-everything (V2X) communication, and other monocular 3D target detection scenarios. It is also more effective for distant, small targets. Multi-scale feature reconstruction, center-weighted depth supervision, and object-level reduction strategies help enhance the effective gradient of small targets during training, making this application more suitable for real-world scenarios with distant, dense, and small targets in a roadside perspective. Furthermore, this application does not rely on LiDAR, multi-view cameras, or additional expensive sensors; it can be implemented solely based on monocular images and camera intrinsics, and can be deployed on roadside servers, edge computing devices, and other 3D perception systems with monocular imaging capabilities.

[0177] Embodiments of this application also provide a target detection device, such as... Figure 7 As shown, the device includes:

[0178] The first processing module 701 is used to input an RGB image into a visual backbone network and obtain multiple visual features output by the visual backbone network. The visual backbone network uses a hybrid strategy of local window attention and global modeling to extract features from the RGB image.

[0179] The second processing module 702 is used to perform feature reconstruction and feature fusion on multiple visual features to obtain fused features;

[0180] The third processing module 703 is used to input the fused features into the deep network to obtain the depth map output by the deep network;

[0181] The fourth processing module 704 is used to input the fused features and the depth map into the depth-aware decoder to obtain the first decoding information output by the depth-aware decoder; the first decoding information includes the category, orientation, 3D information, 2D information, depth information and depth weight information of the first detected target; wherein, the first detected target is the target in the RGB image;

[0182] The fifth processing module 705 is used to perform depth fusion based on the 3D information, the 2D information, the depth information, the depth weight information, and the dense sampling depth corresponding to the depth map to obtain the first target depth of the first detected target;

[0183] The detection module 706 is used to obtain the three-dimensional detection result of the first target based on the first target depth, the category, the orientation, the 3D information and the 2D information.

[0184] The first processing module 701 includes:

[0185] The first processing submodule is used to divide the RGB image into multiple spatially non-overlapping local windows;

[0186] The second processing submodule is used to perform local modeling of each of the local windows in multiple first network layers of the visual backbone network.

[0187] The third processing submodule is used to perform global modeling of the RGB image in multiple second network layers of the visual backbone network;

[0188] The fourth processing submodule is used to obtain multiple visual features based on the results of the local modeling and the results of the global modeling.

[0189] The second processing module 702 includes:

[0190] The fifth processing submodule is used to map multiple visual features to the same channel dimension through multi-scale projection;

[0191] The sixth processing submodule is used to reconstruct the mapped visual features to obtain a multi-scale feature pyramid.

[0192] The seventh processing submodule is used to perform feature fusion on each feature map in the multi-scale feature pyramid to obtain the fused features.

[0193] The seventh processing submodule includes:

[0194] The first processing unit is configured to downsample the P3 feature map in the multi-scale feature pyramid to the main scale and upsample the P5 feature map in the multi-scale feature pyramid to the main scale, wherein the main scale is the scale of the P4 feature map in the multi-scale feature pyramid;

[0195] The second processing unit is used to adaptively weight and fuse the downsampled P3 feature map, the upsampled P5 feature map, and the P4 feature map to obtain the fused feature.

[0196] The third processing module 703 includes:

[0197] The eighth processing module is used for the deep network to predict the depth distribution of each pixel location in the fused features;

[0198] The ninth processing module is used to normalize the depth values ​​in the depth distribution;

[0199] The tenth processing module is used to perform a weighted summation of the normalized depth values ​​based on the probability of each depth in the depth distribution.

[0200] The eleventh processing module is used to obtain the depth map based on the summation results corresponding to each pixel position.

[0201] The device further includes:

[0202] The prediction module is used to predict the depth of the three-dimensional center position of the first detected target by Gaussian sampling in the depth map, thereby obtaining the dense sampling depth.

[0203] The fifth processing module 705 includes:

[0204] The determination submodule is used to determine the geometric depth estimate of the target based on the size in the 3D information, the 2D information, and the camera intrinsic parameters corresponding to the RGB image; the camera is a device that captures the RGB image;

[0205] The fusion submodule is used to perform depth fusion on the dense sampling depth, the geometric depth estimation and the depth information according to the depth weight information to obtain the first target depth.

[0206] The device further includes:

[0207] The seventh processing module is used to construct Gaussian soft distribution labels for multiple depth intervals adjacent to the true depth of the pixel position for each pixel position in the training image.

[0208] The supervision module is used to supervise the depth map network's prediction process of depth values ​​at each pixel location during the training of the target detection model, using the Gaussian distributed soft labels.

[0209] The device further includes:

[0210] The eighth processing module is used, during the training of the target detection model, to back-project the center of the second detected target from the image plane to the three-dimensional camera coordinate system based on the 2D information in the second decoding information, the second target depth of the second detected target, and the camera intrinsic parameters, to obtain the predicted three-dimensional center of the second detected target; wherein, the second detected target is the target in the training image, the second decoding information is the decoding result of the second detected target; and the camera intrinsic parameters are the intrinsic parameters of the camera that captured the training image.

[0211] The determination module is used to determine the consistency loss corresponding to the second detection target based on the predicted 3D center, the true 3D center of the second detection target, and the pre-constructed geometric consistency loss function;

[0212] The training module is used to jointly train the target detection model based on the consistency loss and the loss corresponding to each item in the second decoded information.

[0213] It should be noted that the target detection device provided in this application embodiment can implement all the method steps implemented in the target detection method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0214] Embodiments of this application also provide a target detection device, such as... Figure 8As shown, it includes a transceiver 810, a processor 800, a memory 820, and a program or instructions stored in the memory 820 and executable on the processor 800; when the processor 800 executes the program or instructions, it implements the above-mentioned target detection method.

[0215] The transceiver 810 is used to receive and send data under the control of the processor 800.

[0216] Among them, Figure 8 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 800) and memory (memory 820). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 810 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium.

[0217] The processor 800 is responsible for managing the bus architecture and general processing, while the memory 820 can store the data used by the processor 800 during operation.

[0218] An embodiment of this application provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps in the target detection method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0219] The processor mentioned above is the processor in the target detection device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0220] Through the above description of the implementation methods, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, disk, optical disk) and includes several instructions for executing the methods described in the various embodiments of this application.

[0221] Therefore, embodiments of this application also provide a computer program product, including computer instructions, which, when executed by a processor, implement the target detection method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0222] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different locations, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0223] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0224] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0225] The exemplary embodiments described above are illustrated with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of this application, and therefore, this application should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this application complete and convey the scope of this application to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless explicitly stated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, elements, and / or groups thereof. Unless otherwise indicated, a range of values ​​is stated to include the upper and lower limits of the range and any subranges therebetween.

[0226] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A target detection method, characterized in that, include: An RGB image is input into a visual backbone network to obtain multiple visual features output by the visual backbone network. The visual backbone network uses a hybrid strategy of local window attention and global modeling to extract features from the RGB image. The visual features are reconstructed and fused to obtain fused features; The fused features are input into a deep network to obtain a depth map output by the deep network; The fused features and the depth map are input to a depth-aware decoder to obtain first decoding information output by the depth-aware decoder; the first decoding information includes the category, orientation, 3D information, 2D information, depth information, and depth weight information of the first detected target; wherein, the first detected target is the target in the RGB image; The first target depth of the first detected target is obtained by performing depth fusion based on the 3D information, the 2D information, the depth information, the depth weight information, and the dense sampling depth corresponding to the depth map; Based on the first target depth, the category, the orientation, the 3D information, and the 2D information, the three-dimensional detection result of the first detected target is obtained.

2. The method according to claim 1, characterized in that, An RGB image is input into a visual backbone network to obtain multiple visual features output by the visual backbone network, including: The RGB image is divided into multiple spatially non-overlapping local windows; In the multiple first network layers of the visual backbone network, local modeling is performed on each of the local windows; In the multiple second network layers of the visual backbone network, the RGB image is globally modeled; Based on the results of the local modeling and the global modeling, multiple visual features are obtained.

3. The method according to claim 1, characterized in that, Feature reconstruction and feature fusion are performed on multiple visual features to obtain fused features corresponding to the multiple visual features, including: Multiple visual features are mapped to the same channel dimension via multi-scale projection. The mapped visual features are reconstructed to obtain a multi-scale feature pyramid; The feature maps in the multi-scale feature pyramid are fused to obtain the fused features.

4. The method according to claim 3, characterized in that, The feature fusion is performed on each feature map in the multi-scale feature pyramid to obtain the fused feature, including: The P3 feature map in the multi-scale feature pyramid is downsampled to the main scale, and the P5 feature map in the multi-scale feature pyramid is upsampled to the main scale, wherein the main scale is the scale of the P4 feature map in the multi-scale feature pyramid; The downsampled P3 feature map, the upsampled P5 feature map, and the P4 feature map are adaptively weighted and fused to obtain the fused feature.

5. The method according to claim 1, characterized in that, The fused features are input into a deep network to obtain a depth map output by the deep network, including: The deep network predicts the depth distribution of each pixel location in the fused feature; Normalize the depth values ​​in the depth distribution; Based on the probability of each depth in the depth distribution, the normalized depth values ​​are weighted and summed. The depth map is obtained based on the summation results corresponding to each pixel position.

6. The method according to claim 1, characterized in that, The method further includes: The dense sampling depth is obtained by predicting the depth of the three-dimensional center position of the first detected target through Gaussian sampling in the depth map.

7. The method according to claim 1 or 6, characterized in that, Depth fusion is performed based on the 3D information, the 2D information, the depth information, the depth weight information, and the dense sampling depth corresponding to the depth map to obtain the first target depth of the first detected target, including: The geometric depth estimate of the target is determined based on the size in the 3D information, the 2D information, and the camera intrinsic parameters corresponding to the RGB image; the camera is a device that captures the RGB image. Based on the depth weight information, the dense sampling depth, the geometric depth estimation, and the depth information are fused to obtain the first target depth.

8. The method according to any one of claims 1 to 7, characterized in that, The visual backbone network, the depth map network, and the depth-aware decoder constitute the object detection model, and the method further includes: For each pixel location in the training image, Gaussian soft distribution labels are constructed for multiple depth intervals adjacent to the true depth of the pixel location; During the training process of the target detection model, the depth map network supervises the prediction process of the depth values ​​at each pixel location using the Gaussian distributed soft labels.

9. The method according to claim 8, characterized in that, The method further includes: During the training of the target detection model, based on the 2D information in the second decoding information, the second target depth of the second detected target, and the camera intrinsic parameters, the center of the second detected target is back-projected from the image plane to the three-dimensional camera coordinate system to obtain the predicted three-dimensional center of the second detected target; wherein, the second detected target is the target in the training image, the second decoding information is the decoding result of the second detected target; and the camera intrinsic parameters are the intrinsic parameters of the camera that captured the training image. Based on the predicted 3D center, the true 3D center of the second detection target, and the pre-constructed geometric consistency loss function, determine the consistency loss corresponding to the second detection target; The target detection model is jointly trained based on the consistency loss and the loss corresponding to each item in the second decoded information.

10. A target detection device, characterized in that, include: The first processing module is used to input an RGB image into a visual backbone network and obtain multiple visual features output by the visual backbone network. The visual backbone network uses a hybrid strategy of local window attention and global modeling to extract features from the RGB image. The second processing module is used to perform feature reconstruction and feature fusion on multiple visual features to obtain fused features; The third processing module is used to input the fused features into the deep network to obtain the depth map output by the deep network; The fourth processing module is used to input the fused features and the depth map into the depth-aware decoder to obtain the first decoding information output by the depth-aware decoder; the first decoding information includes the category, orientation, 3D information, 2D information, depth information and depth weight information of the first detected target; wherein, the first detected target is the target in the RGB image; The fifth processing module is used to perform depth fusion based on the 3D information, the 2D information, the depth information, the depth weight information, and the dense sampling depth corresponding to the depth map to obtain the first target depth of the first detected target; The detection module is used to obtain the three-dimensional detection result of the first target based on the first target depth, the category, the orientation, the 3D information and the 2D information.

11. A target detection device, comprising a transceiver, a processor, a memory, and a program stored in the memory and executable on the processor; characterized in that, The transceiver is used to send and receive data under the control of the processor, and the processor implements the target detection method as described in any one of claims 1 to 9 when executing the program.

12. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the target detection method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the target detection method as described in any one of claims 1 to 9.

Images