A monocular 3D target detection method and system based on deep guidance

Abstract

The application discloses a monocular 3D target detection method and system based on depth guidance, and belongs to the technical field of 3D target detection. The method adopts a dynamic positioning and scale perception Transformer method for monocular 3D detection, introduces a dynamic and explicit position coding query (DEP-query) and a scale auxiliary deformation attention (SDA) module to help the original query have valuable spatial and content clues. The DEP-query utilizes the explicit position prior of 3D projection coordinates to improve the accuracy of query positioning, so that the attention layer in the decoder avoids noisy background information. The SDA module optimizes the receptive field learning of the query through the size prior of the corresponding 2D box, so that the query can obtain high-quality visual features. Neither the position prior nor the size prior needs any additional data, and is updated in each layer of the decoder to provide long-term assistance.

Description

Technical Field

[0001] This application belongs to the field of 3D target detection technology, specifically relating to a depth-guided monocular 3D target detection method and system. Background Technology

[0002] Recent advances in 3D object detection have significantly propelled its application across various fields, including autonomous driving and indoor robotics. Monocular 3D object detection, in particular, has garnered considerable attention due to its potential to reduce computational and equipment costs. However, most existing monocular 3D detection methods are based on traditional 2D object detectors, which can only utilize local features around the predicted object center. The lack of scene-level global cues has become a bottleneck for further performance improvements. More importantly, some methods require tedious manual parameter tuning, such as anchor points, region proposals, and NMS thresholding, while others heavily rely on the geometric relationships between 2D and 3D objects.

[0003] In contrast, recent Transformer-based monocular 3D object detectors have successfully improved performance by fully leveraging global cues learned through self-attention mechanisms. MonoDTR, for example, pre-generates a ground truth depth map to train a depth estimation network that accurately infers depth values, which are then used as queries in the Transformer decoder. MonoDETR fully interacts with depth and visual features by incorporating a depth cross-attention layer in the decoder without relying on any additional data. However, MonoDTR requires additional depth information from LiDAR data, and its training convergence is slow and unstable due to unavoidable background noise. Summary of the Invention

[0004] The purpose of this application is to provide a depth-guided monocular 3D target detection method and system, which improves the decoder by introducing a dynamic position and scale-aware decoder (DPS-decoder). The DPS-decoder includes an explicit position encoding query and a scale-assisted deformable attention module, which helps to accurately locate objects at the scene level, avoid background noise, and locate effective regions for category and bounding box prediction, thereby solving at least one of the technical problems involved in the background art.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] This application provides a depth-guided monocular 3D target detection method, including the following steps:

[0007] Step 1: Data preprocessing. Select a 3D object detection benchmark dataset, which includes vehicle images and pedestrian / bicycle images. Preprocess the selected dataset by dividing it into a training set, a validation set, and a test set.

[0008] Step 2: Feature extraction. Input the images from the preprocessed dataset into the ResNet-50 backbone network to obtain the multi-scale visual features and depth features corresponding to the images.

[0009] Step 3: Feature Encoding. A visual encoder and a depth predictor are used to globally encode the obtained visual and depth features, respectively, to generate a visual representation of the image. And foreground depth map D fg ;

[0010] Step 4: Feature Decoding, using a dynamic position and scale-aware decoder with multiple decoder layers to extract the visual representation corresponding to the generated image. And foreground depth map D fg The algorithm adaptively extracts visual and depth features to obtain a query corresponding to an object in the image. Each decoder layer first passes through a depth self-attention layer to remove duplicate predictions, performs interaction between depth embeddings output by the previous depth predictor for depth feature fusion, and then passes through a scale-assisted deformable attention layer to aggregate the embeddings output from the visual encoder to optimize the decoder embeddings, thereby improving the prediction of categories and boxes.

[0011] Step 5: 3D attribute extraction. Multiple detection heads are used to extract 3D attributes from the query obtained after feature decoding.

[0012] As a preferred improvement of this application, in step S1, the 3D object detection benchmark dataset is the KITTI 3D object detection benchmark dataset, which includes 7,481 training images and 7,518 test images, wherein the sub-training set and the validation set contain 3,712 and 3,769 sample images, respectively.

[0013] As a preferred improvement of this application, in step S2, the multi-scale features corresponding to the image are obtained from the last three stages of the ResNet50 backbone network. These are the initial resolutions. After unifying the resolution, element-wise addition and merging are performed, and finally the highest-level feature is used. As a visual feature f V Based on these multi-scale visual features f V The depth features of the image are obtained by using a two-layer convolutional network.

[0014] As a preferred improvement of this application, in step S3, each visual encoder consists of a global self-attention layer and a feedforward network. Through the global self-attention layer, the depth predictor can learn different foreground regions rather than local spatial information. The depth predictor uses a convolutional network to predict a foreground depth map for the depth features, and only predicts the depth value of the foreground object, while setting the pixel value of the background part to 0.

[0015] As a preferred improvement of this application, in step S3, the visual encoder and the depth predictor adopt a parallel structure design and use a self-attention-based transformer structure to interact with the long-distance semantic information within the two features. Specifically, for the visual encoder, three layers of blocks are stacked to parse richer RGB representation information; for the depth predictor, only one layer of block is used to understand the depth information of a single channel.

[0016] As a preferred improvement to this application, the training process of the above method is performed on an RTX A6000 GPU using the Adam optimizer over 100 epochs, with a batch size of 16 and a learning rate of 2×10⁻⁶. -4 The weight decays to 10. -4 .

[0017] This application also provides a depth-guided monocular 3D target detection system for performing the method, comprising:

[0018] The data preprocessing module is used to select a 3D object detection benchmark dataset, which includes vehicle images and pedestrian and bicycle images. The selected dataset is preprocessed and divided into a training set, a validation set, and a test set.

[0019] The feature extraction module is used to input the images in the preprocessed dataset into the ResNet-50 backbone network to obtain the multi-scale visual features and depth features corresponding to the images.

[0020] The feature encoding module is used to globally encode the obtained visual and depth features using a visual encoder and a depth predictor, respectively, to generate a visual representation of the image. And foreground depth map D fg ;

[0021] The feature decoding module is used to extract the visual representation corresponding to the generated image from a dynamic position and scale-aware decoder containing multiple decoder layers. And foreground depth map D fgThe algorithm adaptively extracts visual and depth features to obtain queries corresponding to objects in an image. Each decoder layer first passes through a depth self-attention layer to remove duplicate predictions, performing interactions between depth embeddings output from previous depth predictors for depth feature fusion. Then, a scale-assisted deformable attention layer aggregates embeddings output from the visual encoder to optimize the decoder embeddings, thereby improving category and bounding box predictions.

[0022] The 3D attribute extraction module is used to extract 3D attributes from a query obtained after feature decoding using multiple detection heads.

[0023] The beneficial effects of the embodiments of this application are as follows:

[0024] (1) For the automotive category, the method proposed in this application outperforms several existing monocular 3D object detection methods on the official test set, showing the most efficient performance. It also shows competitive performance compared to methods that require additional data, achieving the highest or second-highest accuracy. Compared to MonoDTR, which also uses the Transformer framework, the method proposed in this application has a training convergence speed that is 2.5 times faster and a testing speed that is 4 milliseconds faster.

[0025] (2) In the categories of “pedestrians” and “bicycles”, the method proposed in this application outperforms all existing methods at three difficulty levels, whether the method requires additional data or not, and achieves accurate detection of objects with diverse appearances.

[0026] (3) By introducing Dynamic and Explicit Location Encoding Query (DEP-query) and Scale-Assisted Deformation Attention (SDA) modules, this application makes full use of valuable location and size priors, helping the query to accurately locate the object in fewer training cycles and generate higher quality query features;

[0027] (4) The method proposed in this application successfully balances training convergence speed, inference time and detection accuracy, indicating that it has high applicability in real-world scenarios. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0029] Figure 1A convergence curve diagram on the KITTI validation set provided for embodiments of this application;

[0030] Figure 2 (a)-(c) are comparison diagrams of MonoDTR, MonoDETR and MonoPSTR proposed in this application provided in the embodiments of this application, wherein, Figure 2 (a) MonoDTR uses the learned deep features as input queries for the decoder and contains only two multi-head linear attention modules in the decoder layer, but requires auxiliary supervision from LiDAR point clouds; Figure 2 (b) The MonoDETR uses random encoded constants and decoder embeddings as queries for the initial attention module, using depth information as keys and values, thus allowing the integration of depth information in global feature interactions; three attention modules are used within a single decoder layer; Figure 2 (c) MonoPSTR cleverly uses dynamically updated reference points to provide location queries (x,y) as part of the query structure;

[0031] Figure 3 An architecture diagram of MonoPSTR provided for embodiments of this application;

[0032] Figure 4 An architecture diagram of a dynamic position and scale-aware decoder comprising multiple decoder layers provided for embodiments of this application;

[0033] Figure 5 The structural framework diagram of the depth-guided monocular 3D target detection system provided in the embodiments of this application is shown.

[0034] In the diagram, 1 is the data preprocessing module; 2 is the feature extraction module; 3 is the feature encoding module; 4 is the feature decoding module; and 5 is the 3D attribute extraction module. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0037] This application provides a depth-guided monocular 3D target detection method, including the following steps:

[0038] Step 1: Data preprocessing. Select a 3D object detection benchmark dataset, which includes vehicle images and pedestrian / bicycle images. Preprocess the selected dataset by dividing it into a training set, a validation set, and a test set.

[0039] Step 2: Feature extraction. Input the images from the preprocessed dataset into the ResNet-50 backbone network to obtain the multi-scale visual features and depth features corresponding to the images.

[0040] Step 3: Feature Encoding. A visual encoder and a depth predictor are used to globally encode the obtained visual and depth features, respectively, to generate a visual representation of the image. And foreground depth map D fg ;

[0041] Step 4: Feature Decoding, using a dynamic position and scale-aware decoder with multiple decoder layers to extract the visual representation corresponding to the generated image. And foreground depth map D fg The algorithm adaptively extracts visual and depth features to obtain a query corresponding to an object in the image. Each decoder layer first passes through a depth self-attention layer to remove duplicate predictions, performs interaction between depth embeddings output by the previous depth predictor for depth feature fusion, and then passes through a scale-assisted deformable attention layer to aggregate the embeddings output from the visual encoder to optimize the decoder embeddings, thereby improving the prediction of categories and boxes.

[0042] Step 5: 3D attribute extraction. Multiple detection heads are used to extract 3D attributes from the query obtained after feature decoding.

[0043] In step S1, the 3D object detection benchmark dataset is the KITTI 3D object detection benchmark dataset, which includes 7,481 training images and 7,518 test images. In order to mitigate the possibility of unlabeled images in the test set, the training data is divided into a sub-training set and a validation set, which contain 3,712 and 3,769 sample images, respectively.

[0044] See Figure 1 and 2 As shown in (a)-(c), extensive experiments on the KITTI 3D object detection benchmark dataset demonstrate that the proposed method converges approximately 1.95 times faster than MonoDETR (see [reference]). Figure 1 The MonoPSTR proposed in this application converges within 100 epochs, while the MonoDETR converges within 195 epochs, while maintaining competitive performance. To the best of our knowledge, this is the first work dedicated to improving the training and testing speed of Transformer-based monocular 3D object detectors, which greatly enhances the usability and scalability of such detectors in practical applications.

[0045] In step S2, the multi-scale features corresponding to the image are obtained from the last three stages of the ResNet50 backbone network. These are the initial resolutions. After unifying the resolution, element-wise addition and merging are performed, and finally the highest-level feature is used. As a visual feature f V Based on these multi-scale visual features f V The depth features of the image are obtained by using a two-layer convolutional network.

[0046] In step S3, each visual encoder consists of a global self-attention layer and a feedforward network. Through the global self-attention layer, the depth predictor can learn different foreground regions rather than local spatial information. The depth predictor uses a convolutional network to predict a foreground depth map for the depth features, and only predicts the depth values ​​of foreground objects, while setting the pixel values ​​of the background to 0.

[0047] After obtaining the visual and depth features of the input image, the visual encoder and depth predictor of this application adopt a parallel structure design to globally encode the two types of features respectively. The visual encoder and depth predictor use a self-attention-based transformer structure to interact with the long-distance semantic information within the two types of features. Specifically, for the visual encoder, three layers of blocks are stacked to parse richer RGB representation information; for the depth predictor, only one layer of block is used to understand the depth information of a single channel.

[0048] The visual encoder and depth predictor can operate in parallel without interfering with each other, each focusing on encoding the 2D visual information and the 3D depth information of the image, respectively.

[0049] Combination Figure 2 As shown in (a)-(c), the method proposed in this application employs attention configuration, eliminating the need for additional data. Furthermore, it specifically utilizes the existing four-dimensional information of the reference point to enhance the two attention modules, thereby gaining a deeper understanding of the construction of queries, keys, and values ​​within the attention mechanism. Finally, the dynamic and layer-by-layer updating of the reference point in this application enables the proposed DPS-Query and SDA modules to facilitate adaptive dynamic learning. Therefore, this method exhibits significant improvements in both the quality and efficiency of feature aggregation capabilities.

[0050] In step S4, it should be noted that after the global encoding of visual and depth features, the candidate objects for detection are defined as a set of queries by referring to DETR. This allows the object queries to preferentially integrate global depth features and use this depth information as a guide to help with subsequent feature extraction.

[0051] Furthermore, Figure 3 The process described involves multiple stages. First, an initial feature map is generated using a feature extraction network. Then, visual representations are generated using a visual encoder and a depth predictor, respectively. And foreground depth map D fg To achieve fast and robust detection of 3D objects, this application designs a dynamic position and scale-aware decoder (DPS-Decoder) comprising multiple decoder layers, such as... Figure 4 As shown, each decoder layer consists of two main layers: (1) a deep self-attention layer for removing duplicate predictions, performing interactions between deep embeddings output from previous depth predictors for deep feature fusion; and (2) a scale-assisted deformable attention layer that aggregates embeddings from the visual encoder output to optimize decoder embeddings, thereby improving category and bounding box predictions. Multiple decoder layers are used to guide each object in adaptive extraction of visual and depth features. These decoder layers require queries, keys, and values ​​for attention-based value aggregation, but their inputs differ.

[0052] The DPS-Decoder in this application contains a set of dynamically and explicitly location-encoded queries, which facilitates continuous updating and precise query localization across all subsequent layers. Furthermore, a scale-assisted deformable attention module is introduced after multi-head self-attention, enabling queries to search a more accurate field of view, thereby forming high-quality query features. Detailed descriptions of the dynamically and explicitly location-encoded queries, the scale-assisted deformable attention module, and the detection head loss are provided in the following sections.

[0053] In step S5, the query obtained after passing through all decoders is input into a detection head composed of a series of MLPs for 3D attribute estimation. After deriving and integrating the attributes, no post-processing operations such as NMS are required, and the 3D bounding box is directly used as the system output.

[0054] In one specific embodiment of this application, the training process of the above method is performed on an RTX A6000 GPU using the Adam optimizer over 100 epochs, with a batch size of 16 and a learning rate of 2×10⁻⁶. -4 The weight decays to 10. -4 .

[0055] See also Figure 5 As shown in the embodiments of this application, a depth-guided monocular 3D target detection system for performing the method is also provided, including a data preprocessing module 1, a feature extraction module 2, a feature encoding module 3, a feature decoding module 4, and a 3D attribute extraction module 5.

[0056] The data preprocessing module 1 is used to select a 3D object detection benchmark dataset, which includes vehicle images and pedestrian / bicycle images. The selected dataset is preprocessed and divided into a training set, a validation set, and a test set.

[0057] The feature extraction module 2 is used to input the images in the preprocessed dataset into the ResNet-50 backbone network to obtain the multi-scale visual features and depth features corresponding to the images.

[0058] The feature encoding module 3 is used to globally encode the obtained visual features and depth features using a visual encoder and a depth predictor, respectively, to generate a visual representation of the image. And foreground depth map D fg .

[0059] The feature decoding module 4 is used to extract the visual representation corresponding to the generated image from a dynamic position and scale-aware decoder containing multiple decoder layers. And foreground depth map D fg The algorithm adaptively extracts visual and depth features to obtain a query corresponding to an object in the image. Each decoder layer first passes through a depth self-attention layer to remove duplicate predictions, performs interaction between depth embeddings output by the previous depth predictor for depth feature fusion, and then passes through a scale-assisted deformable attention layer to aggregate the embeddings output from the visual encoder to optimize the decoder embeddings, thereby improving the prediction of categories and bounding boxes.

[0060] The 3D attribute extraction module 5 is used to extract 3D attributes from the query obtained after feature decoding using multiple detection heads.

[0061] The beneficial effects of the embodiments of this application are as follows:

[0062] (1) For the automobile category, the method proposed in this application outperforms several existing monocular 3D target detection methods on the official test set, showing the most efficient performance. See Table 1 for details.

[0063] Table 1

[0064]

[0065] As shown in Table 1, compared with MonoDTR, which also uses the Transformer framework, MonoDSTR performs better in AP at easy, medium, and hard levels. 3D|R40 It showed advantages of +4.16%, +1.01%, and +0.72% in AP. BEV|R40 It showed advantages of +6.2%, +2.5%, and +2.26%. In terms of performance, this method also has a significant advantage over MonoDETR, except in medium and hard AP levels. 3D|R40 The performance is slightly lower under certain conditions, but it is worth noting that this method's training convergence speed is 2.5 times faster, while the testing speed is 4 milliseconds faster.

[0066] Analysis shows that, compared with AP 3D In comparison, this method has advantages in AP BEV Significant performance improvements were observed in this area. This phenomenon can be attributed to AP. BEV It primarily measures the accuracy of vehicle positioning relative to the road, which largely depends on fine-tuning of the estimated query points. Conversely, AP... 3D The distance between the object's center and the ground was evaluated, and its impact on the accuracy of the query point was relatively small. Overall, this method exhibits the most efficient performance among methods that do not require additional data, and it also demonstrates competitive performance compared to methods that do require additional data, achieving the highest or second-highest accuracy.

[0067] (2) For the categories of “pedestrian” and “bicycle”, see Table 2.

[0068] Table 2

[0069]

[0070] In Table 2, “v.√” and “v.×” indicate improved accuracy compared to methods with / without additional data.

[0071] As shown in Table 2, these two categories present greater challenges compared to the "Automobile" category, primarily due to their smaller size and non-rigid body nature. Thanks to the efficient DPS decoder, our method outperforms all previous methods across all three difficulty levels, regardless of whether they require additional data. The results in Table 2 validate the superior generality and scalability of our model, which relies solely on readily available prior knowledge of different scale categories. Therefore, the proposed method easily achieves accurate detection of objects with diverse appearances.

[0072] In summary, experiments on the KITTI dataset demonstrate that our method successfully balances training convergence speed, inference time, and detection accuracy, indicating its high applicability in real-world scenarios.

[0073] (3) By introducing Dynamic and Explicit Location Encoding Query (DEP-query) and Scale-Assisted Deformation Attention (SDA) modules, this application makes full use of valuable location and size priors, helping the query to accurately locate the object in fewer training cycles and generate higher quality query features;

[0074] (4) The method proposed in this application successfully balances training convergence speed, inference time and detection accuracy, indicating that it has high applicability in real-world scenarios.

[0075] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0076] Furthermore, it should be noted that the scope of the methods and systems in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions in a substantially simultaneous manner or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0077] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A depth-guided monocular 3D target detection method, characterized in that, Includes the following steps: Step 1: Data preprocessing. Select a 3D object detection benchmark dataset, which includes vehicle images and pedestrian / bicycle images. Preprocess the selected dataset by dividing it into a training set, a validation set, and a test set. Step 2: Feature extraction. Input the images from the preprocessed dataset into the ResNet-50 backbone network to obtain the multi-scale visual features and depth features corresponding to the images. Step 3: Feature Encoding. A visual encoder and a depth predictor are used to globally encode the obtained visual and depth features, respectively, to generate a visual representation of the image. And foreground depth map D fg ; Step 4: Feature Decoding, using a dynamic position and scale-aware decoder with multiple decoder layers to extract the visual representation corresponding to the generated image. And foreground depth map D fg The algorithm adaptively extracts visual and depth features to obtain a query corresponding to an object in the image. Each decoder layer first passes through a depth self-attention layer to remove duplicate predictions, performs interaction between depth embeddings output by the previous depth predictor for depth feature fusion, and then passes through a scale-assisted deformable attention layer to aggregate the embeddings output from the visual encoder to optimize the decoder embeddings, thereby improving the prediction of categories and boxes. Step 5: 3D attribute extraction. Multiple detection heads are used to extract 3D attributes from the query obtained after feature decoding.

2. The method according to claim 1, characterized in that, In step S1, the 3D object detection benchmark dataset is the KITTI3D object detection benchmark dataset, which includes 7,481 training images and 7,518 test images. The sub-training set and the validation set contain 3,712 and 3,769 sample images, respectively.

3. The method according to claim 2, characterized in that, In step S2, the multi-scale features corresponding to the image are obtained from the last three stages of the ResNet50 backbone network. These are the initial resolutions. After unifying the resolution, element-wise addition and merging are performed, and finally the highest-level feature is used. As a visual feature f V Based on these multi-scale visual features f V The depth features of the image are obtained by using a two-layer convolutional network.

4. The method according to claim 3, characterized in that, In step S3, each visual encoder consists of a global self-attention layer and a feedforward network. Through the global self-attention layer, the depth predictor can learn different foreground regions rather than local spatial information. The depth predictor uses a convolutional network to predict a foreground depth map for the depth features, and only predicts the depth values ​​of foreground objects, while setting the pixel values ​​of the background to 0.

5. The method according to claim 4, characterized in that, In step S3, the visual encoder and depth predictor adopt a parallel structure design and use a self-attention-based transformer structure to interact with long-distance semantic information within the two features. Specifically, for the visual encoder, three layers of blocks are stacked to parse richer RGB representation information; for the depth predictor, only one layer of block is used to understand the depth information of a single channel.

6. The method according to claim 1, characterized in that, The training process described above was performed on an RTX A6000 GPU using the Adam optimizer over 100 epochs, with a batch size of 16 and a learning rate of 2×10⁻⁶. -4 The weight decays to 10. -4 .

7. A depth-guided monocular 3D target detection system for performing the method according to any one of claims 1-6, characterized in that, include: The data preprocessing module is used to select a 3D object detection benchmark dataset, which includes vehicle images and pedestrian and bicycle images. The selected dataset is preprocessed and divided into a training set, a validation set, and a test set. The feature extraction module is used to input the images in the preprocessed dataset into the ResNet-50 backbone network to obtain the multi-scale visual features and depth features corresponding to the images. The feature encoding module is used to globally encode the obtained visual and depth features using a visual encoder and a depth predictor, respectively, to generate a visual representation of the image. And foreground depth map D fg ; The feature decoding module is used to extract the visual representation corresponding to the generated image from a dynamic position and scale-aware decoder containing multiple decoder layers. And foreground depth map D fg The algorithm adaptively extracts visual and depth features to obtain a query corresponding to an object in the image. Each decoder layer first passes through a depth self-attention layer to remove duplicate predictions, performs interaction between depth embeddings output by the previous depth predictor for depth feature fusion, and then passes through a scale-assisted deformable attention layer to aggregate the embeddings output from the visual encoder to optimize the decoder embeddings, thereby improving the prediction of categories and boxes. as well as The 3D attribute extraction module is used to extract 3D attributes from a query obtained after feature decoding using multiple detection heads.

Images