3D object detection method based on pseudo-monocular for automatic driving

Abstract

The application discloses a 3D target detection method based on pseudo-single purpose for automatic driving, and has the characteristics that the method comprises the following steps: (1) obtaining a stereo image pair; (2) calculating a disparity map and estimating a depth map based on a left image; (3) generating a visual feature; (4) generating a depth feature; (5) encoding and decoding of a transformer; (6) guiding query initialization; (7) object query enhanced by an encoder and a decoder is input into a multi-layer perceptron and a linear projection layer to generate a final prediction. The application guarantees the depth prediction accuracy without additional data input and reduces the calculation cost.

Description

Technical Field

[0001] This invention relates to navigation technology, and more specifically to a target detection method for autonomous driving, particularly a pseudo-monocular 3D target detection method. Background Technology

[0002] The goal of 3D object detection is to locate and identify targets in three-dimensional space. 3D object detection methods using LiDAR and stereo matching can achieve impressive performance. In the current field of autonomous driving, 3D object detection has become an indispensable module. Considering cost factors, using monocular cameras for 3D object detection is of great significance for autonomous driving. Monocular 3D object detection using only a single image, without accurate 3D information, is a challenging task. Pure image monocular 3D object detection methods using only geometric constraints between 2D and 3D have made great progress in recent years. However, due to the lack of accurate depth information, the performance of pure image monocular 3D object detection methods remains limited. To address this issue, recent methods generate pseudo-LiDAR point clouds using pre-trained depth estimators to assist in depth prediction. Experimental results show that accurate depth prediction can improve the 3D detection performance of monocular vision. However, due to the unavoidable errors in the generated Pseudo-LiDAR, the detection performance is still not satisfactory.

[0003] Nicolas et al This paper proposes applying an encoder-decoder transform architecture to the field of monocular 3D object detection (DETR). However, compared with mainstream monocular 3D detectors, DETR has a long training time and cannot detect small targets. The object query and its reference point represent the target's features and location, respectively. However, in DETR, the object query is defined through random initialization and fed into the decoder for deep interaction with the encoded features of the image. This random initialization requires refinement using six decoder layers with cross-attention modules. Therefore, the randomly initialized object query significantly increases the computational cost and prolongs the training time.

[0004] It is worth noting that the depth prediction error generated from stereo images is much smaller than that generated from a single image. Introducing stereo images into transformer-based monocular 3D object detection while reducing computational cost is a promising area of ​​research. Summary of the Invention

[0005] The purpose of this invention is to provide a pseudo-monoscopic 3D target detection method for autonomous driving. By introducing stereo images and combining them with a transformer decoder method, the effectiveness of 3D target detection is improved and the computational cost is reduced.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is: a pseudo-monocular 3D target detection method for autonomous driving, comprising: (1) Obtain stereo image pairs; (2) Calculate the disparity map based on the stereo image and estimate the depth map using the left image as the main body; (3) Visual feature generation: Using the left-side image as the main body, the right-related pixels of the left-foreground pixels are found through depth indexing and foreground probability map, and feature-enhanced visual features are generated. (4) Deep feature generation: The foreground pixels of the left image are projected onto a multi-scale feature map and their feature matching probability with adjacent features is calculated to generate multi-scale depth features. (5) Encoding and decoding of the converter: The transformer consists of a visual encoder, a depth encoder, and a position-aware decoder. The visual encoder and depth encoder generate visual embeddings and deep embeddings, respectively. Each encoder block consists of a self-attention layer and a feedforward neural network. The position-aware decoder uses sparse anchor points as reference points and takes the visually encoded features of the anchor point positions as the input to the decoder. (6) Guided query initialization: Sparse anchor points are used as reference points to assist in initializing object queries, and encoded visual features are used as object queries. By deeply interacting with encoded visual features and encoded depth features, each object can obtain global information and perceive the relationships between objects. (7) The object query enhanced by the encoder and decoder is fed into the multilayer perceptron and linear projection layer to generate the final prediction.

[0007] In step (1), stereo image pairs (I) are acquired from the stereo camera. L ,I R ), where I L The image on the left, I R For the image on the right, extract the features from the left side. and right-side features A feature pyramid network is used to extract multi-scale feature maps of stereo image pairs. Their downsampling rates are respectively .

[0008] In step (2), the stereoscopic image is used to analyze ( I L , I R A pair of related pixels in ) xl and x r Calculate the corresponding depth Z , (1.1) in, f and b These are camera calibration parameters, representing the camera focal length and stereo camera baseline, respectively.

[0009] In step (3), the method for generating visual features is to apply a 1×1 convolutional layer and an S-shaped growth curve function to the left and right multi-scale feature maps of the input, respectively. F s Generate a probability map of the same size as the feature map. P In order to obtain an objective score, (1.2) in, Corresponding to the input feature map F S coordinates on s This represents the stride of the input feature map. This represents the probability that the center of the foreground object is located at the input coordinates. Restored the current coordinates in the input image I Pixel position in; For feature maps of different scales A predefined set of probability thresholds represents the probability of left and right feature map pairs. P Meanwhile, areas below the threshold are considered background regions, and features from the left and right images are linked to enhance visual features. The depth information of the ground truth is transformed through linearly increasing discretization. k +1 depth category, where, the first k The first depth category represents the depth where foreground objects exist, and the last depth category represents the background region. The predicted depths are assigned to the corresponding depth categories and then indexed using depth indexing. express; Based on the predicted probability map and depth map, foreground pixels are searched in the left input image and related pixels are searched in the right input image, respectively. Foreground pixels are selected only in the left input image using the predicted multi-scale foreground probability map. Its coordinates are and foreground features Then, these left foreground pixels are indexed by depth. Find the right associated coordinates and its characteristics , (1.3) in, f Indicates the camera's focal length. b Indicates the baseline of the stereo camera. s Indicates the stride of the feature map. d This represents the depth corresponding to the depth index; For left foreground pixel coordinates where right-related pixels cannot be found, and pixels identified as background regions, no further processing is performed; only left foreground features are considered. Then, the visual features are aggregated from the left and right feature maps element by element. The calculation is as follows: (1.4).

[0010] In step (4), the method for generating depth features is to select foreground pixels. As a multi-scale left feature map The key points are then analyzed, and the distances between each key point and its neighboring feature points are calculated. By setting 3 different distance thresholds Foreground pixels can perform feature queries on feature maps at three different scales; On each feature map, any foreground pixel can obtain a set of neighboring foreground features. For each neighboring feature in the adjacent foreground feature set, its relationship with the foreground pixel feature is calculated. The feature matching probability, (1.5) For each multi-scale feature map, the foreground pixel whose features are closest to the calculated feature matching probability is selected. k 1 feature point, obtained by aggregation k Features are used to generate foreground pixels. Features at this scale , (1.6) in, M (·) represents a multilayer perceptron network; max (·) indicates a maximum pooling operation along the channel.

[0011] Finally, the features of the foreground pixels at three scales are aggregated to generate the depth features of the input image. .

[0012] In step (6), sparse anchor points are selected as reference points for object queries. For each original feature map, in the anchor prediction branch, a 1×1 convolutional layer and an S-curve growth function are applied to each input feature map to compute a probability map. P The left and right input images are the same size, and there are corresponding left and right feature maps at each scale. A set of anchor points can be obtained from the multi-scale feature maps of the left and right input images respectively. and These anchor points can be used to form an anchor bag. , The anchor box prediction branch will use the given feature map and the probability map output by the anchor point prediction branch. P Predict these anchor points with probabilities higher than a threshold The best shape That is, the shape has the highest coverage with the nearest ground truth bounding box; Foreground objects present in the image Calculate its ground truth frame and anchor frame. The intersection-union ratio (IoU), if Then it is considered an anchor Foreground object But if the anchor If the IoU with multiple objects is greater than 0.5, the object with the highest IoU is selected for matching. Thus, the anchor packet... Anchor points in the data are divided into multiple sub-anchor packages based on the number of objects. Then, select the IoU before each anchor package. m Using the anchor point as a reference point for proposal initialization, the attention module can focus on the area around the foreground by using the anchor point of the foreground region as a reference point.

[0013] Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art: 1. This invention addresses the problem of poor depth estimator accuracy in existing monocular 3D target detection algorithms by proposing a pseudo-monocular 3D target detection algorithm that utilizes stereo image pairs. It calculates the disparity map from the stereo image to infer the depth map and generates a feature-enhanced monocular image from the stereo image, ensuring depth prediction accuracy without additional data input.

[0014] 2. In order to further enrich the fine-grained information in the monocular 3D target detection algorithm, this invention proposes a feature enhancement module based on pixel matching. By calculating the matching probability of adjacent features in multiple scales for each foreground pixel, it matches adjacent feature sets for each foreground pixel, and finally aggregates multi-scale features to generate depth features containing rich fine-grained information.

[0015] 3. To reduce the computational cost caused by random initialization of object queries in the converter decoder, this invention proposes a query initialization module based on sparse anchors. Sparse anchors are generated in the stereo image using a foreground probability map, and then anchor features are obtained from the visual encoding feature map as the initialization object query, thereby reducing computational costs. Attached Figure Description

[0016] Figure 1 This is a method framework diagram of an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the generation of foreground visual features in the embodiment; Figure 3 This is a schematic diagram illustrating the generation of foreground depth features in the embodiment; Figure 4 This is a schematic diagram of the anchor point generation method in the embodiment; Figure 5 These are schematic diagrams of the anchor points in the embodiments; Figure 6 This is a schematic diagram of the pseudo-monocular converter architecture in the embodiment; Figure 7 This is an example of a qualitative result from the KITTI dataset in the embodiment; Figure 8 This is a graph showing the qualitative results of depth loss on the KITTI dataset in the example; Figure 9 This is a comparison chart with the baseline under different object depths and IoU thresholds in the embodiment. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments: Example 1: A pseudo-monocular 3D target detection method for autonomous driving, which achieves 3D target detection by inputting a stereo image. For example... Figure 1 As shown, the system is implemented through a backbone network, a pseudo-monocular enhancement module (PMFE), a guided packet anchor module, a depth-aware transformation module, and a 2D-3D detection head. Stereo images are used as input for end-to-end monocular 3D object detection, and foreground pixels in the input image are filtered using disparity and foreground probability maps. Foreground visual features and foreground depth features are extracted using feature association and feature matching, respectively. Then, object queries are constructed from learnable anchor packets and interact with the encoded depth embeddings through a depth-aware transformer.

[0018] Pseudo-monocular feature enhancement: Given a resolution from a stereo camera stereo image pairs (I L ,I R PMFE can extract features from the left side separately. and right-side features Furthermore, a feature pyramid network is employed to extract multi-scale feature maps from stereo image pairs. Their downsampling rates are respectively Besides the stereo image pairs used to generate the 2D feature map, no additional information is required to generate the predicted depth map.

[0019] During the input stereo image stage, PMFE first uses the SGBM algorithm to obtain the disparity map. D Then, depth maps are estimated using the left-hand image as the primary source. Specifically, depth maps are estimated by pairing stereo images (…). I L , I R A pair of related pixels in ) x l and x r The corresponding depth can be calculated. (1.1) in, f and b These are camera calibration parameters, representing the camera focal length and stereo camera baseline, respectively.

[0020] Unlike traditional methods that use off-the-shelf depth estimators, this method obtains more accurate predicted depth maps through linear computation alone, without introducing additional computational costs. To provide reliable visual and depth embeddings, different methods are used to generate visual and depth features.

[0021] Visual feature generation: such as Figure 2 As shown, a 1×1 convolutional layer and an S-curve growth function are applied to the left and right multi-scale feature maps of the input, respectively. F s Generate a probability map of the same size as the feature map. P In order to obtain an objective score, (1.2) in, Corresponding to the input feature map F S coordinates on s This represents the stride of the input feature map. This represents the probability that the center of the foreground object is located at the input coordinates. Restored the current coordinates in the input image I The pixel position in the image.

[0022] For feature maps of different scales A set of probability thresholds is predefined. It's worth noting that only the probability of left and right feature map pairs is considered. PAt the same time, only areas below a threshold are considered background regions. Then, through the PMFE module, the features of the left and right images can be linked to enhance visual features.

[0023] To find a suitable depth interval, the ground truth depth information is discretized through linearly increasing discretization. k +1 depth category (bin). Among them, the first k The first bin represents the depth where foreground objects exist, and the last bin represents the background region. Then, the predicted depths are assigned to the corresponding depth categories (bins) and indexed using depth indexing. This indicates that, unlike existing methods, the depth predicted by this method comes from stereo image pairs, thus eliminating the need for additional depth supervision.

[0024] Specifically, the PMFE module searches for foreground pixels in the left input image and related pixels in the right input image based on the predicted probability map and depth map, respectively. First, it filters only the left input image using the predicted multi-scale foreground probability map. N One foreground pixel Its coordinates are and foreground features Then, these left foreground pixels are indexed by depth. Find the right associated coordinates and its characteristics , (1.3) in, f Indicates the camera's focal length. b Indicates the baseline of the stereo camera. S Indicates the stride of the feature map. d This indicates the depth corresponding to the depth index.

[0025] For left foreground pixel coordinates where a right-related pixel cannot be found, and pixels identified as background regions, no further processing is performed, and only left foreground features are considered. It is worth noting that in this embodiment, the predicted depth is... Pixels outside the area are also considered background. Then, the visual features are aggregated from the left and right feature maps by element-wise aggregation. The calculation is as follows: (1.4) Depth Feature Generation: Additional depth features need to be generated to meet the requirements of monocular 3D object detection. To retain more fine-grained information, multi-scale features are integrated. Similar to the generation of visual features, background pixels are not considered. For example... Figure 3 As shown, the left input image is projected onto a multi-scale feature map.

[0026] Specifically, select foreground pixels As a multi-scale left feature map The key points are then identified. The distances between each key point and its neighboring feature points are then calculated. By setting three different distance thresholds The foreground pixels can then perform feature queries on feature maps at three different scales.

[0027] On each feature map, any foreground pixel can obtain a set of neighboring foreground features. It is worth noting that the foreground pixels in this method are obtained by jointly filtering a multi-scale foreground probability map and depth estimated by disparity. The multi-scale foreground probability map can effectively identify the foreground region in the multi-scale feature map. Therefore, this method can quickly obtain foreground feature points on the multi-scale feature map.

[0028] For each neighboring feature in the adjacent foreground feature set, its relationship with the foreground pixel feature is calculated. The feature matching probability, (1.5) For each multi-scale feature map, the foreground pixel whose features are closest to the calculated feature matching probability is selected. k Each feature point. To ensure that multi-scale features contain more accurate fine-grained information, it is also necessary to aggregate the obtained features. k Features are used to generate foreground pixels. Features at this scale , (1.6) in, M (·) represents a multilayer perceptron network; max (·) indicates a maximum pooling operation along the channel.

[0029] Finally, the features of the foreground pixels at three scales are aggregated to generate the depth features of the input image. As for the pixels in the background area, no additional processing is performed.

[0030] This method further enhances the foreground features at each scale by aggregating multi-scale features. Moreover, compared to ordinary pixel-by-pixel feature aggregation methods, this method significantly reduces computational cost while preserving accurate fine-grained information.

[0031] Guided query initialization: This method selects sparse anchor points as reference points for object lookup. By setting the reference points in the foreground candidate object region, the quality of the reference points can be effectively improved. Most traditional anchor-based methods require manually designing a set of anchor points with different scales and aspect ratios based on the intersection-over-union (IoU) ratio. A These predefined anchor points severely impact the detector's performance. Therefore, this method designs a sparse anchor module to find reference points.

[0032] Specifically, for each original feature map, use as follows Figure 4 The anchor point generation scheme is shown. In the anchor point prediction branch, a 1×1 convolutional layer and an S-curve growth function are applied to each input feature map to compute the probability map. P .

[0033] like Figure 5 As shown, by matching anchor points to input image locations with probabilities higher than a set threshold, the potential object region can be significantly reduced while maintaining recall. In this method, the left and right input images are of the same size, and there are corresponding left and right feature maps at each scale. Therefore, a set of anchor points can be obtained from the multi-scale feature maps of the left and right input images respectively. and Finally, these anchor points can be combined to form an anchor bag. , The anchor box prediction branch will use the given feature map and the probability map output by the anchor point prediction branch. P Predict these anchor points with probabilities higher than a threshold The best shape This means that the shape has the highest coverage with the nearest ground truth bounding box. Unlike conventional methods, this method uses only one predicted optimal shape anchor box at each location, rather than a set of predefined anchor boxes.

[0034] Foreground objects present in the image It will calculate its ground truth frame and anchor frame. The intersection-union ratio (IoU). If Then it is considered an anchor Foreground object But if the anchor If the IoU with multiple objects is greater than 0.5, then the object with the highest IoU is selected for matching. Therefore, anchor packets... Anchor points in a document can be divided into multiple sub-anchor packages based on the number of objects. Then, select the IoU before each anchor packet. m The anchor point is used as a reference point for proposal initialization. By using the anchor point of the foreground region as a reference point, the attention module can focus on the area around the foreground.

[0035] The object lookup in this method is derived from the feature map of the visual encoder. Unlike conventional methods that use random initialization, this method distributes reference points in a more reasonable manner, thereby effectively improving the accuracy of the non-cascaded structure decoder.

[0036] Encoding and decoding of the converter: like Figure 6 As shown, the transformer used in this embodiment consists of a visual encoder, a depth encoder, and a decoder. The decoder uses a cross-attention module to aggregate relevant visual and depth features of the enhanced object query, thereby truly achieving thorough multi-dimensional information interaction.

[0037] Visual and depth encoders: Visual embeddings and depth embeddings are generated separately using encoder blocks. Each encoder block consists of a self-attention layer and a feed-forward neural network (FFN). Three encoder blocks and one encoder block are used for encoding visual features and depth features, respectively. This embodiment uses traditional sine / cosine position encoding to generate the encoder's position embeddings. Then, the flattened features are supplemented using position encoding and input into the transform encoder.

[0038] Position-Aware Decoder: The decoder in this embodiment still uses the transformer's basic architecture as a standard. However, unlike traditional transformer-based methods, it utilizes sparse anchor points as reference points and uses the visually encoded features of the anchor point positions as input to the decoder. The proposed object query initialization method has two main advantages: (1) Each anchor point is located at the center of the foreground object in the image. Compared to random initialization, this method allows the attention module to focus its attention more effectively. Therefore, it reduces the number of decoder layers required while maintaining high performance.

[0039] (2) During training, the number of suggestions in this method is dynamically adjusted by the number of predicted sparse anchor points. Therefore, this method can adjust the required computation based on the number of foreground objects in the image.

[0040] Specifically, the proposed initialization method ensures that initial object queries are all located near the center of the foreground object, and can significantly reduce computational costs.

[0041] After the encoder and decoder, the enhanced object query is fed into a multilayer perceptron and a linear projection layer to generate the final prediction.

[0042] To evaluate the effectiveness of this embodiment, the following loss function is used: Total loss It mainly consists of four main losses: classification loss for object category prediction. Regression loss used for bounding box prediction ; depth loss to aid in the association of left and right foreground pixels; and prediction loss for sparse anchors. , (1.7) in, , , and These represent the regularization weights for classification loss, regression loss, depth loss, and sparse anchor prediction loss, respectively.

[0043] Specifically, to address the problem of imbalanced positive and negative samples, Focal Loss is used for classification loss. Regarding regression loss A smoothed L1 loss function is used for balancing. Depth loss... It is also defined using focus loss. (1.8) in, This represents the pixel region in the left input image where the foreground probability is higher than a threshold. This represents the actual ground depth value generated by the lidar. This represents the right input image pixel associated with the left foreground pixel.

[0044] Considering that the confidence level of anchor points is relatively low in the early stages of training, the Mean-max function is used instead of the traditional Mean function. The Mean-max function can effectively increase the probability of anchor points matching objects in the anchor packet.

[0045] (1.9) in, Indicates the system's network parameters; Represents the foreground probability of the anchor point; Indicates anchor package The ground truth bounding box of the corresponding object; This represents the predicted anchor box and the actual bounding box. loss; The prediction box represents the anchor point; R represents the probability that the anchor point correctly predicts the object; F represents the probability that the anchor point is misclassified as a positive anchor. Indicates the number of objects; Represents the regularization factor; This represents the intersection between the anchor point and the actual ground bounding box.

[0046] The framework of this embodiment was evaluated on the challenging KITTI 3D and MS-COCO 2017 object detection datasets and the latest large-scale open dataset NuScenes.

[0047] The KITTI dataset, currently the most popular benchmark for 3D object detection, contains 7481 training images and 7518 test images to meet the needs of different object detection requirements. This embodiment divides the training samples into 3712 sub-training sets and 3769 validation sets. In the KITTI dataset, different objects are categorized into three levels—easy, medium, and difficult—based on size, occlusion level, and truncation level. For the three categories of cars, pedestrians, and bicycles, this paper sets the IoU threshold to 0.7, 0.5, and 0.5, respectively.

[0048] The NuScenes dataset is a large-scale autonomous driving dataset that collects 1000 complex driving scenarios from Boston and Singapore. These scenarios are split into 700 training scenarios, 150 validation scenarios, and 150 test scenarios. Data for the NuScenes dataset was acquired using six multi-view cameras and one 32-line LiDAR, and the images from the six views were fused to fully cover the 360-degree surrounding environment. In the evaluation, the main metrics used in this embodiment are Average Precision (AP), Average Translation Error (ATE), Average Scale Error (ASE), Average Orientation Error (AOE), Average Velocity Error (AVE), Average Attribute Error (AAE), and the nuScenes detection score (NDS).

[0049] The MS-COCO 2017 dataset contains 118K training images, 5K validation images, and 41K images in the test-dev set. Each image in the dataset contains multiple instances. Standard COCO metrics were used for evaluation, including mean accuracy (AP) and AP at IoU = 0.5. 50 IoU = 0.75 (AP) 75 ) and AP for objects of different sizes S AP M and AP L .

[0050] ResNet-50 is used as the backbone network in this embodiment. Transformer blocks are provided for the visual encoder, depth encoder, and position-aware decoder, with 3, 1, and 1 blocks respectively. For the predicted foreground object region, the minimum and maximum depths are set to 0 and 80m respectively, and the number of bins is... k The maximum number of objects is 50. For the KITTI dataset, the input images are retained at their original resolution of 1280×384 for both training and inference. For the NuScenes dataset, the input images are downsampled to 960×640 to save computation time. The model in this embodiment was trained for 36 epochs on the MS-COCO dataset.

[0051] All models in this embodiment are trained using the AdamW optimizer, with the initial learning rate initialized to 1e. -4 Then, the learning rate is gradually updated using cosine annealing. The network is trained using two NVIDIA RTX 3090 GPUs. For the regularized weights of the loss function, , , and .

[0052] Experimental results on the KITTI dataset First, the proposed method was compared with several state-of-the-art monocular 3D object detection methods on the KITTI dataset. As shown in Tables 1 and 2, the proposed method achieved better performance than other methods on medium-level tasks with IoU thresholds of 0.7 and 0.5. Table 1 shows the performance comparison of the proposed method on the car category of the KITTI test set. Notably, the proposed method not only outperforms methods without data input but also significantly outperforms other depth-assisted methods. Compared with representative methods such as CaDDN and DDMP-3D, the AP of this embodiment... 3D The efficiency was improved by 8.24 / 5.16 / 4.7 and 7.7 / 5.79 / 6.36 respectively, AP BEV The efficiency was improved by 7.9 / 4.76 / 3.5 and 7.76 / 5.78 / 7.25, respectively. Compared with the latest method MonoDETR, this method achieves higher AP. 3D and AP BEV The efficiency increases are 3.76 / 3.76, 2.65 / 2.23, and 3.17 / 2.84, respectively.

[0053] Table 1. Detection performance of the car category in the KITTI test set at an IoU threshold of 0.7. .

[0054] Table 2 further illustrates the performance comparison of our proposed method on the KITTI test set for pedestrian and bicycle categories. Since the models for these two categories are smaller than those for cars, they are more difficult to locate during detection. Overall, our proposed method significantly outperforms other methods in both categories. This demonstrates that the detection framework proposed in this embodiment can effectively improve model performance by fully utilizing the complementary information from the left and right images for 3D object detection.

[0055] Table 2. Detection performance of pedestrian and bicycle categories in the KITTI test set at an IoU threshold of 0.5. .

[0056] To further validate the performance of the proposed method, experiments were conducted on the KITTI validation set with an IoU threshold of 0.7. As shown in Table 3, the proposed method still outperforms image-only methods on the set with better data performance. Specifically, compared to the latest MonoDETR, the proposed method achieves better performance on AP. 3D and AP BEV The above achieved improvements of 4.74 / 2.78, 3.58 / 2.69, and 3.76 / 2.22 respectively.

[0057] Table 3 shows the detection performance of the car category in the KITTI validation set at an IoU threshold of 0.7. .

[0058] Figure 7 This example demonstrates qualitative results from this embodiment on the KITTI dataset.

[0059] Experimental results on the NuScenes dataset The proposed method was experimentally tested on the more challenging NuScenes dataset. Further experiments on different datasets can further validate the effectiveness and generalization of the method.

[0060] Table 4 compares state-of-the-art object detection methods using RGB images as input. mAP and NDS are the most important official evaluation metrics for the NuScenes dataset. Our proposed method achieves significantly better performance than other methods, with mAP of 40.3 and NDS of 46.1. Compared to the previous best method, this embodiment achieves a 1.7% improvement in mAP. The improvement in NDS is mainly due to a significant reduction in mAVE. This is because, compared to methods using a single image as input, our method uses stereo images as input, which allows us to obtain effective velocity information from consecutive frames, thus greatly reducing mAVE. Therefore, although our method lags behind methods using LiDAR data as input in some aspects, it still maintains high performance in both mAP and NDS.

[0061] Table 4 Performance comparison on the NuScenes dataset .

[0062] Experimental results on the MS-COCO dataset To fully demonstrate the impact of the number of decoders on model performance, this paper additionally uses the MS-COCO dataset for evaluation. Compared to the DETR method, our method maintains the same fast convergence characteristic as Efficient DETR. Compared to the state-of-the-art Efficient DETR, this embodiment still maintains a significant performance lead. Specifically, AP is improved by 1.1, and AP50 and AP75 are improved by 2.2 and 0.8, respectively. Table 5 demonstrates that stacking multiple decoders does indeed enhance model performance. However, considering the computational cost, this paper still chooses to use a single decoder.

[0063] Table 5 shows the performance comparison on the MS-COCO 2017 validation set. (1-d) represents the model built in a 1-decoder structure. DETR-R50 500 86 41 42.0 62.4 44.2 20.5 45.8 61.1 DETR-DC5-R50 500 187 41 43.3 63.1 45.9 22.5 47.3 61.1 Deformable DETR-R50 50 173 40 43.8 62.6 47.7 26.4 47.1 58.0 Efficient DETR-R50 36 159 32 44.2 62.2 48.0 28.4 47.5 56.6 Efficient DETR(1-d)-R50 36 210 35 45.1 63.1 49.1 28.3 48.4 59.0 This embodiment - R50 36 148 34 45.3 64.4 48.8 27.9 48.6 62.1 This embodiment (1-d)-R50 36 232 37 45.8 65.2 49.6 28.1 48.4 64.7 ablation experiment To fully demonstrate the effectiveness of the proposed module, extensive ablation studies were conducted on the KITTI dataset. All models were evaluated on the car category of the KITTI validation set.

[0064] PMFE Module: This embodiment proposes a feature enhancement module, PMFE, that uses stereo images as input and integrates foreground probability estimation and multi-scale feature fusion. To illustrate the effectiveness of the proposed module, a comparison with the powerful MonoDETR is chosen. This embodiment directly adds several MLP-based prediction heads to the DETR model to construct a baseline paradigm, as shown in the first row of Table 6. MonoDETR achieves better performance than the baseline without depth data input through deformable attention and a depth-guided improvement scheme. Unlike MonoDETR, which uses deformable attention for multi-scale feature encoding, this method selects foreground pixels through a foreground probability map and aggregates multi-scale features through multi-scale feature matching probabilities. Therefore, this method can retain more fine-grained information without additional data input.

[0065] Table 6 Effectiveness of PMFE Modules .

[0066] It is worth noting that stereo images were chosen as input because a reliable depth estimate was desired without additional data. Figure 8 As shown, the loss curve in this embodiment is more convex, and the depth corresponding to the minimum loss is closer to the ground truth. Therefore, the loss of this method is more robust than the geometric error used in a single image. Furthermore... Figure 8 (b) in the figure illustrates a failure case. Due to occlusion and low visibility, the geometric and semantic cues of depth recovery estimated from a single image cannot be guaranteed to be accurate, so the constraints based on a single image fail to converge at this location.

[0067] Guided query initialization: Based on the IoU calculated with the ground truth, this embodiment assigns the predicted sparse anchors to the foreground objects with the highest IoU. Then, these anchors are sorted by their IoU with the corresponding ground truth objects. k The proposal assigns positive samples and uses them as reference points for object query initialization. Table 7 shows the performance impact of assigning multiple positive samples to a foreground object. As shown in Table 7, with... k As the number of assignments increases, model performance improves significantly. However, when the number of assignments increases... k When the difficulty level reached 15, the model performance actually decreased by 0.46 / 0.13 / 0.3 at different difficulty levels. Although increasing the number of suggestions during model evaluation can significantly improve model performance, excessive allocation not only increases computational costs but also leads to repeated predictions of foreground objects.

[0068] Table 7. Impact of different allocations on performance .

[0069] Number of encoders and decoders: Since the reference points for object query initialization in this embodiment are concentrated near the center of the foreground object, a simpler architecture is sufficient. As shown in Table 8, unlike the cascaded structure of 6 encoder blocks and 6 decoder blocks in DETR, this method achieves the best performance when stacking 3 encoder blocks. However, when stacking decoder blocks, the model performance does not significantly improve with the increase in the number of decoder blocks. This further indicates that using a single decoder block is a better choice in this embodiment. Cascading multiple decoder layers slightly improves model performance but also introduces a significant amount of computation.

[0070] Table 8 Converter Settings .

[0071] The effectiveness of each module proposed in this method is analyzed in Table 9, which examines the impact of the proposed components on the overall model performance. A method using only context-aware 3D object detection is used as the baseline. With an IoU threshold of 0.7, this method, through the assistance of the proposed components, achieves significant improvements at all difficulty levels compared to the baseline.

[0072] Table 9 Analysis of different modules in the proposed method .

[0073] like Figure 9 As shown, to further demonstrate the effectiveness of the module, the method was evaluated under different IoU thresholds and object depths, and compared with baseline performance.

Claims

1. A pseudo-monocular 3D target detection method for autonomous driving, characterized in that, include: (1) Obtain stereo image pairs; (2) Calculate the disparity map based on the stereo image and estimate the depth map using the left image as the main body; (3) Visual feature generation: Using the left-side image as the main body, the right-related pixels of the left-foreground pixels are found through depth indexing and foreground probability map, and feature-enhanced visual features are generated. (4) Deep feature generation: The foreground pixels of the left image are projected onto a multi-scale feature map and their feature matching probability with adjacent features is calculated to generate multi-scale depth features. (5) Encoding and decoding of the converter: The transformer consists of a visual encoder, a depth encoder, and a position-aware decoder. The visual encoder and depth encoder generate visual embeddings and deep embeddings, respectively. Each encoder block consists of a self-attention layer and a feedforward neural network. The position-aware decoder uses sparse anchor points as reference points and takes the visually encoded features of the anchor point positions as the input to the decoder. (6) Guided query initialization: Sparse anchor points are used as reference points to assist in initializing object queries, and encoded visual features are used as object queries. By deeply interacting with encoded visual features and encoded depth features, each object can obtain global information and perceive the relationships between objects. (7) The object query enhanced by the encoder and decoder is fed into the multilayer perceptron and linear projection layer to generate the final prediction.

2. The pseudo-monocular 3D target detection method for autonomous driving according to claim 1, characterized in that: In step (1), stereo image pairs (I) are acquired from the stereo camera. L ,I R ), where I L The image on the left, I R For the image on the right, extract the features from the left side. and right-side features A feature pyramid network is used to extract multi-scale feature maps of stereo image pairs. Their downsampling rates are respectively .

3. The pseudo-monocular 3D target detection method for autonomous driving according to claim 1, characterized in that: In step (2), the stereoscopic image is used to analyze ( I L , I R A pair of related pixels in ) x l and x r Calculate the corresponding depth Z , (1.1) in, f and b These are camera calibration parameters, representing the camera focal length and stereo camera baseline, respectively.

4. The pseudo-monocular 3D target detection method for autonomous driving according to claim 1, characterized in that: In step (3), the method for generating visual features is to apply a 1×1 convolutional layer and an S-shaped growth curve function to the left and right multi-scale feature maps of the input, respectively. F s Generate a probability map of the same size as the feature map. P In order to obtain an objective score, (1.2) in, Corresponding to the input feature map F S coordinates on s This represents the stride of the input feature map. This represents the probability that the center of the foreground object is located at the input coordinates. Restored the current coordinates in the input image I Pixel position in; For feature maps of different scales A predefined set of probability thresholds represents the probability of left and right feature map pairs. P Meanwhile, areas below the threshold are considered background regions, and features from the left and right images are linked to enhance visual features. The depth information of the ground truth is transformed through linearly increasing discretization. k +1 depth category, where, the first k The first depth category represents the depth where foreground objects exist, and the last depth category represents the background region. The predicted depths are assigned to the corresponding depth categories and then indexed using depth indexing. express; Based on the predicted probability map and depth map, foreground pixels are searched in the left input image and related pixels are searched in the right input image, respectively. Foreground pixels are selected only in the left input image using the predicted multi-scale foreground probability map. Its coordinates are and foreground features Then, these left foreground pixels are indexed by depth. Find the right associated coordinates and its characteristics , (1.3) in, f Indicates the camera's focal length. b Indicates the baseline of the stereo camera. s Indicates the stride of the feature map. d This represents the depth corresponding to the depth index; For left foreground pixel coordinates where right-related pixels cannot be found, and pixels identified as background regions, no further processing is performed; only left foreground features are considered. Then, the visual features are aggregated from the left and right feature maps element by element. The calculation is as follows: (1.4)。 5. The pseudo-monocular 3D target detection method for autonomous driving according to claim 4, characterized in that: In step (4), the method for generating depth features is to select foreground pixels. As a multi-scale left feature map The key points are then analyzed, and the distances between each key point and its neighboring feature points are calculated. By setting 3 different distance thresholds Foreground pixels can perform feature queries on feature maps at three different scales; On each feature map, any foreground pixel can obtain a set of neighboring foreground features. For each neighboring feature in the adjacent foreground feature set, its relationship with the foreground pixel feature is calculated. The feature matching probability, (1.5) For each multi-scale feature map, the foreground pixel whose features are closest to the calculated feature matching probability is selected. k 1 feature point, obtained by aggregation k Features are used to generate foreground pixels. Features at this scale , (1.6) in, M (·) represents a multilayer perceptron network; max (·) indicates a maximum pooling operation along the channel.

6. Finally, the features of the foreground pixels at the three scales are aggregated to generate the depth features of the input image. .

7. The pseudo-monocular 3D target detection method for autonomous driving according to claim 1, characterized in that: In step (6), sparse anchor points are selected as reference points for object queries. For each original feature map, in the anchor prediction branch, a 1×1 convolutional layer and an S-curve growth function are applied to each input feature map to compute a probability map. P The left and right input images are the same size, and there are corresponding left and right feature maps at each scale. A set of anchor points can be obtained from the multi-scale feature maps of the left and right input images respectively. and These anchor points can be used to form an anchor bag. , The anchor box prediction branch will use the given feature map and the probability map output by the anchor point prediction branch. P Predict these anchor points with probabilities higher than a threshold The best shape That is, the shape has the highest coverage with the nearest ground truth bounding box; Foreground objects present in the image Calculate its ground truth frame and anchor frame. The intersection-union ratio (IoU), if Then it is considered an anchor Foreground object But if the anchor If the IoU with multiple objects is greater than 0.5, the object with the highest IoU is selected for matching. Thus, the anchor packet... Anchor points in the data are divided into multiple sub-anchor packages based on the number of objects. Then, select the IoU before each anchor package. m Using the anchor point as a reference point for proposal initialization, the attention module can focus on the area around the foreground by using the anchor point of the foreground region as a reference point.

Images