Multi-view 3D target detection method and system assisted by 2D detection result

Abstract

The invention discloses a 2D detection result assisted multi-view 3D target detection method and system, and belongs to the field of image data processing and image detection. The method comprises the following steps: obtaining a multi-view feature map through a backbone network, and inputting the multi-view feature map into a 2D detection head to obtain a classification score and a bounding box; enhancing the multi-view feature map through the classification score; carrying out region-of-interest feature interception on the multi-view feature map through a bounding box and obtaining an adaptive Query; and obtaining a detection result by using the enhanced feature map and self-adaptive Query. The system comprises a feature extraction module, a 2D detection head module, a category attention module, a spatial appearance perception Query generator module and a 3D detection head module. According to the method, 2D detection is used as a pre-stage of 3D detection, and a Query method of dynamic space-time appearance perception is used, so that a small target and a long-distance target with fewer pixel points are fully concerned, and the problem that fixed reference point sparse query neglects the small and long-distance targets in a dynamic environment is solved.

Description

Technical Field

[0001] The present invention relates to the fields of image data processing and image detection, and in particular to a multi-view 3D target detection method and system assisted by 2D detection results. Background Art

[0002] Multi-view 3D object detection is a crucial task in autonomous driving systems. Compared to traditional 2D object detection, multi-view 3D object detection provides the actual location coordinates and category information of objects around the vehicle. Due to its crucial role in autonomous driving and its lower deployment cost compared to LiDAR-based solutions, multi-view 3D object detection has garnered widespread attention in the computer vision field in recent years. Existing 3D object detection methods can be roughly divided into two categories: dense bird's-eye view (BEV)-based methods and sparse query-based detection methods.

[0003] However, BEV methods convert perspective features into BEV features for dense detection, but their computational cost grows quadratically with the spatial extent, resulting in a significant increase in computational cost. Sparse query methods, inspired by DETR, use sparse learnable queries to aggregate features from perspective images. These queries are typically based on fixed 3D spatial reference points, enabling them to focus on feature aggregation in specific regions. While a fixed number of sparse queries can reduce computational cost, the reliance on empirical priors for the reference points in 3D space can result in reference points being too far from the target in dynamic environments, making it impossible to correctly localize the target and resulting in only a small number of positive queries. For example, 3D detectors such as StreamPETR often detect fewer bounding boxes than 2D detectors such as YOLOX, demonstrating a significant gap in robustness. This gap stems from the design of sparse fixed reference points in 3D object detectors, which can cause sparse queries to miss some objects. These missed objects are often small or distant, occupying fewer pixels and containing weaker semantic information, making sparse queries susceptible to interference from background or other pixel-rich objects. Several methods have proposed improvements to address these issues. For example, MV2D uses 2D bounding boxes to initialize 3D object queries, and the generated queries only interact with relevant foreground features. This allows 3D detection to benefit from more robust 2D detection while reducing the interference of background features on target features. Although the MV2D method alleviates some challenges in detecting small and distant objects, it still has the following shortcomings: (1) MV2D directly predicts target depth through neural networks to obtain 3D proposals. This regression-based method lacks robustness without explicit depth supervision signals; (2) The queries generated by MV2D only aggregate foreground features and ignore background features. Depth information usually comes from the overall feature context, and background features are not completely useless. Therefore, instead of removing background features, it is better to enhance foreground features to eliminate interference and noise that hinder localization performance; (3) MV2D uses RoIAlign to crop feature maps into fixed-size RoI features using bounding boxes of various shapes, which irreversibly distorts spatial information and loses background details, making it unreliable to predict depth directly from such features. Summary of the Invention

[0004] To solve the above problems, the present invention provides a multi-view 3D target detection method and system assisted by 2D detection results.

[0005] To implement the above technology, the details are as follows:

[0006] A multi-view 3D object detection method assisted by 2D detection results, comprising:

[0007] S1. Extract features from the input multi-view image through the backbone network, and then use the feature pyramid network to perform multi-scale feature fusion on the image features to obtain a multi-view feature map;

[0008] Multi-view image I∈R N×H×W×C As input, N represents the number of views, H and W represent the height and width of the image respectively, and C represents the channel of the multi-view image;

[0009] The backbone network extracts features from the input multi-view images as follows: the backbone network downsamples the multi-view images to obtain feature maps of multiple scales, namely, feature maps with a sampling rate of 1 / 8, feature maps with a sampling rate of 1 / 16, feature maps with a sampling rate of 1 / 32, and feature maps with a sampling rate of 1 / 64;

[0010] The way the feature pyramid network fuses feature maps is to upsample the feature map with a sampling rate of 1 / 64 to make it consistent with the feature map with a sampling rate of 1 / 32. Figure 1 The fusion feature map with a sampling rate of 1 / 32 is obtained by adding the 1×1 convolutions to adjust the number of channels; the fusion feature map with a sampling rate of 1 / 32 is upsampled to make it consistent with the feature map with a sampling rate of 1 / 16. Figure 1 The fused feature map with a sampling rate of 1 / 16 is obtained by adding the lateral connections; the feature map with a sampling rate of 1 / 64 is convolved by 1×1 to obtain a fused feature map with a sampling rate of 1 / 64; the fused feature map with a sampling rate of 1 / 64 is then downsampled with a step size of 2 to obtain a fused feature map with a sampling rate of 1 / 128, and the fused feature maps with sampling rates of 1 / 16, 1 / 32, 1 / 64, and 1 / 128 are output to the subsequent steps, namely, multi-view feature maps.

[0011] S2. Input the multi-view feature map into the 2D detection head to obtain the 2D detection results of the multi-view feature map. The 2D detection results include: classification score S cls with a bounding box.

[0012] S3, enhance the foreground features of the multi-view feature map using the classification scores in the 2D detection results;

[0013] S3.1. Generate channel attention weights for the multi-view feature map, and restore the channel attention weight dimension to the multi-view image channel attention weight size at the time of input by broadcasting, so as to obtain the basic channel attention weight A base , the expression is as follows:

[0014] A base =Broadcast(MLP(AvgPool(F)));

[0015] Where, F represents the multi-view feature map;

[0016] S3.2, classification score S cls Perform average pooling and maximum pooling operations in the channel dimension to obtain The results of the two operations are concatenated in the channel dimension, and then the channel dimension is changed to 1 through a convolution layer to obtain the spatial attention weight A. spatial ; Then the channel dimension is restored to the multi-view image channel dimension C at the time of input by broadcasting and the channel attention weight is added element by element to obtain the spatial channel attention weight A sc , the expression is as follows:

[0017] A spatial =Conv(Concat(AvgPool(S cls ),MaxPool(S cls ))),

[0018] A sc =A base +Broadcast(A spatial );

[0019] S3.3, the classification score S cls and spatial channel attention weight A sc Splicing in the channel dimension, then passing through a point convolution, and then passing through the Sigmoid function to obtain the category attention weight W cls , the expression is as follows:

[0020] W cls =Sigmoid(Conv(Concat(A sc ,S cls )));

[0021] S3.4. Set the category attention weight W cls Perform element-wise multiplication with the multi-view feature map to obtain a feature map enhanced with the classification score:

[0022] F ca =W class ⊙Conv(F).

[0023] S4. Based on the bounding box in the 2D detection result, the RoIAlign mechanism is used to extract the region of interest features from the multi-view feature map; and the bounding box, the region of interest features and the camera parameters of the corresponding image are input into the query generator to obtain an adaptive query. The steps are as follows:

[0024] S4.1. First, use the bounding box in the 2D detection result to use the RoIAlign mechanism to extract the object's region of interest features from the feature map;

[0025] The features of the region of interest are pre-set target features;

[0026] S4.2. Use convolution and pooling to change the spatial size of the intercepted region of interest features from h×w to 1×1, thereby obtaining compressed appearance information I a , the expression is as follows:

[0027] I a =Conv&Pool(O);

[0028] S4.3. Compress the appearance information I a Input into a multi-layer perceptron network, and then pass through a softmax to output the height classification probability, and compare the height classification probability with the pre-set 30 uniformly distributed height grids H grid ∈[0.5,4.5] multiplied to obtain the predicted object height H e , the expression is as follows:

[0029]

[0030] S4.4. Using the predicted object height H e , the height h of the bounding box in the 2D detection result b And the pixel focal length f on the y-axis, the depth d of the object is calculated by the pinhole imaging principle, and the expression is as follows:

[0031]

[0032] S4.5. Using the depth d of the object, the pixel coordinates of the bounding box output by 2D detection, and the camera parameters, the preliminary 3D coordinates P of the corresponding object can be calculated through coordinate transformation. rf , the expression is as follows:

[0033]

[0034] Where, represents the inverse matrix of the camera intrinsic parameter matrix; u and v represent the 2D pixel coordinates of the object in the image;

[0035] S4.6. Input the preliminary 3D coordinates into the position encoder to obtain the position code. Then, the position code is concatenated with the compressed appearance information in the channel dimension. After that, the feature is fused through the MLP layer to obtain the position code of spatial appearance perception. The expression is as follows:

[0036] PE a =MLP(Concat(PositionEncoder(P rf ),I a ));

[0037] S4.7. Add the spatial appearance-aware position encoding to the target of each layer in the transformer to obtain a query guided by appearance and spatial information.

[0038] S5. The feature map enhanced by the classification score is flattened and input into the Transformer decoder with the adaptive query for cross-attention mechanism. The query aggregates information on the feature map. Finally, the query that aggregates the object information is decoded by the linear layer to obtain the category label and spatial coordinates to complete the detection.

[0039] A multi-view 3D object detection system assisted by 2D detection results includes: a feature extraction module, a 2D detection head module, a category attention module, a spatial appearance perception query generator module, and a 3D detection head module.

[0040] The feature extraction module performs the following operations:

[0041] S1. Extract features from the input multi-view image through the backbone network, and then use the feature pyramid network to perform multi-scale feature fusion on the image features to obtain a multi-view feature map;

[0042] The 2D detection head module performs the following operations:

[0043] S2, inputting the multi-view feature maps into the 2D detection head to obtain the 2D detection results of the multi-view feature maps;

[0044] The category attention module performs the following operations:

[0045] S3, classification score S in 2D detection results cls Perform foreground feature enhancement on multi-view feature maps;

[0046] The spatial appearance query generator module performs the following operations:

[0047] S4. Extract the region of interest features from the multi-view feature map using the RoIAlign mechanism based on the bounding box in the 2D detection result; and input the bounding box, the region of interest features, and the camera parameters of the corresponding image into the query generator to obtain an adaptive query;

[0048] The 3D detection head module performs the following operations:

[0049] S5. The feature map enhanced by the classification score is flattened and input into the Transformer decoder with the adaptive query for cross-attention mechanism. The query aggregates information on the feature map. Finally, the query that aggregates the object information is decoded by the linear layer to obtain the category label and spatial coordinates.

[0050] Beneficial effects of the present invention:

[0051] 1. This paper considers 2D detection as a precursor to 3D detection, allowing 3D detection to fully utilize 2D detection results. A multi-view 3D target detection method assisted by 2D detection results is designed, which allows small targets occupying fewer pixels and distant targets to receive more attention.

[0052] 2. This paper proposes a dynamic spatiotemporal appearance-aware query, which solves the problem that sparse queries with fixed reference points ignore small and distant targets in dynamic environments.

[0053] 3. Compared with existing multi-view Figure 3 Compared with 2D target detection algorithms, the present invention can better detect small objects and distant objects that occupy fewer pixels. Since these small and distant objects occupy fewer pixels, their semantic information is weak and they are easily overlooked. The present invention strengthens the foreground features by using the classification information output by 2D detection and injects the object existence information output by 2D detection into the query to obtain an adaptive query that changes with the image. This allows the 3D detection head to pay more attention to foreground objects and improves 3D target detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of the steps of the present invention;

[0055] Figure 2 It is a network structure diagram of the present invention;

[0056] Figure 3 It is a detailed diagram of the network structure diagram of the present invention;

[0057] Figure 4 This is the baseline model detection effect diagram;

[0058] Figure 5 This is a detection effect diagram of the model proposed in this invention using ResNet-50 as the backbone network. DETAILED DESCRIPTION

[0059] The present invention uses nuScenes as an example dataset, which is further described in detail below with reference to specific examples.

[0060] Example 1:

[0061] like Figure 1 、 Figure 2 and Figure 3 As shown, a multi-view 3D target detection method assisted by 2D detection results includes the following steps:

[0062] S1. Extract features from the input multi-view image through the backbone network, and then use the feature pyramid network to perform multi-scale feature fusion on the image features to obtain a multi-view feature map;

[0063] Multi-view image I∈R N×H×W×C As input, N represents the number of viewpoints, which is 6 in the nuScenes dataset, H and W represent the height and width of the image respectively, C represents the channel of the multi-view image, C is 3, and the multi-view image in this experiment is set to 256*704;

[0064] The backbone network extracts features from the input multi-view images as follows: the backbone network downsamples the multi-view images to obtain feature maps of multiple scales, namely, feature maps with a sampling rate of 1 / 8, feature maps with a sampling rate of 1 / 16, feature maps with a sampling rate of 1 / 32, and feature maps with a sampling rate of 1 / 64;

[0065] The backbone network framework uses ResNet-50 to extract features;

[0066] The way the feature pyramid network fuses feature maps is to upsample the feature map with a sampling rate of 1 / 64 to make it consistent with the feature map with a sampling rate of 1 / 32. Figure 1 The fusion feature map with a sampling rate of 1 / 32 is obtained by adding the 1×1 convolutions to adjust the number of channels; the fusion feature map with a sampling rate of 1 / 32 is upsampled to make it consistent with the feature map with a sampling rate of 1 / 16. Figure 1 The fused feature map with a sampling rate of 1 / 16 is obtained by adding the lateral connections; the feature map with a sampling rate of 1 / 64 is convolved by 1×1 to obtain a fused feature map with a sampling rate of 1 / 64; the fused feature map with a sampling rate of 1 / 64 is then downsampled with a step size of 2 to obtain a fused feature map with a sampling rate of 1 / 128, and the fused feature maps with sampling rates of 1 / 16, 1 / 32, 1 / 64, and 1 / 128 are output to the subsequent steps, namely, multi-view feature maps.

[0067] S2. Input the multi-view feature map into the 2D detection head to obtain the 2D detection results of the multi-view feature map. The 2D detection results include: classification score S cls with a bounding box.

[0068] S3, enhance the foreground features of the multi-view feature map using the classification scores in the 2D detection results;

[0069] S3.1. Generate channel attention weights for the multi-view feature map, and restore the channel attention weight dimension to the multi-view image channel attention weight size at the time of input by broadcasting, so as to obtain the basic channel attention weight A base , the expression is as follows:

[0070] A base =Broadcast(MLP(AvgPool(F)));

[0071] Where, F represents the multi-view feature map;

[0072] S3.2, classification score S cls Perform average pooling and maximum pooling operations in the channel dimension to obtain The results of the two operations are concatenated in the channel dimension, and then the channel dimension is changed to 1 through a convolution layer to obtain the spatial attention weight A. spatial ; Then the channel dimension is restored to the multi-view image channel dimension C at the time of input by broadcasting and the channel attention weight is added element by element to obtain the spatial channel attention weight A sc , the expression is as follows:

[0073] A spatial =Conv(Concat(AvgPool(S cls ),MaxPool(S cls ))),

[0074] A sc =A base +Broadcast(A spatial );

[0075] S3.3, the classification score S cls and spatial channel attention weight A sc Splicing in the channel dimension, then passing through a point convolution, and then passing through the Sigmoid function to obtain the category attention weight W cls , the expression is as follows:

[0076] W cls =Sigmoid(Conv(Concat(A sc ,S cls )));

[0077] S3.4. Set the category attention weight W cls Perform element-wise multiplication with the multi-view feature map to obtain a feature map enhanced with the classification score:

[0078] F ca =W class ⊙Conv(F).

[0079] S4. Based on the bounding box in the 2D detection result, the RoIAlign mechanism is used to extract the region of interest features from the multi-view feature map; and the bounding box, the region of interest features and the camera parameters of the corresponding image are input into the query generator to obtain an adaptive query. The steps are as follows:

[0080] S4.1. First, use the bounding box in the 2D detection result to use the RoIAlign mechanism to extract the object's region of interest features from the feature map;

[0081] The features of the region of interest are pre-set target features;

[0082] S4.2. Use convolution and pooling to change the spatial size of the intercepted region of interest features from h×w to 1×1, thereby obtaining compressed appearance information I a , the expression is as follows:

[0083] I a =Conv&Pool(O);

[0084] S4.3. Compress the appearance information I a Input into a multi-layer perceptron network, and then pass through a softmax to output the height classification probability, and compare the height classification probability with the pre-set 30 uniformly distributed height grids H grid ∈[0.5,4.5] multiplied to obtain the predicted object height H e , the expression is as follows:

[0085]

[0086] S4.4. Using the predicted object height H e , the height h of the bounding box in the 2D detection result b And the pixel focal length f on the y-axis, the depth d of the object is calculated by the pinhole imaging principle, and the expression is as follows:

[0087]

[0088] S4.5. Using the depth d of the object, the pixel coordinates of the bounding box output by 2D detection, and the camera parameters, the preliminary 3D coordinates P of the corresponding object can be calculated through coordinate transformation. rf , the expression is as follows:

[0089]

[0090] Where, represents the inverse matrix of the camera intrinsic parameter matrix; u and v represent the 2D pixel coordinates of the object in the image;

[0091] S4.6. Input the preliminary 3D coordinates into the position encoder to obtain the position code. Then, the position code is concatenated with the compressed appearance information in the channel dimension. After that, the feature is fused through the MLP layer to obtain the position code of spatial appearance perception. The expression is as follows:

[0092] PE a =MLP(Concat(PositionEncoder(P rf ),I a ));

[0093] S4.7. Add the spatial appearance-aware position encoding to the target of each layer in the transformer to obtain a query guided by appearance and spatial information.

[0094] S5. The feature map enhanced by the classification score is flattened and input into the Transformer decoder with the adaptive query for cross-attention mechanism. The query aggregates information on the feature map. Finally, the query that aggregates the object information is decoded by the linear layer to obtain the category label and spatial coordinates to complete the detection.

[0095] A multi-view 3D object detection system assisted by 2D detection results includes: a feature extraction module, a 2D detection head module, a category attention module, a spatial appearance perception query generator module, and a 3D detection head module.

[0096] The feature extraction module performs the following operations:

[0097] S1. Extract features from the input multi-view image through the backbone network, and then use the feature pyramid network to perform multi-scale feature fusion on the image features to obtain a multi-view feature map;

[0098] The 2D detection head module performs the following operations:

[0099] S2, inputting the multi-view feature maps into the 2D detection head to obtain the 2D detection results of the multi-view feature maps;

[0100] The category attention module performs the following operations:

[0101] S3, classification score S in 2D detection results cls Perform foreground feature enhancement on multi-view feature maps;

[0102] The spatial appearance query generator module performs the following operations:

[0103] S4. Extract the region of interest features from the multi-view feature map using the RoIAlign mechanism based on the bounding box in the 2D detection result; and input the bounding box, the region of interest features, and the camera parameters of the corresponding image into the query generator to obtain an adaptive query;

[0104] The 3D detection head module performs the following operations:

[0105] S5. The feature map enhanced by the classification score is flattened and input into the Transformer decoder with the adaptive query for cross-attention mechanism. The query aggregates information on the feature map. Finally, the query that aggregates the object information is decoded by the linear layer to obtain the category label and spatial coordinates.

[0106] like Figure 4 and Figure 5 As shown in Figure 2, the present invention visualizes the detection results. The results show that the model pays more attention to small objects or distant objects that occupy fewer pixels, thereby successfully detecting objects that were previously missed.

[0107] The present invention was evaluated on the nuScenes dataset, using ResNet-50 as the feature backbone to extract features. The specific results are shown in Table 1:

[0108] Table 1 Comparison results of ResNet-50

[0109]

[0110] Among these methods, the method of the present invention achieved the best accuracy. When using ResNet-50 as the backbone network, the method of the present invention achieved 47.2% mAP and 56.8% NDS, surpassing the current state-of-the-art method StreamPETR, improving mAP by 2.2% and NDS by 1.8%. This is because difficult targets usually occupy fewer pixels, and thus these difficult targets are easily ignored by previous algorithms, resulting in missed detection. In contrast, by introducing 2D detection information, the present invention enables the model to strengthen its focus on foreground objects, thereby bringing significant performance improvements.

[0111] Example 2:

[0112] Example 2 is similar to Example 1, except that: the input multi-view image is set to 512*1408;

[0113] ResNet-101 is used as the feature backbone to extract features. The specific results are shown in Table 2:

[0114] Table 2 Comparison results of ResNet-101

[0115]

[0116] When using the larger ResNet-101 backbone network and increasing the image size to 1408×512, our method achieved a mAP of 52.3% and an NDS of 60.6%, an improvement of 1.9% in mAP and 1.4% in NDS over StreamPETR. This is because difficult objects typically occupy fewer pixels, making them easily overlooked by previous algorithms, leading to missed detections. In contrast, our method, by introducing 2D detection information, allows the model to focus more on foreground objects, resulting in significant performance improvements.

[0117] The embodiments described above are merely descriptions of preferred implementations of the present invention and are not intended to limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary engineering technicians in this field should fall within the scope of protection of the present invention. The technical contents to be protected by the present invention have been fully recorded in the technical requirements.

Claims

1. A multi-view 3D object detection method assisted by 2D detection results, characterized in that: The following steps are involved: S1. Extract features from the input multi-view image through the backbone network, and then use the feature pyramid network to perform multi-scale feature fusion on the image features to obtain a multi-view feature map; S2. Input the multi-view feature map into the 2D detection head to obtain the 2D detection results of the multi-view feature map. The 2D detection results include: classification score S cls with bounding box; S3, classification score S in 2D detection results cls Perform foreground feature enhancement on multi-view feature maps; S4. Extract the region of interest features from the multi-view feature map using the RoIAlign mechanism based on the bounding box in the 2D detection result; and input the bounding box, the region of interest features, and the camera parameters of the corresponding image into the query generator to obtain an adaptive query; S5. The feature map enhanced by the classification score is flattened and input into the Transformer decoder with the adaptive query for cross-attention mechanism. The query aggregates information on the feature map. Finally, the query that aggregates the object information is decoded by the linear layer to obtain the category label and spatial coordinates to complete the detection.

2. The multi-view 3D object detection method assisted by 2D detection results according to claim 1, characterized in that: The steps of performing foreground feature enhancement on the multi-view feature map using the classification scores in the 2D detection results are as follows: S3.

1. Generate channel attention weights for the multi-view feature map, and restore the channel attention weight dimension to the multi-view image channel attention weight size at the time of input by broadcasting, so as to obtain the basic channel attention weight A base ; S3.2, classification score S cls Perform average pooling and maximum pooling operations in the channel dimension to obtain The results of the two operations are concatenated in the channel dimension, and then the channel dimension is changed to 1 through a convolutional layer to obtain the spatial attention weight A. spatial ; Then the channel dimension is restored to the multi-view image channel dimension C at the time of input by broadcasting and the channel attention weight is added element by element to obtain the spatial channel attention weight A sc ; S3.3, the classification score S cls and spatial channel attention weight A sc Splicing in the channel dimension, then passing through a point convolution, and then passing through the Sigmoid function to obtain the category attention weight W cls ; S3.

4. Set the category attention weight W cls Perform element-wise multiplication with the multi-view feature map to obtain a feature map enhanced with the classification score.

3. The multi-view 3D object detection method assisted by 2D detection results according to claim 1, characterized in that: The steps of extracting the region of interest features from the multi-view feature map using the RoIAlign mechanism according to the bounding box in the 2D detection result and inputting the bounding box, the region of interest features and the camera parameters of the corresponding image into the query generator to obtain the adaptive query are as follows: S4.

1. First, use the bounding box in the 2D detection result to use the RoIAlign mechanism to extract the object's region of interest features from the feature map; The features of the region of interest are pre-set target features; S4.

2. Use convolution and pooling to change the spatial size of the intercepted region of interest features from h×w to 1×1, thereby obtaining compressed appearance information I a ; S4.

3. Compress the appearance information I a Input into a multi-layer perceptron network, and then pass through a softmax to output the height classification probability, and compare the height classification probability with the pre-set 30 uniformly distributed height grids H grid ∈[0.5,4.5] multiplied to obtain the predicted object height H e ; S4.

4. Using the predicted object height H e , the height h of the bounding box in the 2D detection result b And the pixel focal length f on the y-axis, the depth d of the object is calculated by the pinhole imaging principle; S4.

5. Using the depth d of the object, the pixel coordinates of the bounding box output by 2D detection, and the camera parameters, the preliminary 3D coordinates P of the corresponding object can be calculated through coordinate transformation. rf ; S4.

6. Input the preliminary 3D coordinates into the position encoder to obtain a position code, then perform a channel-wise concatenation operation on the position code and the compressed appearance information. The feature fusion is then performed through the MLP layer to obtain a spatial appearance-aware position code. S4.

7. Add the spatial appearance-aware position encoding to the target of each layer in the transformer to obtain a query guided by appearance and spatial information.

4. A multi-view 3D object detection system assisted by 2D detection results, characterized in that: include: Feature extraction module, 2D detection head module, category attention module, spatial appearance perception query generator module, 3D detection head module; The feature extraction module performs the following operations: S1. Extract features from the input multi-view image through the backbone network, and then use the feature pyramid network to perform multi-scale feature fusion on the image features to obtain a multi-view feature map; The 2D detection head module performs the following operations: S2, inputting the multi-view feature maps into the 2D detection head to obtain the 2D detection results of the multi-view feature maps; The category attention module performs the following operations: S3, classification score S in 2D detection results cls Perform foreground feature enhancement on multi-view feature maps; The spatial appearance query generator module performs the following operations: S4. Extract the region of interest features from the multi-view feature map using the RoIAlign mechanism based on the bounding box in the 2D detection result; and input the bounding box, the region of interest features, and the camera parameters of the corresponding image into the query generator to obtain an adaptive query; The 3D detection head module performs the following operations: S5. The feature map enhanced by the classification score is flattened and input into the Transformer decoder with the adaptive query for cross-attention mechanism. The query aggregates information on the feature map. Finally, the query that aggregates the object information is decoded by the linear layer to obtain the category label and spatial coordinates.

Images