3D object detection method based on instance-scene fusion enhancement

Abstract

The application discloses a 3D target detection method based on instance-scene fusion enhancement, and aims to improve the 3D target detection performance of multi-modal sensor fusion. First, the application effectively transmits instance information from the image view to the bird's eye view (BEV) through the cross-view correlation mechanism, realizes the preliminary fusion of the instance features and the scene features. Secondly, the application further enhances the instance-related features in the instance-scene BEV features through the geometric and semantic dual attention mechanisms, and realizes the deep enhancement of the instance-scene fusion. The experimental results show that the method of the application significantly improves the precision and robustness of 3D target detection in the 4D millimeter wave radar and image fusion scene.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to a 3D target detection method based on instance-scene fusion enhancement, which aims to improve the 3D target detection performance of multimodal sensor fusion. Background Technology

[0002] With the rapid advancement of autonomous driving technology, 3D object detection plays a crucial role in autonomous driving systems. It enables vehicles to accurately perceive their surroundings and acquire vital information such as position and speed, thereby ensuring driving safety. Currently, the fusion of LiDAR (Light Detection and Ranging) and cameras is the primary method for achieving 3D object detection. LiDAR provides high-precision geometric depth data, while cameras capture rich semantic information; their complementarity significantly enhances the perception capability of 3D object detection. However, LiDAR sensors are expensive, and their performance may degrade under adverse weather conditions.

[0003] In contrast, millimeter-wave radar (4D radar) is gradually becoming an alternative in the field of autonomous driving perception due to its stable operation in adverse weather conditions and lower cost. 4D millimeter-wave radar can provide not only distance, velocity, and angle information of objects, but also height data, enabling higher resolution in complex scenes. However, the sparseness and high noise of radar point cloud data limit the performance of millimeter-wave radar in target detection tasks.

[0004] Existing multimodal fusion methods utilize rich semantic features from camera images, which to some extent improves the sparse and noisy information from millimeter-wave radar, thus achieving good progress in 3D object detection. However, existing radar-camera fusion methods mainly rely on scene-level fusion or instance-level fusion, and each method has inherent limitations. Scene-level fusion can provide global scene awareness, but due to inaccurate viewpoint transitions and insufficient attention to instance-related features, it leads to feature blurring. For example, BEVFusion achieves cross-modal information fusion in the BEV space, and the LXL method further utilizes monocular estimated depth to distinguish image features in the space, but neither of them considers using instance features that are easily detected in the perspective view for enhancement. While instance-level fusion performs well in optimizing instance features, it lacks global awareness and deep multimodal interaction, leading to insufficient features. For example, CeterFusion uses 2D object detection to filter the view frustum space, and CRAFT uses an attention mechanism to fuse instances obtained from multimodal detection, but both lack awareness of the entire 3D space. Although both methods have their advantages, there is currently no suitable method to effectively combine them. Summary of the Invention

[0005] Considering the insufficient accuracy and robustness of existing 3D target detection methods that fuse images and radar, this invention proposes a 3D target detection method based on instance-scene fusion enhancement. This method achieves preliminary fusion of instance features and scene features through a cross-view correlation mechanism to obtain instance-scene BEV features; it then achieves deep enhancement of instance-scene fusion through a geometric and semantic dual attention mechanism to obtain enhanced instance-scene BEV features; finally, it decodes the enhanced instance-scene BEV features to achieve 3D target detection and localization.

[0006] The present invention is specifically implemented using the following technical solutions:

[0007] A 3D object detection method based on instance-scene fusion enhancement, the specific steps of which are as follows:

[0008] S1: Through a cross-view correlation mechanism, 2D instance features are transferred to the bird's-eye view, achieving preliminary fusion of instance and scene features to obtain instance-scene BEV features. Details are as follows.

[0009] S11: Extract radar BEV features from millimeter-wave radar point clouds, simultaneously extract semantics, and transform the semantics into a bird's-eye view to obtain camera BEV features. The two are then fused to obtain scene features. Specifically, for images... Image features are obtained through encoding. F 2D Semantics are further obtained through feature extractors. Where C is the number of channels in the feature map, H and W represent the height and width of the image, respectively, and n represents the downsampling rate. Meanwhile, F... 2D Depth is obtained through feature extractor Where D is the number of predefined discrete depth intervals. Then, by taking the outer product of semantic C and depth D, the three-dimensional spatial features can be obtained. Ultimately, regarding F 3D Voxel pooling (·) is performed to obtain camera BEV features, and radar BEV features are extracted from millimeter-wave radar point clouds. The scene features are then fused with the camera BEV features and radar BEV features. Where X and Y represent the length and width dimensions of the BEV space; the three-dimensional spatial feature F 3D and scene features F S The calculation is expressed as follows:

[0010] F 3D =D×C,

[0011] F S =voxelpool(F 3D ).

[0012] S12: Perform 2D object detection and aggregation on the semantics described in S11 to obtain aggregated instance features. Specifically, for semantic C, a 2D object detector detects N instances, and feature pooling is used to pool the features within the bounding boxes of the N instances to obtain instance features. Next, a learnable vector is used to aggregate all instance features, resulting in aggregated instance features. Specifically, F I After concatenating with the vector, a multi-head self-attention mechanism is used to aggregate instance information. Then, the vector is extracted to obtain the aggregated instance feature v.

[0013] S13: For scene features F in S11 S In addition to the aggregated instance feature v in S12, a cross-view correlation mechanism is used to fuse the scene features and the aggregated instance features to obtain the instance-scene BEV feature, which enables the transfer of perspective view instance features to bird's-eye view scene features. The specific method is as follows:

[0014] F S Foreground scene features are obtained by foreground encoder and background encoder. and background scene features Aggregated instance features v and F o and F b Perform matrix correlation operations to obtain the foreground segmentation mask. Background segmentation mask m o and m b The calculation process is described as follows:

[0015] m o =σ(F o v T ),

[0016] m b =σ(F b v T ),

[0017] Where σ(·) represents the sigmoid function.

[0018] Cosine similarity is calculated for the foreground segmentation mask and the background segmentation mask using foreground scene features and background scene features respectively. o With F o Calculating the cosine similarity yields the foreground similarity vector v. o m b With F b Calculating the cosine similarity yields the background similarity vector v. b Each element in the vector represents the similarity between scene features and the mask.

[0019] Ultimately, v o With F o Performing dot product allows for the initial fusion of instance and scene features, yielding instance-scene BEV features.

[0020] In this process, two types of losses are used for supervision. On the one hand, the foreground segmentation mask and the background segmentation mask are supervised using ground truth foreground occupancy and ground truth background occupancy. On the other hand, the foreground similarity vector and the background similarity vector are supervised to make their element values ​​close to 1, so as to encourage foreground features and background features to approach the shape of ground truth occupancy during the encoding process.

[0021] S2: Through a geometric and semantic dual attention mechanism, the instance-related features in the instance-scene BEV features are further enhanced to achieve deep enhancement of instance-scene fusion, resulting in enhanced instance-scene BEV features. Specifically, this includes the following steps:

[0022] S21: Using an attention mechanism, the instance-scene BEV features described in S1 are semantically enhanced to obtain semantically enhanced instance-scene BEV features. Specifically, firstly, the instance-scene BEV features F are used... IS Initialize the query in the attention mechanism; then, use the semantically generated 3D spatial features F 3D As the value in the attention mechanism, the initialization of the semantic attention mechanism is completed; finally, the instance-scene BEV features are enhanced through the semantic attention mechanism to obtain the semantically enhanced instance-scene BEV features. S21 specifically includes the following steps: (1) Each query is lifted from the BEV plane to the column according to its position in the BEV space, and 3D reference points are sampled from the column, and the 3D reference points are assigned values ​​with the corresponding query; (2) The 3D reference points are used as queries, the three-dimensional space features are used as values, and these 3D reference points are projected to F 3D The process involves performing 3D deformable cross-attention operations; the specific description of the above process is as follows:

[0023] For the 3D query Q at position q q The semantically enhanced result is obtained through the 3D deformable attention mechanism 3DDCA operation.

[0024]

[0025] Where N represents the number of times the 3D query at position q is sampled in the deformable attention mechanism; Represents the camera projection function; A n ∈[0,1] are learnable attention weights; W represents feature projection weights; This represents the predicted offset for position q; This represents trilinear interpolation;

[0026] Ultimately, all semantic enhancements are completed. By reorganizing the features into the shape of the BEV feature space, we can obtain semantically enhanced instance-scene BEV features.

[0027] S22: Using an attention mechanism, the instance-scene BEV features described in S1 are enhanced using point clouds to obtain geometrically enhanced instance-scene BEV features. Specifically, firstly, the instance-scene BEV features F are used... IS The query in the attention mechanism is initialized; then, the radar BEV features generated from the point cloud are used as the value in the attention mechanism to complete the initialization of the geometric attention mechanism; finally, the F is enhanced through the geometric attention mechanism. IS The process involves obtaining geometrically enhanced instance-scene BEV features. Specifically, S22 includes the following steps: Downsampling the instance-scene BEV features and radar BEV features using convolutional methods, then using neighborhood cross attention (NCA) to interactively fuse the query and value to update the query. This process is repeated until the downsampling size reaches 8 times. Next, upsampling the instance-scene BEV features and radar BEV features using convolutional methods, then using neighborhood cross attention to interactively fuse the query and value to update the query. This process is repeated until the size is the same as the input size. The final output query is the geometrically enhanced instance-scene BEV feature.

[0028] S23: Add the semantically enhanced instance-scene BEV features described in S21 to the geometrically enhanced instance-scene BEV features described in S22 to obtain the enhanced instance-scene BEV features.

[0029] S3: Decode the enhanced instance-scene BEV features obtained in step S2 to achieve the detection and localization of 3D targets.

[0030] The beneficial effects of this invention are as follows:

[0031] This invention utilizes a cross-view correlation mechanism to effectively transfer instance information from the perspective view to the bird's-eye view (BEV), achieving initial fusion of instance features and scene features. Secondly, this invention further enhances instance-related features in the instance-scene BEV features through geometric attention and semantic attention mechanisms, resulting in enhanced instance-scene BEV features and achieving deep enhancement of instance-scene fusion. The enhanced instance-scene BEV features are then decoded to achieve 3D target detection and localization. This invention effectively combines scene-level fusion and instance-level fusion paradigms, leveraging the advantages of both. Experimental results show that this invention significantly improves the detection performance of current deep learning-based multimodal 3D target detection algorithms in complex scenes. Attached Figure Description

[0032] Figure 1 This is a flowchart of a 3D target detection method based on instance-scene fusion enhancement provided by an embodiment of the present invention.

[0033] Figure 2 This is a diagram illustrating the enhanced generalization effect of an embodiment of the present invention. Detailed Implementation

[0034] The following description, in conjunction with specific embodiments and accompanying drawings, provides further details.

[0035] Example 1

[0036] This embodiment provides a 3D object detection device based on instance-scene fusion enhancement, which is used to implement the 3D object detection method based on instance-scene fusion enhancement of the present invention. In the 3D object detection process, 2D instance features are transferred to the bird's-eye view through a cross-view correlation mechanism, achieving initial fusion of instance features and scene features to obtain instance-scene BEV features. Through a geometric and semantic dual attention mechanism, instance-related features in the instance-scene BEV features are further enhanced, achieving deep enhancement of instance-scene fusion to obtain enhanced instance-scene BEV features. The enhanced instance-scene BEV features are then decoded to achieve 3D object detection and localization. Ultimately, deep interaction and fusion of multimodal features can be achieved, thereby improving the performance of 3D object detection.

[0037] The detection device includes a feature extraction module, a feature fusion module, an instance-scene fusion module, and a decoding head;

[0038] The feature extraction module is used to extract features from the original millimeter-wave radar and monocular images;

[0039] The feature fusion module is used to obtain BEV features from the bird's-eye view based on the original millimeter-wave radar and monocular image features, and to fuse the radar BEV features and camera BEV features from the bird's-eye view to obtain scene features.

[0040] The instance-scene fusion module, based on a cross-perspective correlation mechanism, achieves preliminary fusion of instance features and scene features to obtain instance-scene BEV features. Then, based on a geometric and semantic dual attention mechanism, it achieves deep enhancement of instance-scene fusion, resulting in enhanced instance-scene BEV features.

[0041] The decoding head is used to decode the enhanced instance-scene BEV features to achieve the detection and localization of 3D targets.

[0042] like Figure 1 The detection process of this detection device can be divided into the following four stages:

[0043] In the first stage, the feature extraction module is used to extract features from the original millimeter-wave radar and monocular images, such as... Figure 1 As shown. For the image Image features are obtained through encoding. Here, C represents the number of channels in the feature map, H and W represent the height and width of the image, respectively, and n represents the downsampling rate. For millimeter-wave radar, a point cloud encoder is used to encode the millimeter-wave radar point cloud. The process includes steps such as cylinder voxelization, shallow feature extraction, and deep feature extraction to obtain the radar BEV features.

[0044] In the second stage, a feature fusion module is used to fuse radar BEV features and camera BEV features from a bird's-eye view perspective to obtain scene features, such as... Figure 1 As shown. F 2D Semantics are further obtained through feature extractors. Where C is the number of channels in the feature map, H and W represent the height and width of the image, respectively, and n represents the downsampling rate. Meanwhile, F... 2D Depth is obtained through feature extractor Where D is the number of predefined discrete depth intervals. Then, by taking the outer product of semantic C and depth D, the three-dimensional spatial features can be obtained. Ultimately, regarding F 3D Voxel pooling (·) is performed to obtain the camera BEV features. Then, the radar BEV features from the first stage are fused with the camera BEV features through a cross-modal fusion network to obtain the scene features. Where X and Y represent the length and width dimensions of the BEV space. Three-dimensional spatial features F 3D and scene features F S The calculation is expressed as follows:

[0045] F 3D =D×C,

[0046] F S =voxelpool(F 3D ).

[0047] In the third stage, the instance-scene fusion module achieves preliminary fusion of instance features and scene features based on a cross-view correlation mechanism to obtain instance-scene BEV features. Then, a deeper enhancement of instance-scene fusion is achieved based on a geometric and semantic dual attention mechanism to obtain enhanced instance-scene BEV features. The process is as follows: Figure 1 As shown.

[0048] First, a preliminary fusion of instance and scene features is achieved based on a cross-view correlation mechanism. Specifically, for semantic C, a 2D object detector detects N instances, and feature pooling is used to pool the features within the bounding boxes of the N instances to obtain instance features. Next, a learnable vector is used to aggregate all instance features, resulting in aggregated instance features. Specifically, F I After concatenating with the vector, a multi-head self-attention mechanism is used to aggregate instance information. Then, the vector is extracted to obtain the aggregated instance feature v. For the scene feature F... S And aggregated instance features v, using a cross-view correlation mechanism to initially fuse scene features and aggregated instance features, to obtain instance-scene BEV features. Specifically, F S Foreground scene features are obtained by foreground encoder and background encoder. and background scene features Aggregated instance features v and F o and F b Perform matrix correlation operations to obtain the foreground segmentation mask. Background segmentation mask m o and m b The calculation process is described as follows:

[0049] m o =σ(F o v T ),

[0050] m b =σ(F b v T ),

[0051] Where σ(·) represents the sigmoid function. Furthermore, cosine similarity is calculated for the foreground and background segmentation masks using foreground scene features and background scene features respectively. o With Fo Calculating the cosine similarity yields the foreground similarity vector v. o m b With F b Calculating the cosine similarity yields the background similarity vector v. b Each element in the vector represents the similarity between scene features and the mask. Finally, v... o With F o Performing dot product allows for the initial fusion of instance and scene features, yielding instance-scene BEV features. In this process, two types of losses are used for supervision. On the one hand, the foreground segmentation mask and the background segmentation mask are supervised using ground truth foreground occupancy and ground truth background occupancy. On the other hand, the foreground similarity vector and the background similarity vector are supervised to make their element values ​​close to 1, so as to encourage foreground features and background features to approach the shape of ground truth occupancy during the encoding process.

[0052] Next, through a geometric and semantic dual attention mechanism, the instance-related features in the instance-scene BEV features are further enhanced to achieve deep enhancement of instance-scene fusion, resulting in enhanced instance-scene BEV features. Specifically, F IS By enhancing the instance-scene BEV features separately using geometric attention and semantic attention mechanisms, and then summing the resulting features, a fully enhanced instance-scene BEV feature can be obtained. For the semantic attention mechanism, firstly, the instance-scene BEV feature F is used... IS Initialize the query in the attention mechanism; then, use the semantically generated 3D spatial features F 3D As the value in the attention mechanism, the semantic attention mechanism is initialized. Finally, the instance-scene BEV features are enhanced through the semantic attention mechanism to obtain semantically enhanced instance-scene BEV features. For the geometric attention mechanism, firstly, the instance-scene BEV features F are used... IS The query in the attention mechanism is initialized; then, the radar BEV features generated from the point cloud are used as the value in the attention mechanism to complete the initialization of the geometric attention mechanism; finally, the F is enhanced through the geometric attention mechanism. IS This yields geometrically enhanced instance-scene BEV features. The semantically enhanced instance-scene BEV features are then added to the geometrically enhanced instance-scene BEV features to obtain the enhanced instance-scene BEV features.

[0053] In the fourth stage, an anchor-based feature decoding head is used to decode the enhanced instance-scene BEV features to achieve the detection and localization of 3D targets.

[0054] Figure 2This figure shows the results of the proposed method on the View-of-Delft and TJ4DRadSet datasets. Each image corresponds to a data frame containing the image and radar points (grey), with red triangles marking the vehicle's position. Orange and yellow boxes represent the ground truth bounding boxes in the perspective and bird's-eye view, respectively. Green and blue boxes represent the bounding boxes predicted by the proposed method. The upper right corner of the figure shows a visualization of the BEV feature map. This figure demonstrates the detection performance of the proposed method for cars, pedestrians, and bicycles on the dataset.

Claims

1. A 3D target detection method based on instance-scene fusion enhancement, characterized in that, First, 2D instance features are transferred to a bird's-eye view to achieve initial fusion of instance features and scene features, resulting in instance-scene BEV features. Then, instance-related features in the instance-scene BEV features are further enhanced to achieve deep enhancement of instance-scene fusion, resulting in enhanced instance-scene BEV features. Finally, the enhanced instance-scene BEV features are decoded to achieve detection and localization of 3D targets. The specific steps of the instance-scene fusion-enhanced 3D object detection method are as follows: S1: Through a cross-view correlation mechanism, 2D instance features are transferred to the bird's-eye view to achieve the initial fusion of instance and scene features, resulting in instance-scene BEV features; S11: Extract radar BEV features from millimeter-wave radar point clouds, extract semantics, and transform the semantics into a bird's-eye view to obtain camera BEV features. Then, fuse the two to obtain scene features. S12: Perform 2D object detection and aggregation on the semantics described in S11 to obtain aggregated instance features; S13: The scene features described in S11 and the aggregated instance features described in S12 are initially fused through a cross-perspective correlation mechanism to obtain instance-scene BEV features; S2: By using a geometric and semantic dual attention mechanism, the instance-related features in the instance-scene BEV features are further enhanced to achieve deep enhancement of instance-scene fusion and obtain the enhanced instance-scene BEV features; S21: Using an attention mechanism, semantic enhancement is applied to the instance-scene BEV features described in S1 to obtain semantically enhanced instance-scene BEV features; S22: The instance-scene BEV features described in S1 are enhanced using point cloud through an attention mechanism to obtain geometrically enhanced instance-scene BEV features; S23: Add the semantically enhanced instance-scene BEV features described in S21 to the geometrically enhanced instance-scene BEV features described in S22 to obtain the enhanced instance-scene BEV features; S3: Decode the enhanced instance-scene BEV features obtained in step S2 to achieve the detection and localization of 3D targets.

2. The 3D target detection method based on instance-scene fusion enhancement according to claim 1, characterized in that, Step S11 specifically involves: for the image Image features are obtained through encoding. F 2D Semantics are obtained through feature extractor Where C is the number of channels in the feature map, H and W represent the height and width of the image, respectively, and n represents the downsampling rate; meanwhile, F 2D Depth is obtained through feature extractor Where D is the number of predefined discrete depth intervals; then, the semantic C and depth D are outer products to obtain the three-dimensional spatial features. Ultimately, regarding F 3D Voxel pooling (·) is performed to obtain camera BEV features, and radar BEV features are extracted from millimeter-wave radar point clouds. The scene features are then fused with the camera BEV features and radar BEV features. Where X and Y represent the length and width dimensions of the BEV space; the three-dimensional spatial feature F 3D and scene features F S The calculation is expressed as follows: F 3D =D×C, F S =voxelpool(F 3D )。 3. The 3D target detection method based on instance-scene fusion enhancement according to claim 1, characterized in that, Step S12 specifically involves: For semantic C, a 2D object detector detects N instances, and feature pooling is used to pool the features within the bounding boxes of the N instances to obtain instance features. Next, a learnable vector is used to aggregate all instance features, resulting in aggregated instance features. Specifically, F I After concatenating with the vector, a multi-head self-attention mechanism is used to aggregate instance information. Then, the vector is extracted to obtain the aggregated instance feature v.

4. The 3D target detection method based on instance-scene fusion enhancement according to claim 1, characterized in that, In step S13: For scene feature F S And aggregated instance features v, using a cross-view correlation mechanism to initially fuse scene features and aggregated instance features to obtain instance-scene BEV features; the specific method is as follows: F s Foreground scene features are obtained by foreground encoder and background encoder. and background scene features X and Y represent the length and width dimensions of the BEV space, C is the number of channels in the feature map, and the aggregated instance feature v is related to F. o and F b Perform matrix correlation operations to obtain the foreground segmentation mask. Background segmentation mask m o and m b The calculation process is described as follows: m o =σ(F o v T ), m b =σ(F b v T ), Where σ(·) represents the sigmoid function; Cosine similarity is calculated for the foreground segmentation mask and the background segmentation mask using foreground scene features and background scene features respectively; m o With F o Calculate the cosine similarity to obtain the foreground similarity vector v o m b With F b Calculating the cosine similarity yields the background similarity vector v. b Each element in the vector represents the similarity between scene features and the mask; Ultimately, v o With F o Perform dot product to achieve preliminary fusion of instance and scene features, obtaining instance-scene BEV features. In this process, two types of loss are used for supervision. On the one hand, the foreground segmentation mask and the background segmentation mask are supervised using the ground truth foreground mask and the ground truth background mask. On the other hand, the foreground similarity vector and the background similarity vector are supervised to make their element values ​​close to 1.0, so as to encourage the foreground scene features and the background scene features to approach the shape of the ground truth mask during the encoding process.

5. The 3D target detection method based on instance-scene fusion enhancement according to claim 1, characterized in that, S21 specifically refers to: First, using the instance-scenario BEV feature F IS Initialize the query in the attention mechanism; then, use the semantically generated 3D spatial features F 3D As the value in the attention mechanism, the initialization of the semantic attention mechanism is completed; finally, the instance-scene BEV features are enhanced through the semantic attention mechanism to obtain semantically enhanced instance-scene BEV features.

6. The 3D target detection method based on instance-scene fusion enhancement according to claim 1, characterized in that, Step S21 specifically includes the following steps: (1) lifting each query from the BEV plane to the column according to its position in the BEV space, sampling 3D reference points from the column, and assigning values ​​to the 3D reference points with the corresponding query; (2) using the 3D reference points as queries and the three-dimensional spatial features as values, and projecting these 3D reference points onto F 3D The process involves performing 3D deformable cross-attention operations; the specific description of the above process is as follows: For the 3D query Q at position q q The semantically enhanced result is obtained through a three-dimensional deformable cross-attention mechanism. Among them, F 3D Represents the three-dimensional spatial features, where N represents the number of samples taken from the 3D query at position q in the three-dimensional deformable cross-attention mechanism; Represents the camera projection function; A n ∈[0,1] are learnable attention weights; W represents feature projection weights; This represents the predicted offset for position q; This represents trilinear interpolation; Ultimately, all semantic enhancements are completed. By reorganizing the features into the shape of the BEV feature space, we can obtain semantically enhanced instance-scene BEV features.

7. The 3D target detection method based on instance-scene fusion enhancement according to claim 1, characterized in that, S22 specifically involves: First, using the instance-scenario BEV feature F IS The query in the attention mechanism is initialized; then, the radar BEV features generated from the point cloud are used as the value in the attention mechanism to complete the initialization of the geometric attention mechanism; finally, the F is enhanced through the geometric attention mechanism. IS This yields geometrically enhanced instance-scene BEV features.

8. The 3D target detection method based on instance-scene fusion enhancement according to claim 1, characterized in that, Step S22 specifically includes the following steps: Downsampling the instance-scene BEV features and radar BEV features using convolutional methods, then using a neighborhood attention mechanism to interactively fuse the query and value to update the query, repeating this process until the downsampling size reaches 8 times; Next, upsampling the instance-scene BEV features and radar BEV features using convolutional methods, then using a neighborhood attention mechanism to interactively fuse the query and value to update the query, repeating this process until the size is the same as the input size; the final output query is the geometrically enhanced instance-scene BEV feature.

Images