A Multimodal 3D Perception Method Based on Deformable Attention Fusion

Abstract

This invention provides a multimodal 3D perception method based on deformable attention fusion. Employing a multimodal 3D perception architecture, it leverages geometric information from point clouds and semantic information from images to improve detection performance, achieving better detection results through the fusion of complementary features. Simultaneously, it proposes a multimodal fusion module based on deformable attention, dynamically capturing necessary scene features by parsing BEV features and aggregating them according to weights. Building upon this, to fully acquire the original geometric information from the point cloud, the point cloud BEV features are added to the aggregated features through residual connections, resulting in improved performance compared to single-modal methods. This invention enables effective dynamic aggregation of cross-modal features, offering greater flexibility and adaptability compared to traditional convolutional fusion methods.

Description

Technical Field

[0001] This invention relates to the field of multimodal 3D target detection, and more specifically to a multimodal 3D perception method based on deformable attention fusion. Background Technology

[0002] In recent years, with the rise of autonomous driving, perception has become a prerequisite for the normal operation of the system. Generally speaking, the sensors commonly used in autonomous vehicles include LiDAR and cameras. The point clouds and images collected can each serve as independent data sources for the perception system. Point clouds provide geometric and structural information about the driving scene, while images provide semantic information. The two types of data complement each other, and by constructing an appropriate fusion module to combine the advantages of both types of information, the performance of detection and perception can be effectively improved.

[0003] Various deep learning-based multimodal 3D perception methods have been proposed. One particularly high-performance method generates a unified bird's-eye view feature from point cloud and image data through viewpoint transformation. The resulting BEV (Bird's-Eye View) features from the point cloud and the image are then fed into a fusion module to obtain fused BEV features, which contain both geometric information from the point cloud and semantic information from the image. However, most current fusion modules are based on convolutional layers. On the one hand, convolutional layers, due to their limited receptive field, can only fuse information within the range of the convolutional kernel size, and the receptive field may contain redundant information, which is not conducive to effectively extracting fused features. On the other hand, due to the inaccuracy of depth estimation during image-to-BEV feature generation and the possible errors in viewpoint transformation, misalignment may occur between the BEV features of the two modalities, a problem that convolutional layers alone cannot solve. There is still considerable room for exploration in how to dynamically and adaptively select and effectively fuse multimodal features.

[0004] To achieve multimodal feature fusion, the BEV features of each modality must first be obtained. For point clouds, the SECOND model, commonly used in point cloud perception and detection, is employed. First, irregular point clouds in 3D space are organized into voxel data through voxelization. Then, spatial feature aggregation and extraction are performed using 3D sparse convolution. Finally, the 3D voxel features are reduced in height to obtain the 2D BEV features. For images, the CaDDN model from image perception and detection is used. First, image features are extracted. Depth prediction is performed based on the image features. The predicted depth distribution is multiplied by the image features to obtain the view frustum features. Then, the view frustum is projected into 3D space using the camera's intrinsic and extrinsic parameter matrices to obtain pseudo-voxel features. Finally, the pseudo-voxel features are reduced in height to obtain the 2D image BEV features.

[0005] Existing methods often fuse multimodal BEV features by constructing convolutional networks. The drawback of this method is that the receptive field during the fusion process is limited by the size of the convolutional kernel, and some redundant and unnecessary features in the receptive field are also included in the fusion. It cannot obtain the offset and corresponding weight of the features to be fused by parsing the BEV features point by point, nor can it capture and aggregate the features at the sampling positions point by point according to weight. Therefore, it cannot achieve flexible convolutional fusion. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a multimodal 3D perception method based on deformable attention fusion. Employing a multimodal 3D perception architecture, it leverages geometric information from point clouds and semantic information from images to improve detection performance, achieving better detection results through the fusion of complementary features. Simultaneously, it proposes a multimodal fusion module based on deformable attention, dynamically capturing necessary scene features by analyzing BEV features and aggregating them according to weights. Building upon this, to fully acquire the original geometric information from the point cloud, the point cloud BEV features are added to the aggregated features via residual connections.

[0007] Specifically, after obtaining BEV features from two modalities, this invention achieves fusion based on deformable attention and residual structures. Deformable attention is initially used for feature extraction and aggregation of 2D images. By analyzing keypoint features, it obtains the positional offsets and weights representing the importance of associated features. Sampling and weighted summation of associated features effectively aggregates features around keypoints dynamically. To more effectively aggregate multimodal features in 3D perception, this invention proposes a dynamic fusion module that uses deformable attention in the fusion of BEV features from 3D perception. Furthermore, since point clouds have significantly better detection performance than images, to retain more geometric structural information from point clouds, this invention also utilizes residual structures to add the BEV features generated from the point cloud to the multimodal aggregated features as the final fusion output.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal 3D perception method based on deformable attention fusion includes the following steps: S1: Input a set of N×d point cloud data and the corresponding frame's 3×h×w RGB image, where N is the number of points, d is the input feature dimension of the point cloud, and h and w are the image height and image width, respectively. S2: Obtain the BEV features of the two modalities, namely the point cloud BEV features and the image BEV features; S3: Element-wise summation of the BEV features of the two modalities is used as the query features for subsequent deformable attention. Query features containing multimodal information can better aggregate cross-modal information. S4: Using image BEV features as keys, query features dynamically aggregate image BEV features through deformable attention; S5: The features obtained by S4 aggregation are added element-wise to the original point cloud BEV features through residual connections to obtain the final output fused features; S6: The fused features are fed into the detection head to obtain the detection results of multimodal 3D perception.

[0009] Furthermore, in S2, obtaining the BEV characteristics of each of the two modes includes: For the image branch, the image backbone network is first used to extract features from the image to obtain high-level image features containing rich semantic information. Then, the image features are fed into the depth prediction network to obtain the depth probability distribution. The distribution is multiplied by the extracted image features to obtain the image frustum features. The 3D coordinates are projected into the frustum space through the camera's intrinsic and extrinsic parameters for interpolation sampling to obtain the pseudo-voxel features generated by the image. The pseudo-voxel features are compressed in the height direction to obtain the image BEV features. For point clouds, the point cloud is first voxelized and the features within the voxels are encoded to transform irregular point clouds into regular voxel features. Then, sparse 3D convolution operators are used to aggregate and extract voxel features. Finally, voxels are compressed in the height direction to obtain the BEV features of the point cloud.

[0010] Furthermore, S3 includes: after obtaining the BEV features of the two modes, sending them into a dynamic aggregation and fusion module based on deformable attention; The structure of the dynamic aggregation module is as follows: After obtaining the BEV features of the two modalities, the point cloud BEV features are denoted as... The BEV features of the image are denoted as The image BEV features consist of multiple levels. The hierarchical index is used, and the BEV feature scale is the same for both modalities; firstly, the BEV features are added element-wise to obtain the preliminary BEV fusion features. : ; in, The coordinates are on the BEV plane; The feature information, containing two modalities, is used as the query feature for deformable attention. This feature is then analyzed through a linear layer to obtain the sampled displacement offset and its corresponding weight. ; ; in, This is the sampling offset. The weights of the sampled features, and These are learnable parameters; Based on the sampling offset and weights, multi-level image BEV features are used as keys to dynamically aggregate cross-modal BEV features: F A p = ∑ m=1 M W m [ ∑ l=1 L ∑ k=1 K w mlk p W m ' F I (p+∆ p mlk p ) ] ; in, The BEV features obtained from aggregation, and For learnable parameters, For the number of heads of variable attention, is the number of sampling points, and m is the index value.

[0011] Furthermore, S5 includes: after obtaining the aggregated features, retaining the original geometric structure information from the point cloud, and completing the process by constructing residual connections: F LI p = W f [ F A p + ∑ l=1 L F I l p , F L p ] ; in, For learnable parameters, [*,*] To be spliced ​​along the channel, The final output is the fused feature; Integrated BEV features The scale is the same as the original point cloud BEV features.

[0012] Beneficial effects: 1) The features of point cloud and image are transformed to the same BEV perspective for fusion, so that the fused features contain geometric structure information from point cloud and semantic information from image. Detection is performed through fused features, which improves the performance compared to single modality. 2) This invention proposes a cross-modal feature aggregation method based on deformable attention, which can effectively and dynamically aggregate cross-modal features and is more flexible and adaptable than the traditional convolutional fusion method; 3) Based on the above, the BEV features of the point cloud are added to the aggregated features. The structure of the residual connection can preserve the geometric and structural information of the point cloud, and can achieve a stable improvement on the original performance.

[0013] In summary, this invention can obtain the offset and corresponding weight of the features to be fused by parsing BEV features point by point, and capture and aggregate the features at the sampling positions point by point according to weight. Through this dynamic and flexible fusion method, it can solve the shortcomings of existing convolutional fusion methods. Attached Figure Description

[0014] Figure 1 This is a flowchart of a multimodal 3D perception method based on deformable attention fusion according to the present invention; Figure 2 This is a schematic diagram of the overall framework for multimodal 3D fusion perception. Figure 3 A schematic diagram of the dynamic fusion structure of multimodal BEV features; Figure 4 This is a comparison chart of the multimodal fusion detection results. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0016] like Figure 1 As shown, a multimodal 3D perception method based on deformable attention fusion according to the present invention includes the following steps: STEP1: Input a set of N×d point cloud data and the corresponding frame's 3×h×w RGB image, where N is the number of points, d is the input feature dimension of the point cloud, and h and w are the image heights, respectively. STEP2: Obtain the BEV features of the two modalities, namely the point cloud BEV features and the image BEV features; STEP3: Element-wise sum the BEV features of the two modalities and use them as query features for subsequent deformable attention. Query features containing multimodal information can better aggregate cross-modal information. After obtaining the BEV features of the two modalities, in order to obtain query features containing multimodal information for better subsequent information aggregation, we first perform simple element operations on the BEV features of the two modalities to obtain preliminary fusion features, which will be used as query features for subsequent deformable attention.

[0017] STEP4: Using image BEV features as keys, query features dynamically aggregate image BEV features through deformable attention; STEP 5: Add the aggregated features to the original point cloud BEV features element-wise through residual connections to obtain the final output fused features; The features obtained by aggregation are added element-wise to the original BEV features obtained from the point cloud using the residual structure, and the resulting fused feature is the final output. This fused feature contains both the scene information dynamically captured and aggregated from the BEV features of the image and the geometric structure information of the original point cloud.

[0018] STEP 6: Input the fused features into the detection head to obtain the detection results of multimodal 3D perception.

[0019] like Figure 2 As shown, this invention uses BEV features under each modality as input. Therefore, it is first necessary to obtain the BEV features under each modality. The process of obtaining BEV features is as follows: For the image branch, the image backbone network is first used to extract features from the image to obtain high-level image features containing rich semantic information; then the image features are fed into the depth prediction network to obtain the depth probability distribution, and the distribution is multiplied by the extracted image features to obtain the image frustum features; the 3D coordinates are projected into the frustum space through the camera's intrinsic and extrinsic parameters for interpolation sampling to obtain the pseudo-voxel features generated by the image; the pseudo-voxel features are compressed in the height direction to obtain the image BEV features.

[0020] For point clouds, the point cloud is first voxelized and the features within the voxels are encoded to transform irregular point clouds into regular voxel features. Then, sparse 3D convolution operators are used to aggregate and extract voxel features. Finally, voxels are compressed in the height direction to obtain the BEV features of the point cloud.

[0021] After obtaining the BEV features of the two modalities, they are fed into the dynamic aggregation and fusion module based on deformable attention (starting from STEP 3) to obtain the fused BEV features, and the detection results are parsed by the detection head.

[0022] The structure of the dynamic aggregation module is as follows: like Figure 3 As shown, after obtaining the BEV features of the two modes, the point cloud BEV features are denoted as... The BEV features of the image are denoted as The image BEV features consist of multiple levels. The hierarchy number indicates that the BEV feature scale is the same for both modalities. First, the BEV features are element-wise summed to obtain preliminary BEV fusion features. : ; in, These are the coordinates on the BEV plane.

[0023] The data already contains feature information from two modalities, which can be used as query features for deformable attention. This data can then be analyzed through a linear layer to obtain the sampled displacement and its corresponding weights. ; ; in, This is the sampling offset. The weights of the sampled features, and These are learnable parameters.

[0024] Based on the sampling offset and weights, multi-level image BEV features are used as keys to dynamically aggregate cross-modal BEV features: F A p = ∑ m=1 M W m [ ∑ l=1 L ∑ k=1 K w mlk p W m ' F I (p+∆ p mlk p ) ] ; in, The BEV features obtained from aggregation, and For learnable parameters, For the number of heads of variable attention, is the number of sampling points. m is the index value.

[0025] After obtaining the aggregated features, since point cloud perception generally performs far better than image perception, it is still necessary to retain the original geometric structure information from the point cloud. This is accomplished by constructing residual connections. F LI p = W f [ F A p + ∑ l=1 L F I l p , F L p ] ; in, For learnable parameters, [*,*] To be spliced ​​along the channel, This refers to the final output fused features.

[0026] Integrated BEV features The scale is the same as the original point cloud BEV features, so it can be directly fed into the subsequent 3D perception detection head for prediction result output.

[0027] Figure 4 The diagram shows the sensory detection effect of the present invention. Figure 4 The red box represents the ground truth (GT) box, and the blue box represents the predicted bounding box output by the model. The left side shows the detection results of the single-modal point cloud, and the right side shows the detection results of the fusion perception method proposed in this invention. The black and white elliptical boxes indicate targets that this invention can detect but the baseline model misses. It can be seen that for the detection of objects of different sizes, the method of this invention can significantly reduce the number of missed detections.

Claims

1. A multimodal 3D perception method based on deformable attention fusion, characterized in that, Includes the following steps: S1: Input a set of N×d point cloud data and the corresponding frame's 3×h×w RGB image, where N is the number of points, d is the input feature dimension of the point cloud, and h and w are the image height and image width, respectively. S2: Obtain the BEV features of the two modalities, namely the point cloud BEV features and the image BEV features; S3: Element-wise summation of the BEV features of the two modalities is used as the query features for subsequent deformable attention. Query features containing multimodal information can better aggregate cross-modal information, including: after obtaining the BEV features of the two modalities, sending them into the dynamic aggregation and fusion module based on deformable attention. The structure of the dynamic aggregation module is as follows: After obtaining the BEV features of the two modalities, the point cloud BEV features are denoted as... The BEV features of the image are denoted as The image BEV features consist of multiple levels. The hierarchical index is used, and the BEV feature scale is the same for both modalities; firstly, the BEV features are added element-wise to obtain the preliminary BEV fusion features. : ； in, The coordinates are on the BEV plane; The feature information, containing two modalities, is used as the query feature for deformable attention. This feature is then analyzed through a linear layer to obtain the sampled displacement offset and its corresponding weight. ； ； in, This is the sampling offset. The weights of the sampled features, and These are learnable parameters; Based on the sampling offset and weights, multi-level image BEV features are used as keys to dynamically aggregate cross-modal BEV features: ； in, The BEV features obtained from aggregation, and For learnable parameters, For the number of heads of variable attention, m is the number of sampling points, and m is the index value; S4: Using image BEV features as keys, query features dynamically aggregate image BEV features through deformable attention; S5: The features obtained by S4 aggregation are added element-wise to the original point cloud BEV features through residual connections to obtain the final output fused features; S6: The fused features are fed into the detection head to obtain the detection results of multimodal 3D perception.

2. The multimodal 3D perception method based on deformable attention fusion according to claim 1, characterized in that, In step S2, obtaining the BEV characteristics of each of the two modes includes: For the image branch, the image backbone network is first used to extract features from the image to obtain high-level image features containing rich semantic information. Then, the image features are fed into the depth prediction network to obtain the depth probability distribution. The distribution is multiplied by the extracted image features to obtain the image frustum features. The 3D coordinates are projected into the frustum space through the camera's intrinsic and extrinsic parameters for interpolation sampling to obtain the pseudo-voxel features generated by the image. The pseudo-voxel features are compressed in the height direction to obtain the image BEV features. For point clouds, the point cloud is first voxelized and the features within the voxels are encoded to transform irregular point clouds into regular voxel features. Then, sparse 3D convolution operators are used to aggregate and extract voxel features. Finally, voxels are compressed in the height direction to obtain the BEV features of the point cloud.

3. The multimodal 3D perception method based on deformable attention fusion according to claim 1, characterized in that, S5 includes: after obtaining the aggregated features, retaining the original geometric structure information from the point cloud, and completing the process by constructing residual connections. ； in, For learnable parameters, To be spliced ​​along the channel, The final output is the fused feature; Integrated BEV features The scale is the same as the original point cloud BEV features.

Images