Multi-modal three-dimensional target detection method based on attention mechanism

Abstract

The invention provides a multi-modal three-dimensional target detection method based on an attention mechanism, and the method comprises the steps: obtaining image data and LiDAR point cloud data, carrying out the preprocessing of the image data and the LiDAR point cloud data, carrying out the feature extraction of the processed image data and the processed LiDAR point cloud data, and carrying out the feature extraction of the processed image data and the processed LiDAR point cloud data; and fusing the extracted image features and the point cloud features based on a multimodal feature fusion algorithm of bidirectional information interaction, optimizing the fused features, eliminating redundancy and noise, performing refined processing on the fused features in a specific region of interest, inputting the fused and optimized features into a target detection module, and performing target detection on the target. Through candidate box generation, classification and regression tasks, classification, positioning and identification of a multi-modal three-dimensional target are completed, based on a non-maximum suppression algorithm, a redundant detection box is removed, and a detection result is output, including a category label of the target, a bounding box coordinate and a confidence score. According to the invention, the efficiency and detection precision of multi-modal fusion are improved, and the method is convenient to use.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal three-dimensional target detection, and in particular to a multimodal three-dimensional target detection method based on an attention mechanism. Background Art

[0002] With the rapid development of autonomous driving, intelligent transportation, unmanned aerial vehicles and other fields, three-dimensional object detection has become a crucial technical topic. By analyzing the depth information and two-dimensional image data obtained by the sensor, three-dimensional object detection technology can more accurately identify and locate objects, providing key support for autonomous driving decision-making systems. However, the limitations of a single sensor have made traditional three-dimensional object detection methods face many challenges, such as insufficient robustness in complex environments, recognition errors caused by target occlusion or sensor noise, etc. At present, sensors such as LiDAR and cameras have become mainstream three-dimensional perception devices. LiDAR can provide high-precision depth information and is suitable for large-scale and high-precision object detection, but it is relatively weak in processing detail information and color information. Although traditional two-dimensional image data can provide rich visual information, it has great limitations in depth perception and spatial positioning. Therefore, how to effectively fuse LiDAR and image data to overcome the limitations of a single sensor has become an important research direction in the field of multimodal three-dimensional object detection. In recent years, the application of deep learning technology, especially convolutional neural networks and attention mechanisms, has provided new ideas for multimodal fusion. By effectively combining data from different sensors through feature-level fusion, decision-level fusion, etc., it is expected to improve detection accuracy and robustness.

[0003] There are two main forms of existing multimodal fusion methods: serial fusion and parallel fusion. Serial fusion is to use the image features of the specified area to enhance the point cloud features. Although this method can enrich the point cloud representation, the lidar signal is always affected by various conditions such as weather, lighting, and reflection. They are essentially over-reliant on the LiDAR modality, do not deeply explore the importance of images for target positioning, lack the full use of the rich semantic information of the camera sensor, and cannot produce any prediction results when the lidar signal fails. Although parallel fusion has solved the dependence of the current serial fusion method on the lidar input and has two modalities to further improve the detection accuracy, since most of the current parallel fusion methods are a single modality guiding another modality, information can only be transferred from one modality to another, which may limit the flow of information. They do not consider that the two-way interaction of the two modalities can make better use of complementary information. Therefore, the current fusion model is specific to the association between the geometric and depth information of sparse point cloud features and the dense semantic features of the image. At the same time, although the existing multimodal three-dimensional target detection algorithm solves the problem of three-dimensional spatial information loss by converting two-dimensional images into pseudo point clouds, the large number of points in the pseudo point cloud leads to a large amount of calculation and poor attention to the spatial information features in the target area, resulting in a lack of rich structural information, which directly affects the understanding, identification and positioning of targets in the area of ​​interest.

[0004] Therefore, it is necessary to design a multimodal 3D object detection method based on attention mechanism. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a multimodal three-dimensional object detection method based on an attention mechanism.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] The present invention provides a multimodal three-dimensional object detection method based on an attention mechanism, comprising:

[0008] Obtain image data and LiDAR point cloud data and preprocess them;

[0009] Perform feature extraction on the processed image data and LiDAR point cloud data;

[0010] The extracted image features and point cloud features are fused using a multimodal feature fusion algorithm based on two-way information interaction;

[0011] Optimize the fused features to eliminate redundancy and noise. At the same time, refine the fused features in a specific region of interest.

[0012] The fused and optimized features are input into the target detection module, and the classification, positioning and recognition of multimodal 3D targets are completed through candidate box generation, classification and regression tasks.

[0013] Based on the non-maximum suppression algorithm, redundant detection boxes are removed and the detection results are output, including the target's category label, bounding box coordinates, and confidence score.

[0014] Preferably, the image data and LiDAR point cloud data are both obtained from the public dataset KITTI.

[0015] Preferably, the image data and LiDAR point cloud data are preprocessed, specifically:

[0016] The image data is input into the depth completion module to convert the 2D image data into a pseudo point cloud to obtain the same 3D spatial information as the original point cloud;

[0017] Perform point cloud filtering, coordinate conversion and voxelization on the LiDAR point cloud data to convert the LiDAR point cloud data into a standard format.

[0018] Preferably, feature extraction is performed on the processed image data and LiDAR point cloud data, specifically:

[0019] For image data, the key spatial and texture features are extracted through convolutional neural networks, and voxelization and sparseness are performed to obtain pseudo point cloud RoI features of image branches;

[0020] For LiDAR point cloud data, the deep learning method Voxel-RCNN is used as the three-dimensional backbone network to process the LiDAR point cloud data, extract the spatial information, depth information and local structure features of the LiDAR point cloud data, and obtain the LiDAR RoI features of the point cloud branch. Based on the self-attention RoI feature extraction module, more effective information is captured in the key area to enrich the LiDAR RoI features.

[0021] Preferably, the extracted image features and point cloud features are fused using a multimodal feature fusion algorithm based on two-way information interaction, specifically:

[0022] The extracted image features and point cloud features are fused. During the fusion process, the complementary information between the two modalities is aggregated based on the cross-attention mechanism to obtain the pseudo point cloud RoI features and LiDAR RoI features, and the features are bidirectionally interacted, namely L2P branch and P2L branch. Among them, the L2P branch uses LiDAR RoI features as queries and pseudo point cloud RoI features as keys and values. The pseudo point cloud RoI features are used to guide the pseudo point cloud RoI features, and the correlation between the two modalities is dynamically captured through a multi-head cross-attention mechanism; the P2L branch uses pseudo point cloud RoI features as queries and LiDAR RoI features as keys and values. The pseudo point cloud RoI features are used to guide the LiDAR RoI features. The pseudo point cloud guides the search for relevant information of the LiDAR point cloud, and the interactive features are obtained through a multi-head cross-attention mechanism.

[0023] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0024] The present invention provides a multimodal three-dimensional target detection method based on an attention mechanism, the method comprising: acquiring image data and LiDAR point cloud data, and preprocessing the data, performing feature extraction on the processed image data and LiDAR point cloud data, fusing the extracted image features and point cloud features based on a multimodal feature fusion algorithm for two-way information interaction, optimizing the fused features, eliminating redundancy and noise, and at the same time, refining the fused features in a specific region of interest, inputting the fused and optimized features into a target detection module, completing the classification, positioning and recognition of the multimodal three-dimensional target through candidate frame generation, classification and regression tasks, removing redundant detection frames based on a non-maximum suppression algorithm, and outputting detection results, including the category label, bounding box coordinates and confidence score of the target.

[0025] The present invention has the following advantages:

[0026] 1. The present invention implements self-attention RoI feature extraction (SARoIFE), which can focus on the details of the target area while reducing the overall computational workload, greatly improving the detection efficiency;

[0027] 2. The present invention adopts multimodal feature fusion (FBIF) with two-way information interaction. By performing two-way information transmission between different modal features during the fusion process, the complementarity between modalities is enhanced, so that image features and point cloud features are mutually transmitted, updated and optimized during the fusion process, thereby better capturing the spatial features and detail information of the target, and significantly improving the efficiency and detection accuracy of multimodal fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0030] Figure 2 A flowchart of a specific method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The purpose of the present invention is to provide a multimodal three-dimensional target detection method based on an attention mechanism, which can realize multimodal three-dimensional target detection and improve the efficiency and detection accuracy of multimodal fusion, and is easy to use.

[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Figure 1 A flow chart of a method provided by an embodiment of the present invention, Figure 2 A flowchart of a specific method provided by an embodiment of the present invention is shown in FIG. Figure 1 and Figure 2 As shown, the present invention provides a multimodal three-dimensional target detection method based on an attention mechanism, comprising:

[0035] Step 100: Obtain image data and LiDAR point cloud data, and pre-process them;

[0036] Step 200: extracting features from the processed image data and LiDAR point cloud data;

[0037] Step 300: fusing the extracted image features and point cloud features using a multimodal feature fusion algorithm based on two-way information interaction;

[0038] Step 400: Optimize the fused features to eliminate redundancy and noise, and at the same time, refine the fused features in a specific region of interest;

[0039] Step 500: Input the fused and optimized features into the target detection module, and complete the classification, positioning and recognition of multimodal three-dimensional targets through candidate box generation, classification and regression tasks;

[0040] Step 600: Based on the non-maximum suppression algorithm, redundant detection boxes are removed and the detection results are output, including the category label, bounding box coordinates and confidence score of the target.

[0041] In step 100, the image data and LiDAR point cloud data are both obtained from the public dataset KITTI.

[0042] In step 100, the image data and LiDAR point cloud data are preprocessed, specifically:

[0043] The image data is input into the Depth Completion module to convert the 2D image data into a pseudo point cloud to obtain the same 3D spatial information as the original point cloud.

[0044] The LiDAR point cloud data is filtered, coordinate transformed and voxelized to convert the LiDAR point cloud data into a standard format so that the point cloud data can be input into the subsequent network in a standard format.

[0045] In step 200, feature extraction is performed on the processed image data and LiDAR point cloud data, specifically:

[0046] Feature extraction of images and point clouds are two separate branches;

[0047] For image data, the key spatial and texture features are extracted through convolutional neural networks (Convs), and voxelization and sparseness are performed to obtain pseudo point cloud RoI features of image branches. These features mainly include the edge, shape, color, etc. of the object;

[0048] For LiDAR point cloud data, the deep learning method Voxel-RCNN is used as the three-dimensional backbone network to process the LiDAR point cloud data, extract the spatial information, depth information and local structure features of the LiDAR point cloud data, and obtain the LiDAR RoI features of the point cloud branch. Based on the self-attention RoI feature extraction module (SARoIFE), more effective information is captured in the key area to enrich the LiDAR RoI features.

[0049] In step 300, the extracted image features and point cloud features are fused using a multimodal feature fusion algorithm based on two-way information interaction, specifically:

[0050] The extracted image features and point cloud features are fused. During the fusion process, the complementary information between the two modalities is aggregated based on the cross-attention mechanism to obtain the pseudo point cloud RoI features and LiDAR RoI features, and the features are bidirectionally interacted, namely L2P branch and P2L branch. Among them, the L2P branch uses LiDAR RoI features as queries and pseudo point cloud RoI features as keys and values. The pseudo point cloud RoI features are used to guide the pseudo point cloud RoI features, and the correlation between the two modalities is dynamically captured through a multi-head cross-attention mechanism; the P2L branch uses pseudo point cloud RoI features as queries and LiDAR RoI features as keys and values. The pseudo point cloud RoI features are used to guide the LiDAR RoI features. The pseudo point cloud guides the search for relevant information of the LiDAR point cloud, and the interactive features are obtained through a multi-head cross-attention mechanism.

[0051] In step 400, the fused features are optimized to eliminate redundancy and noise. At the same time, the fused features are refined in a specific region of interest, specifically:

[0052] Use multi-layer perceptron (MLP) or other feature optimization modules to optimize the fused features, eliminate redundancy and noise, and ensure that the fused features are more compact and effective. At the same time, in a specific region of interest, the fused features are refined, and the attention mechanism is introduced to fuse the interactive features to enhance the detection accuracy.

[0053] In step 500 of this embodiment, the fused and optimized features are input into the target detection module, and the specific implementation steps of completing the classification, positioning and recognition of the multimodal three-dimensional target through candidate box generation, classification and regression tasks are as follows:

[0054] First, the fused and optimized multimodal features are input into the candidate box generation module. This module generates initial candidate boxes through the Region Proposal Network (RPN) or the Anchor-based mechanism. Specifically, the RPN slides a window on the feature map, uses predefined Anchor boxes (boxes of different sizes and proportions) to scan the locations where the target may appear, and generates a set of candidate boxes. These candidate boxes will filter out the areas that may contain the target through a preliminary confidence assessment, and predict the preliminary bounding box coordinates of each candidate box.

[0055] For the generated candidate boxes, the corresponding feature regions (Region of Interest, RoI) are further extracted. Through RoI pooling or RoI alignment (RoIAlign) operations, the features of the candidate box area are mapped to fixed-size feature representations. These feature representations will be input into the subsequent classification and regression networks for refined processing. At the same time, the feature extraction module based on the self-attention mechanism will capture more effective information in the key areas of the candidate box, further enriching the feature representation of the candidate box.

[0056] In the classification task, the features of the candidate box are input into the classification branch, and the target category of the candidate box is predicted using the fully connected layer or convolutional layer. The classification branch outputs the probability distribution (Softmax output) that each candidate box belongs to different categories, and determines the target category label of the candidate box based on the probability value. At the same time, the classification branch also outputs a confidence score to measure the model's confidence in the classification result of the candidate box.

[0057] In the regression task, the features of the candidate box are input into the regression branch to predict the precise bounding box coordinates of the candidate box. The regression branch outputs the bounding box offset of each candidate box (i.e., the adjustment value relative to the initial candidate box), including the three-dimensional position (x, y, z), size (width, height, depth) and orientation of the target. The candidate box is adjusted by these offsets to generate a more accurate target bounding box, thereby achieving precise positioning of the target.

[0058] The classification and regression tasks are jointly optimized. The model trains the classification branch and the regression branch simultaneously through a multi-task loss function (such as the weighted sum of the classification loss and the regression loss). In the inference phase, the model will filter out high-confidence candidate boxes based on the confidence score of the classification branch, and combine the bounding box adjustment results of the regression branch to generate the final detection box. Finally, the non-maximum suppression (NMS) algorithm is used to remove redundant detection boxes, output the category label, precise 3D bounding box coordinates and confidence score of each target, and complete the classification, positioning and recognition of multimodal 3D targets.

[0059] Furthermore, in order to improve the performance of the non-maximum suppression (NMS) algorithm, especially in the multimodal 3D object detection task, this embodiment proposes a dynamic non-maximum suppression algorithm based on multimodal feature weights (Dynamic Multi-Modal Weighted NMS, DMW-NMS), which further optimizes the removal process of redundant frames by combining the confidence weights of multimodal features and the dynamic IoU threshold adjustment mechanism. The details are as follows:

[0060] In traditional NMS, the confidence of the candidate box usually comes from only a single branch (such as the confidence score of the classification branch). In this embodiment, the confidence of multimodal features (such as image features and LiDAR point cloud features) is weighted and fused to generate a comprehensive confidence score. By fusing multimodal confidence, the reliability of the candidate box can be more comprehensively evaluated to avoid misjudgment caused by insufficient confidence of a single modality.

[0061] The formula is: fused =α×S image +β×S LiDAR ; Among them, S image and S LiDAR They represent the confidence scores of the image branch and the LiDAR branch respectively, and α and β are dynamic weights, which are adjusted according to the modal importance of the current scene (for example, increasing the LiDAR weight in dense scenes and increasing the image weight in texture-rich scenes).

[0062] In traditional NMS, the IoU threshold is usually fixed (such as 0.5 or 0.7), which may lead to poor performance in different target density scenarios. In the steps of this embodiment, a dynamic IoU threshold adjustment mechanism is introduced to dynamically adjust the IoU threshold according to the comprehensive confidence of the candidate box and the target density:

[0063] The formula is: IoU dynamic =IoU base +γ×(1-S fused ), where IoU base is the basic IoU threshold, γ is the adjustment coefficient, S fused is the comprehensive confidence score. When the confidence is low, the IoU threshold is increased to reduce the retention of redundant boxes; when the confidence is high, the IoU threshold is reduced to allow more boxes to be retained to avoid missed detection.

[0064] When calculating the IoU between candidate boxes, in addition to the traditional geometric overlap, multimodal feature similarity is introduced as a weighting factor. By calculating the similarity of candidate boxes in the image feature space and the LiDAR feature space, the removal of redundant boxes is further optimized:

[0065] The formula is: IoU weighted =IoU×(1+λ×Sim multi );Sim multi It represents the similarity of the candidate boxes in the multimodal feature space (such as calculated by cosine similarity or Euclidean distance), and λ is the similarity weight. When the feature similarity of two candidate boxes is high, the weight of their IoU value is increased, and redundant boxes with similar features are removed first.

[0066] In areas with dense targets (such as crowded scenes), traditional NMS may mistakenly delete some valid frames. To this end, this embodiment adds a regional density perception mechanism to dynamically adjust the priority of the retained frames by counting the distribution density of candidate frames in the local area:

[0067] Count the number of candidate boxes within each candidate box's neighborhood (such as within a certain radius) and calculate the regional density.

[0068] For high-density areas, lower the confidence threshold to allow more boxes to be retained; for low-density areas, increase the confidence threshold to reduce redundant boxes.

[0069] Final frame screening and output.

[0070] Combining the above steps, sorting by fusion confidence score, combining dynamic IoU threshold and multimodal feature similarity weighting, the final detection box is gradually screened out. For each candidate box, the boxes with high confidence and large feature differences are retained first, and the redundant boxes with low confidence or similar features are removed. Finally, the optimized detection results are output, including target category, 3D bounding box coordinates and confidence score.

[0071] The present invention provides an embodiment. In order to compare and analyze the method of the present invention with other methods on the public data set KITTI, the results are submitted to the KITTI server for evaluation. Specifically, the method of the present invention is compared with the existing multimodal methods and LiDAR methods. The detection result of the method of the present invention at the Hard level reaches 79.47%. At the same time, on the KITTI verification set, the detection results of the method of the present invention at the Easy, Moderate, and Hard levels reach 95.83%, 89.10%, and 86.53%, respectively. The detection results have now exceeded most of the existing multimodal fusion methods and LiDAR methods.

[0072] Compared with the traditional multimodal 3D target detection method, the present invention mainly adopts point / voxel level feature fusion, which can capture the local detail information of the target at a finer granularity, but this method will significantly increase the amount of calculation. The number of points in the pseudo point cloud is extremely large, and point-by-point or voxel-by-voxel fusion calculation will lead to huge resource consumption, which is not suitable for real-time application scenarios. The present invention realizes self-attention RoI feature extraction (SARoIFE). In contrast, RoI fusion only operates on the region of interest, so it can focus on the details of the target area while reducing the overall computational workload, greatly improving the detection efficiency. The choice of RoI fusion can effectively alleviate the computational burden. At the same time, by reducing the processing resources of non-target areas, the system can focus the computing power on the real target area, thereby improving the real-time performance of the system. This strategy is particularly important in scenarios such as autonomous driving that have high requirements for response speed, and helps the system to detect targets quickly and accurately. The RoI fusion strategy can further improve the ability to capture details of the target area through the regional attention mechanism. Compared with the indiscriminate fusion at the point / voxel level, the regional attention mechanism can guide the model to pay more attention to the key details within the RoI, such as the boundary, contour and position features of the object. This strategy ensures that RoI fusion not only reduces the overall computational complexity, but also can capture more effective information in the key area and improve the detection accuracy. At the same time, the introduction of structural information in RoI level fusion can effectively enhance the integrity of the features, so that the features obtained by the system after RoI fusion have both spatial accuracy and structural integrity.

[0073] Secondly, many traditional multimodal 3D target detection methods only achieve one-way data fusion. They usually input image or LiDAR point cloud features into a fusion module respectively, and then generate the final detection result through simple feature-level or decision-level fusion. Although these methods can achieve information integration between modalities to a certain extent, they lack deep and two-way information interaction between multiple modalities. Therefore, although each modality can provide certain useful information, due to the lack of sufficient interaction and complementarity, the potential advantages between modalities cannot be maximized, thereby affecting the overall effect of target detection, especially in complex or dynamic environments. In the present invention, multimodal feature fusion (FBIF) with two-way information interaction is adopted. By integrating features of different modalities, the FBIF can achieve the best detection effect. Bidirectional information transmission is performed during the fusion process to enhance the complementarity between modalities. Specifically, image features and point cloud features are transmitted, updated and optimized to each other during the fusion process to better capture the spatial features and detail information of the target. Bidirectional information interaction allows images and point clouds to complement each other at different levels and stages, so that the strengths of each modality can make up for the shortcomings of the other modality. Images can provide rich texture, color and structural information, while point clouds can provide accurate spatial positioning and depth information. Through bidirectional interaction, image features can help point clouds better identify the boundaries and textures of targets, while point cloud features provide images with depth perception and object position support, greatly enhancing the complementarity between the two. This method significantly improves the efficiency and detection accuracy of multimodal fusion.

[0074] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0075] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multimodal three-dimensional object detection method based on attention mechanism, characterized in that: include: Obtain image data and LiDAR point cloud data and pre-process them; Perform feature extraction on the processed image data and LiDAR point cloud data; The extracted image features and point cloud features are fused using a multimodal feature fusion algorithm based on two-way information interaction; Optimize the fused features to eliminate redundancy and noise. At the same time, refine the fused features in a specific region of interest. The fused and optimized features are input into the target detection module, and the classification, positioning and recognition of multimodal 3D targets are completed through candidate box generation, classification and regression tasks. Based on the non-maximum suppression algorithm, redundant detection boxes are removed and the detection results are output, including the target's category label, bounding box coordinates, and confidence score.

2. The method according to claim 1, characterized in that The image data and LiDAR point cloud data are both obtained from the public dataset KITTI.

3. The method according to claim 1, characterized in that Preprocess the image data and LiDAR point cloud data, specifically: The image data is input into the depth completion module to convert the 2D image data into a pseudo point cloud to obtain the same 3D spatial information as the original point cloud; Perform point cloud filtering, coordinate conversion and voxelization on the LiDAR point cloud data to convert the LiDAR point cloud data into a standard format.

4. The method according to claim 1, characterized in that: Feature extraction is performed on the processed image data and LiDAR point cloud data, specifically: For image data, the key spatial and texture features are extracted through convolutional neural networks, and voxelization and sparseness are performed to obtain pseudo point cloud RoI features of image branches; For LiDAR point cloud data, the deep learning method Voxel-RCNN is used as the three-dimensional backbone network to process the LiDAR point cloud data, extract the spatial information, depth information and local structure features of the LiDAR point cloud data, and obtain the LiDAR RoI features of the point cloud branch. Based on the self-attention RoI feature extraction module, more effective information is captured in the key area to enrich the LiDAR RoI features.

5. The method according to claim 4, characterized in that The multimodal feature fusion algorithm based on two-way information interaction fuses the extracted image features and point cloud features, specifically: The extracted image features and point cloud features are fused. During the fusion process, the complementary information between the two modalities is aggregated based on the cross-attention mechanism to obtain the pseudo point cloud RoI features and LiDAR RoI features, and the features are bidirectionally interacted, namely L2P branch and P2L branch. Among them, the L2P branch uses LiDAR RoI features as queries and pseudo point cloud RoI features as keys and values. The pseudo point cloud RoI features are used to guide the pseudo point cloud RoI features, and the correlation between the two modalities is dynamically captured through a multi-head cross-attention mechanism; the P2L branch uses pseudo point cloud RoI features as queries and LiDAR RoI features as keys and values. The pseudo point cloud RoI features are used to guide the LiDAR RoI features. The pseudo point cloud guides the search for relevant information of the LiDAR point cloud, and the interactive features are obtained through a multi-head cross-attention mechanism.

Images