Multi-modal three-dimensional object detection method based on attention mechanism

Abstract

The application provides a multi-modal three-dimensional target detection method based on an attention mechanism, which comprises the following steps: acquiring image data and LiDAR point cloud data, and pre-processing the same; extracting features from the processed image data and LiDAR point cloud data; fusing the extracted image features and point cloud features based on a multi-modal feature fusion algorithm of bidirectional information interaction; optimizing the fused features to eliminate redundancy and noise; simultaneously, performing fine processing on the fused features in a specific region of interest; inputting the fused and optimized features into a target detection module; completing the classification, positioning and identification of multi-modal three-dimensional targets through a candidate box generation, classification and regression task; removing redundant detection boxes based on a non-maximum suppression algorithm; and outputting a detection result, including a class label, a bounding box coordinate and a confidence score of the target. The application improves the efficiency and detection accuracy of multi-modal fusion and is convenient to use.

Description

Technical Field

[0001] This invention relates to the field of multimodal three-dimensional target detection technology, and in particular to a multimodal three-dimensional target detection method based on an attention mechanism. Background Technology

[0002] With the rapid development of autonomous driving, intelligent transportation, and unmanned aerial vehicles, 3D object detection has become a crucial technical issue. 3D object detection technology analyzes depth information acquired by sensors and 2D image data to more accurately identify and locate objects, providing key support for autonomous driving decision-making systems. However, the limitations of single sensors pose many challenges to traditional 3D object detection methods, such as insufficient robustness in complex environments and recognition errors caused by target occlusion or sensor noise. Currently, sensors such as LiDAR and cameras have become mainstream 3D perception devices. LiDAR can provide high-precision depth information, suitable for large-scale, high-precision object detection, but it is relatively weak in processing detailed and color information. While traditional 2D image data can provide rich visual information, it has significant limitations in depth perception and spatial localization. Therefore, how to effectively integrate LiDAR and image data to overcome the limitations of single sensors has become an important research direction in the field of multimodal 3D object detection. In recent years, the application of deep learning technology, especially convolutional neural networks and attention mechanisms, has provided new ideas for multimodal fusion. By effectively combining data from different sensors through feature-level fusion, decision-level fusion, and other methods, it is expected to improve detection accuracy and robustness.

[0003] Existing multimodal fusion methods mainly take two forms: serial fusion and parallel fusion. Serial fusion uses image features from a specified region to enhance point cloud features. While this method can enrich point cloud representations, LiDAR signals are always affected by various conditions such as weather, lighting, and reflections. These methods are inherently overly reliant on LiDAR modes, failing to deeply explore the importance of images for target localization and lacking full utilization of the rich semantic information from camera sensors. Furthermore, they cannot produce any prediction results when the LiDAR signal malfunctions. Parallel fusion, while eliminating the dependence of current serial fusion methods on LiDAR input and possessing two modes to further improve detection accuracy, often relies on a single mode guiding another. Information can only be transferred from one mode to another, potentially limiting information flow. These methods do not consider the bidirectional interaction between the two modes to fully utilize complementary information. Therefore, the correlation between the geometric and depth information specific to sparse point cloud features and the dense semantic features of the image in current fusion models may be limited. Meanwhile, although existing multimodal 3D target detection algorithms solve the problem of lost 3D spatial information by converting 2D images into pseudo point clouds, the computational load is very large due to the large number of points in the pseudo point cloud, and the lack of rich structural information due to poor attention to spatial information features in the target area directly affects the understanding, recognition and localization of targets in the region of interest.

[0004] Therefore, it is essential to design a multimodal 3D object detection method based on an attention mechanism. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a multimodal three-dimensional target detection method based on an attention mechanism.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] This invention provides a multimodal 3D target detection method based on an attention mechanism, comprising:

[0008] Acquire image data and LiDAR point cloud data, and preprocess them;

[0009] Feature extraction is performed on the processed image data and LiDAR point cloud data;

[0010] A multimodal feature fusion algorithm based on two-way information interaction is used to fuse extracted image features and point cloud features;

[0011] The fused features are optimized to eliminate redundancy and noise. At the same time, the fused features are refined within specific regions of interest.

[0012] The fused and optimized features are input into the target detection module, and the classification, localization and recognition of multimodal 3D targets are completed through candidate box generation, classification and regression tasks.

[0013] Based on the nonmaximum suppression algorithm, redundant detection boxes are removed, and the detection results are output, including the target's category label, bounding box coordinates, and confidence score.

[0014] Preferably, both the image data and the LiDAR point cloud data are obtained from the publicly available dataset KITTI.

[0015] Preferably, the image data and LiDAR point cloud data are preprocessed, specifically as follows:

[0016] The image data is input into the depth completion module, which converts the 2D image data into a pseudo point cloud to obtain the same 3D spatial information as the original point cloud.

[0017] The LiDAR point cloud data is processed by point cloud filtering, coordinate transformation and voxelization to convert the LiDAR point cloud data into a standard format.

[0018] Preferably, feature extraction is performed on the processed image data and LiDAR point cloud data, specifically as follows:

[0019] For image data, key spatial and texture features are extracted using a convolutional neural network, and voxelization and sparsification are performed to obtain pseudo-point cloud RoI features of image branches.

[0020] For LiDAR point cloud data, Voxel-RCNN, a deep learning method, is used as the 3D backbone network to process the LiDAR point cloud data, extracting spatial information, depth information, and local structural features to obtain LiDAR RoI features of point cloud branches. Based on the self-attention RoI feature extraction module, more effective information is captured in key areas to enrich the LiDAR RoI features.

[0021] Preferably, a multimodal feature fusion algorithm based on bidirectional information interaction is used to fuse the extracted image features and point cloud features, specifically as follows:

[0022] The extracted image features and point cloud features are fused. During the fusion process, complementary information between the two modalities is aggregated based on a cross-attention mechanism to obtain pseudo-point cloud RoI features and LiDAR RoI features. These features are then subjected to bidirectional interaction, with L2P and P2L branches. In the L2P branch, LiDAR RoI features are used as the query and pseudo-point cloud RoI features are used as the key and value. The pseudo-point cloud RoI features guide the pseudo-point cloud RoI features, and the correlation between the two modalities is dynamically captured through a multi-head cross-attention mechanism. In the P2L branch, pseudo-point cloud RoI features are used as the query and LiDAR RoI features are used as the key and value. The pseudo-point cloud RoI features guide the LiDAR RoI features, and the pseudo-point cloud guides the search for relevant information in the LiDAR point cloud. The interactive features are obtained through a multi-head cross-attention mechanism.

[0023] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0024] This invention provides a multimodal 3D target detection method based on an attention mechanism. The method includes: acquiring image data and LiDAR point cloud data, preprocessing them, extracting features from the processed image data and LiDAR point cloud data, fusing the extracted image features and point cloud features based on a multimodal feature fusion algorithm with bidirectional information interaction, optimizing the fused features to eliminate redundancy and noise, and refining the fused features within a specific region of interest. The fused and optimized features are then input into a target detection module. Through candidate box generation, classification, and regression tasks, the classification, localization, and recognition of multimodal 3D targets are completed. Based on a nonmaximum suppression algorithm, redundant detection boxes are removed, and the detection results are output, including the target's category label, bounding box coordinates, and confidence score.

[0025] The present invention has the following advantages:

[0026] 1. This invention realizes self-attention RoI feature extraction (SARoIFE), which can focus on the details of the target region while reducing the overall computational load, thus greatly improving the detection efficiency;

[0027] 2. This invention employs bidirectional information interaction multimodal feature fusion (FBIF), which enhances the complementarity between modalities by transmitting information bidirectionally between different modal features during the fusion process. This allows image features and point cloud features to be mutually transmitted, updated, and optimized during the fusion process, thereby better capturing the spatial features and detailed information of the target and significantly improving the efficiency and detection accuracy of multimodal fusion. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart of the method provided in an embodiment of the present invention;

[0030] Figure 2 A flowchart illustrating a specific method provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] The purpose of this invention is to provide a multimodal 3D target detection method based on an attention mechanism, which can realize multimodal 3D target detection, improve the efficiency and detection accuracy of multimodal fusion, and is easy to use.

[0033] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Figure 1 This is a flowchart of a method provided in an embodiment of the present invention. Figure 2 A flowchart illustrating a specific method provided in an embodiment of the present invention is shown below. Figure 1 and Figure 2 As shown, this invention provides a multimodal 3D target detection method based on an attention mechanism, comprising:

[0035] Step 100: Acquire image data and LiDAR point cloud data, and preprocess them;

[0036] Step 200: Extract features from the processed image data and LiDAR point cloud data;

[0037] Step 300: A multimodal feature fusion algorithm based on bidirectional information interaction is used to fuse the extracted image features and point cloud features;

[0038] Step 400: Optimize the fused features to eliminate redundancy and noise. At the same time, refine the fused features within specific regions of interest.

[0039] Step 500: Input the fused and optimized features into the target detection module, and complete the classification, localization and recognition of multimodal 3D targets through candidate box generation, classification and regression tasks;

[0040] Step 600: Based on the nonmaximum suppression algorithm, remove redundant detection boxes and output the detection results, including the target's category label, bounding box coordinates, and confidence score.

[0041] In step 100, both the image data and the LiDAR point cloud data are obtained from the publicly available dataset KITTI.

[0042] In step 100, the image data and LiDAR point cloud data are preprocessed, specifically as follows:

[0043] The image data is input into the Depth Completion module, which converts the 2D image data into a pseudo point cloud to obtain the same 3D spatial information as the original point cloud.

[0044] The LiDAR point cloud data is processed by point cloud filtering, coordinate transformation and voxelization to convert the LiDAR point cloud data into a standard format, so that the point cloud data can be input into the subsequent network in a standard format.

[0045] In step 200, feature extraction is performed on the processed image data and LiDAR point cloud data, specifically as follows:

[0046] Feature extraction from images and point clouds are two separate branches;

[0047] For image data, key spatial and texture features are extracted using a convolutional neural network (Convs), and voxelization and sparsification are performed to obtain pseudo point cloud RoI features of image branches. These features mainly include the edges, shapes, and colors of objects.

[0048] For LiDAR point cloud data, Voxel-RCNN, a deep learning method, is used as the 3D backbone network to process the LiDAR point cloud data, extracting spatial information, depth information, and local structural features to obtain LiDAR RoI features of point cloud branches. Based on the self-attention RoI feature extraction module (SARoIFE), more effective information is captured in key areas to enrich the LiDAR RoI features.

[0049] In step 300, a multimodal feature fusion algorithm based on bidirectional information interaction fuses the extracted image features and point cloud features, specifically as follows:

[0050] The extracted image features and point cloud features are fused. During the fusion process, complementary information between the two modalities is aggregated based on a cross-attention mechanism to obtain pseudo-point cloud RoI features and LiDAR RoI features. These features are then subjected to bidirectional interaction, with L2P and P2L branches. In the L2P branch, LiDAR RoI features are used as the query and pseudo-point cloud RoI features are used as the key and value. The pseudo-point cloud RoI features guide the pseudo-point cloud RoI features, and the correlation between the two modalities is dynamically captured through a multi-head cross-attention mechanism. In the P2L branch, pseudo-point cloud RoI features are used as the query and LiDAR RoI features are used as the key and value. The pseudo-point cloud RoI features guide the LiDAR RoI features, and the pseudo-point cloud guides the search for relevant information in the LiDAR point cloud. The interactive features are obtained through a multi-head cross-attention mechanism.

[0051] In step 400, the fused features are optimized to eliminate redundancy and noise. Simultaneously, within specific regions of interest, the fused features are refined, specifically as follows:

[0052] Multilayer perceptron (MLP) or other feature optimization modules are used to optimize the fused features, eliminate redundancy and noise, and ensure that the fused features are more compact and effective. At the same time, the fused features are refined in specific regions of interest, and an attention mechanism is introduced to fuse the interacting features, thereby enhancing detection accuracy.

[0053] In step 500 of this embodiment, the fused and optimized features are input into the target detection module. Through candidate box generation, classification, and regression tasks, the specific implementation steps for classifying, locating, and recognizing multimodal 3D targets are as follows:

[0054] First, the fused and optimized multimodal features are input into the candidate bounding box generation module. This module generates initial candidate bounding boxes using a Region Proposal Network (RPN) or an anchor-based mechanism. Specifically, the RPN slides a window across the feature map, using predefined anchor boxes (boxes of different sizes and proportions) to scan for possible locations where the target might appear, and generates a set of candidate boxes. These candidate boxes undergo preliminary confidence evaluation to filter out regions that may contain the target, while simultaneously predicting the preliminary bounding box coordinates for each candidate box.

[0055] For the generated candidate boxes, their corresponding Region of Interest (RoI) features are further extracted. Through RoI pooling or RoI alignment (RoIAlign), the features of the candidate box regions are mapped to fixed-size feature representations. These feature representations are then fed into subsequent classification and regression networks for refinement. Simultaneously, a feature extraction module based on a self-attention mechanism captures more effective information in the key regions of the candidate boxes, further enriching their feature representations.

[0056] In classification tasks, the features of candidate boxes are input into the classification branch, which uses fully connected layers or convolutional layers to predict the target class of the candidate boxes. The classification branch outputs a probability distribution (Softmax output) of each candidate box belonging to different classes and determines the target class label of the candidate box based on the probability values. Simultaneously, the classification branch also outputs a confidence score to measure the model's confidence in the classification result of that candidate box.

[0057] In regression tasks, the features of candidate boxes are fed into the regression branch to predict their precise bounding box coordinates. The regression branch outputs the bounding box offset (i.e., the adjustment value relative to the initial candidate box) for each candidate box, including the target's 3D position (x, y, z), dimensions (width, height, depth), and orientation. These offsets are used to adjust the candidate boxes, generating more accurate target bounding boxes, thereby achieving precise target localization.

[0058] The classification and regression tasks are jointly optimized. The model is trained simultaneously on both the classification and regression branches using a multi-task loss function (such as a weighted sum of classification and regression losses). During the inference phase, the model selects high-confidence candidate boxes based on the confidence scores of the classification branch and adjusts the results by combining the bounding boxes from the regression branch to generate the final detection boxes. Finally, redundant detection boxes are removed using the non-maximum suppression (NMS) algorithm, and the model outputs the class label, precise 3D bounding box coordinates, and confidence score for each target, thus completing the classification, localization, and recognition of multimodal 3D targets.

[0059] Furthermore, to improve the performance of the non-maximum suppression (NMS) algorithm, especially in multimodal 3D object detection tasks, this embodiment proposes a dynamic multi-modal weighted NMS (DMW-NMS) algorithm based on multimodal feature weights. By combining the confidence weights of multimodal features and a dynamic IoU threshold adjustment mechanism, the redundant bounding box removal process is further optimized. Specifically:

[0060] In traditional NMS, the confidence score of candidate boxes is usually derived from a single branch (such as the confidence score of the classification branch). In this embodiment, the confidence scores of multimodal features (such as image features and LiDAR point cloud features) are weighted and fused to generate a comprehensive confidence score. By fusing multimodal confidence scores, the reliability of candidate boxes can be evaluated more comprehensively, avoiding misjudgments caused by insufficient confidence scores of a single modality.

[0061] The formula is: S fused =α×S image +β×S LiDAR Among them, S image and S LiDAR α and β represent the confidence scores of the image branch and the LiDAR branch, respectively. α and β are dynamic weights that are adjusted according to the modal importance of the current scene (e.g., increasing the LiDAR weight in dense scenes and increasing the image weight in textured scenes).

[0062] In traditional NMS, the IoU threshold is usually fixed (e.g., 0.5 or 0.7), which may lead to poor performance under different target densities. In this embodiment, a dynamic IoU threshold adjustment mechanism is introduced, dynamically adjusting the IoU threshold based on the overall confidence of the candidate boxes and the target density:

[0063] The formula is: IoU dynamic =IoU base +γ×(1-S fused ); where IoU base The base IoU threshold, γ is the adjustment coefficient, and S fused The overall confidence score is calculated. When the confidence score is low, the IoU threshold is increased to reduce the retention of redundant boxes; when the confidence score is high, the IoU threshold is decreased to allow more boxes to be retained to avoid missed detections.

[0064] When calculating the IoU between candidate boxes, in addition to the traditional geometric overlap, multimodal feature similarity is introduced as a weighting factor. By calculating the similarity of candidate boxes in the image feature space and the LiDAR feature space, the removal of redundant boxes is further optimized.

[0065] The formula is: IoU weighted =IoU×(1+λ×Sim) multi ); where Sim multi The similarity of candidate boxes in the multimodal feature space is represented by λ (e.g., calculated using cosine similarity or Euclidean distance), where λ is the similarity weight. When two candidate boxes have high feature similarity, their IoU value is weighted, prioritizing the removal of redundant boxes with similar features.

[0066] In areas with dense target density (such as crowded scenes), traditional NMS may mistakenly delete some valid bounding boxes. To address this, this embodiment incorporates a region density-aware mechanism, dynamically adjusting the priority of retained boxes by statistically analyzing the distribution density of candidate bounding boxes within local regions.

[0067] Count the number of candidate boxes within the neighborhood of each candidate box (e.g., within a certain radius) and calculate the region density.

[0068] For high-density regions, lower the confidence threshold to allow more boxes to be retained; for low-density regions, increase the confidence threshold to reduce redundant boxes.

[0069] Final box filtering and output.

[0070] Combining the above steps, the detection boxes are sorted according to their fusion confidence scores, and then weighted using a dynamic IoU threshold and multimodal feature similarity to progressively select the final detection boxes. For each candidate box, boxes with high confidence and significant feature differences are prioritized for retention, while low-confidence or redundant boxes with similar features are removed. The final optimized detection result is output, including the target category, 3D bounding box coordinates, and confidence score.

[0071] This invention provides an embodiment in which the method of this invention is compared and analyzed with other methods on the public dataset KITTI. The results are submitted to the KITTI server for evaluation. Specifically, the method of this invention is compared with existing multimodal methods and LiDAR methods. The method of this invention achieves a detection rate of 79.47% at the Hard level. At the same time, on the KITTI validation set, the detection results of the method of this invention reach 95.83%, 89.10%, and 86.53% at the Easy, Moderate, and Hard levels, respectively. The detection results now surpass most existing multimodal fusion methods and LiDAR methods.

[0072] Compared with traditional multimodal 3D target detection methods, this invention firstly uses point / voxel level feature fusion, which can capture local details of the target at a finer granularity. However, this method significantly increases the computational load. The number of points in the pseudo point cloud is extremely large, and point-by-point or voxel-by-voxel fusion calculation will lead to huge resource consumption, which is not suitable for real-time application scenarios. This invention implements Self-Attention RoI Feature Extraction (SARoIFE). In contrast, RoI fusion only operates on the region of interest, thus focusing on the details of the target region while reducing the overall computational load, greatly improving detection efficiency. The choice of RoI fusion can effectively alleviate the computational burden, and by reducing the processing resources for non-target regions, the system can concentrate its computing power on the real target region, thereby improving the real-time performance of the system. This strategy is particularly important in scenarios with high requirements for response speed, such as autonomous driving, helping the system to detect targets quickly and accurately. The RoI fusion strategy can further improve the ability to capture details of the target region through the region attention mechanism. Compared with point / voxel-level indiscriminate fusion, the region attention mechanism can guide the model to pay more attention to key details within the RoI, such as the boundaries, contours, and positional features of objects. This strategy ensures that RoI fusion not only reduces the overall computational complexity, but also captures more effective information in key regions, improving detection accuracy. At the same time, the introduction of structural information in RoI-level fusion can effectively enhance the integrity of features, so that the features obtained by the system after RoI fusion have both spatial accuracy and structural integrity.

[0073] Secondly, many traditional multimodal 3D object detection methods only achieve unidirectional data fusion. They typically input image or LiDAR point cloud features into a fusion module separately, then generate the final detection result through simple feature-level or decision-level fusion. While these methods can achieve information integration between modalities to some extent, they lack deep, bidirectional information interaction between modalities. Therefore, although each modality can provide some useful information, the potential advantages between modalities are not maximized due to insufficient interaction and complementarity, thus affecting the overall performance of object detection, especially in complex or dynamic environments. In this invention, a bidirectional information interaction multimodal feature fusion (FBIF) is employed, which integrates features from different modalities... Bidirectional information transfer is performed during the fusion process to enhance the complementarity between modalities. Specifically, image features and point cloud features are mutually transferred, updated, and optimized during the fusion process, thereby better capturing the spatial features and detailed information of the target. Bidirectional information interaction allows images and point clouds to complement each other at different levels and stages. In this way, the strengths of each modality can make up for the weaknesses of the other modality. Images can provide rich texture, color, and structural information, while point clouds can provide accurate spatial positioning and depth information. Through bidirectional interaction, image features can help point clouds better identify the boundaries and textures of targets, while point cloud features provide depth perception and object position support for images, greatly enhancing the complementarity between the two. This method significantly improves the efficiency and detection accuracy of multimodal fusion.

[0074] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0075] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multimodal 3D target detection method based on an attention mechanism, characterized in that, include: Acquire image data and LiDAR point cloud data, and preprocess them; Feature extraction is performed on the processed image data and LiDAR point cloud data; A multimodal feature fusion algorithm based on two-way information interaction is used to fuse extracted image features and point cloud features; The fused features are optimized to eliminate redundancy and noise. At the same time, the fused features are refined within specific regions of interest. The fused and optimized features are input into the target detection module, and the classification, localization and recognition of multimodal 3D targets are completed through candidate box generation, classification and regression tasks. Based on the nonmaximum suppression algorithm, redundant detection boxes are removed, and the detection results are output, including the target's category label, bounding box coordinates, and confidence score. Feature extraction is performed on the processed image data and LiDAR point cloud data, specifically as follows: For image data, key spatial and texture features are extracted using a convolutional neural network, and voxelization and sparsification are performed to obtain pseudo-point cloud RoI features of image branches. For LiDAR point cloud data, Voxel-RCNN, a deep learning method, is used as the 3D backbone network to process the LiDAR point cloud data, extracting spatial information, depth information and local structural features of the LiDAR point cloud data to obtain LiDAR RoI features of the point cloud branches. Based on the self-attention RoI feature extraction module, more effective information is captured in key areas to enrich the LiDAR RoI features. A multimodal feature fusion algorithm based on bidirectional information interaction fuses extracted image features and point cloud features, specifically as follows: The extracted image features and point cloud features are fused. During the fusion process, complementary information between the two modalities is aggregated based on a cross-attention mechanism to obtain pseudo-point cloud RoI features and LiDAR RoI features. These features are then subjected to bidirectional interaction, with L2P and P2L branches. In the L2P branch, LiDAR RoI features are used as the query and pseudo-point cloud RoI features are used as the key and value. The pseudo-point cloud RoI features guide the pseudo-point cloud RoI features, and the correlation between the two modalities is dynamically captured through a multi-head cross-attention mechanism. In the P2L branch, pseudo-point cloud RoI features are used as the query and LiDAR RoI features are used as the key and value. The pseudo-point cloud RoI features guide the LiDAR RoI features, and the pseudo-point cloud guides the search for relevant information in the LiDAR point cloud. The interactive features are obtained through a multi-head cross-attention mechanism.

2. The method according to claim 1, characterized in that, The image data and LiDAR point cloud data were obtained from the publicly available dataset KITTI.

3. The method according to claim 1, characterized in that, Preprocessing of image data and LiDAR point cloud data is performed as follows: The image data is input into the depth completion module, which converts the 2D image data into a pseudo point cloud to obtain the same 3D spatial information as the original point cloud. The LiDAR point cloud data is processed by point cloud filtering, coordinate transformation and voxelization to convert the LiDAR point cloud data into a standard format.

Images