Multi-modal 3D target detection method based on false point cloud enhancement and gated attention fusion

Abstract

The invention discloses a multi-modal 3D target detection method based on false point cloud enhancement and gated attention fusion, and belongs to the technical field of three-dimensional target detection. Aiming at the defects of an existing method in the aspects of false point cloud feature modeling and cross-modal information fusion, systematic improvement is carried out from two levels of false point cloud feature modeling and cross-modal information fusion. On one hand, a false point cloud feature enhancement module is designed, geometric representation and semantic stability of a false point cloud are enhanced through context modeling of a multi-layer perceptron, and sparse 3D feature quality generated from an image is optimized. And on the other hand, a gating multi-head attention fusion module is innovatively provided, a cross-modal semantic dependency relationship between the false point cloud and the LiDAR point cloud is captured by using a multi-head attention mechanism, and the fusion strength is adaptively adjusted through a gating mechanism, so that refined multi-modal information interaction is realized. The method provided by the invention effectively solves the technical problem of low target detection precision of a long-distance or sparse region in the existing method.

Description

A Multimodal 3D Object Detection Method Based on Pseudo-point Cloud Augmentation and Gated Attention Fusion Technical Field

[0001] This invention belongs to the field of 3D target detection technology, specifically involving a multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion, which is particularly suitable for high-precision 3D perception and detection of multiple types of targets in road environments under complex traffic scenarios. Background Technology

[0002] Autonomous driving technology, as a core development direction for future transportation systems, has received widespread attention globally. Accurate and reliable 3D object detection, as one of the key technologies of autonomous driving perception systems, is directly related to vehicle safety and the reliability of intelligent decision-making.

[0003] Currently, 3D target detection in the field of autonomous driving mainly relies on the following methods: (1) Detection methods based on a single LiDAR point cloud: This type of method directly extracts features and detects targets from point cloud data. Although it performs well in close-range and dense point cloud areas, LiDAR point clouds are sparse in distant areas, resulting in insufficient geometric information for small targets and distant targets, and a significant decrease in detection accuracy.

[0004] (2) Detection methods based on single camera images: These methods use deep learning networks to predict 3D bounding boxes from 2D images. Although they can utilize the rich semantic information of the images, they are significantly lacking in distance estimation and 3D localization accuracy due to the lack of accurate 3D spatial information.

[0005] (3) Detection methods based on multimodal fusion: To overcome the limitations of a single sensor, multimodal fusion methods that combine the advantages of LiDAR point clouds and camera images have become the mainstream research direction. Among them, the method of converting camera images into pseudo-point clouds and then fusing them with LiDAR point clouds has received widespread attention. Using pseudo-point clouds as a sparse 3D representation of images can effectively supplement the perception capabilities of LiDAR in distant or sparse regions. However, existing pseudo-point cloud-based fusion methods still face key challenges: the feature representation capability of pseudo-point clouds is limited, and existing fusion methods are unable to fully exploit the complementary information between different modalities.

[0006] Therefore, this invention proposes a multimodal 3D target detection method that integrates pseudo-point cloud enhancement and gated attention. The pseudo-point cloud feature enhancement module improves the geometric representation capability and semantic stability of the pseudo-point cloud. At the same time, a gated multi-head attention fusion module is designed to achieve adaptive fusion intensity adjustment, giving full play to the complementary advantages of pseudo-point cloud and real point cloud, and effectively improving the target detection accuracy in long distances or sparse regions. Summary of the Invention

[0007] The present invention aims to solve at least one of the technical problems existing in the prior art: To this end, the present invention proposes a multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion, which is used to solve the technical problem that the insufficient pseudo-point cloud modeling and the coarse cross-modal fusion strategy in the existing methods result in low target detection accuracy in long distances or sparse regions.

[0008] To achieve the above objectives, this invention provides a multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion, comprising: S1: collecting LiDAR point cloud data and corresponding camera image data from several autonomous driving scenarios, preprocessing and labeling the point cloud data and image data to obtain a dataset; wherein, the dataset includes point cloud data, image data, and ground truth labels; S2: performing voxelization on the LiDAR point cloud data, extracting sparse 3D features through a compressed 2D backbone network, and generating candidate boxes and corresponding LiDAR RoI features by a region proposal network; S3: performing depth completion on the camera image data to convert the 2D image into a pseudo-point cloud representation; inputting the pseudo-point cloud into a pseudo-point cloud feature enhancement module, performing feature enhancement on the pseudo-point cloud through saliency modeling and density-aware mechanisms to obtain high-quality pseudo-point cloud RoI features; S4: processing the LiDAR point cloud data into a pseudo-point cloud representation; and preprocessing and labeling the point cloud data and image data to obtain high-quality pseudo-point cloud RoI features; RoI features and pseudo-point cloud RoI features are input into a gated multi-head attention fusion module. Feature fusion is performed through intra-modal enhancement and cross-modal attention mechanisms to obtain fused features. S5: The fused features are input into the detection head network, which outputs the category prediction and 3D bounding box parameters for each candidate box to obtain the detection results. The detection results are compared with the real labels through a loss function, and the network parameters are iteratively optimized based on the loss value to obtain the trained multimodal 3D object detection model.

[0009] By using a pseudo-point cloud feature enhancement module to perform deep context modeling on pseudo-point clouds, problems such as depth error, structural discontinuity, and semantic drift in pseudo-point clouds are effectively alleviated, improving the geometric representation ability and semantic stability of pseudo-point clouds. Through a gated multi-head attention fusion module, the cross-modal semantic dependencies between LiDAR point clouds and pseudo-point clouds are captured using a multi-head attention mechanism, and the fusion intensity is adaptively adjusted through a gating mechanism, realizing refined multimodal information interaction and effectively improving the target detection accuracy in long-distance or sparse regions.

[0010] Furthermore, the preprocessing and annotation of point cloud data and image data includes: S1-1: acquiring LiDAR point cloud data and camera image data collected in real urban road scenes; S1-2: preprocessing the acquired point cloud data and image data, including coordinate system alignment, timestamp synchronization and data format standardization.

[0011] S1-3: Use annotation tools to annotate the preprocessed data with 3D targets. The annotation information includes the target category, the center coordinates of the 3D bounding box, the size, and the rotation angle.

[0012] S1-4: Divide the labeled data into training and test sets in a 1:1 ratio, which are used for model training and performance evaluation, respectively.

[0013] Further, the LiDAR point cloud data is voxelized, feature extracted, and candidate box generated, including: S2-1: The original LiDAR point cloud data is voxelized, the three-dimensional space is divided into regular voxel grids, and the points in the point cloud are mapped to the corresponding voxel units to obtain a voxelized representation.

[0014] S2-2: The voxelized point cloud is encoded by a sparse convolutional network to extract local features at the voxel level; then a high compression operation is performed to compress the three-dimensional voxel features along the height dimension into a two-dimensional bird's-eye view representation.

[0015] S2-3: Utilize a 2D backbone network to extract multi-scale features from BEV features, obtaining multi-scale feature representations containing rich spatial information.

[0016] S2-4: Using an anchor-based mechanism, a number of candidate bounding boxes and their corresponding confidence scores are generated on multi-scale features through a Region Proposal Network (RPN); RoI pooling is performed on each candidate box to extract fixed-size region features from the multi-scale features, thus obtaining LiDAR RoI features.

[0017] Furthermore, the pseudo-point cloud feature enhancement module includes a feature encoder, a saliency weight generator, and a feature fusion unit.

[0018] The input features of each point in the pseudo-point cloud are represented as follows: .in It is a three-dimensional geometric feature. Image semantic features, For image projection features, For depth features, each pseudo-point cloud Have a neighborhood set , indicating the current pseudo-point cloud of The index of the neighboring points.

[0019] Furthermore, the feature encoder employs a dense perceptual network. Its calculation formula is ;in Indicates input features, For points, For the input feature dimension, These are the parameters for the first-level linear transformation. To output feature dimensions, for Activation function These are the parameters for the second-level linear transformation. For layer normalization operation, These are the encoded output features.

[0020] Furthermore, the saliency weight generator describes the geometric offset relationship by constructing local difference features, and its calculation formula is as follows: Formula ;in Representing the geometric and projection features of a pseudo-point cloud. This represents the corresponding characteristics of its neighboring points. This represents the local difference feature between two points. The number of neighboring points. Indicates the first The weight generation network of the layer adopts a dense perceptron network. structure, Indicates the first The significance weight of neighboring points to the center point in the layer.

[0021] Furthermore, the feature fusion unit performs weighted fusion of neighborhood encoded features, and its calculation formula is as follows: ;in For the first The output features of the layer (the original features after normalization in the first layer) Subsequent layers are the fused outputs of the previous layer. For the first Encoding features of points in the layer Encoding features of neighboring points, For the first Layer feature encoder, For significance weight, For the first Layer fusion feature encoder, This represents the original features after normalization. For splicing operations, Enhanced features for the final output.

[0022] The pseudo-point cloud feature enhancement module effectively addresses the weaknesses in geometric representation and semantic stability of pseudo-point clouds in traditional methods by employing deep contextual modeling with a multilayer perceptron and adaptively weighting neighborhood features using a saliency weighting mechanism. This module utilizes a multi-level feature extraction and fusion strategy to progressively enhance the local structure perception capability and global semantic consistency of the pseudo-point cloud, while preserving complete information from low-level details to high-level semantics through multi-scale feature concatenation. This hierarchical modeling and feature fusion mechanism significantly improves the feature quality of the pseudo-point cloud, providing a high-quality semantic and geometric foundation for subsequent cross-modal feature fusion, thereby enhancing the model's target detection accuracy in long-distance or sparse regions.

[0023] Furthermore, the gated multi-head attention fusion module includes an intra-modal enhancement branch and a cross-modal fusion branch.

[0024] Furthermore, the intramodal enhancement branch employs a multi-head attention mechanism, and its calculation formula is as follows: in As input features, For batch size, For the number of channels, The number of feature points, for Convolution operation, for Convolution operation, These are respectively query, key, and value representation. For normalization operations, In this embodiment, the number of attention heads is set to 4. For each dimension of attention head, The first Each attention head corresponds to a query, key, and value. express transpose, Scaling factor for Activation function For the first The output of each attention head, For splicing operations, This is a feature resulting from enhanced multi-head attention.

[0025] Furthermore, the cross-modal fusion branch employs a gating mechanism to control cross-modal interaction, and its calculation formula is as follows: .in The original input features, The weight parameters for the gated convolution are... This represents the convolution operation. for Activation function For gating mapping, These are features of the LiDAR branch after intramodal enhancement. These are features of the pseudo-point cloud branches after intramodal enhancement. and The gating weights are for LiDAR and pseudo point clouds, respectively. This represents element-wise multiplication. and These are the features after gated noise suppression. For splicing operations, Features after splicing To fuse the weight parameters of the convolution, For batch normalization operations, for Activation function This is the final fusion feature.

[0026] The gated multi-head attention fusion module achieves multimodal information interaction through intra-modal enhancement branches and cross-modal fusion branches. The intra-modal enhancement branch divides the channel dimension into multiple attention heads using a multi-head attention mechanism, performing global dependency modeling on LiDAR features and pseudo-point cloud features respectively, combined with local contextual information. The cross-modal fusion branch generates gated mappings through a gating mechanism, adaptively suppressing feature responses before fusion, and then concatenating the features of the two modalities along the channel dimension before fusion processing. This fully exploits the complementary information between LiDAR point clouds and pseudo-point clouds, significantly improving target detection accuracy in long-distance or sparse regions.

[0027] Furthermore, the loss function employs a multi-stage joint loss strategy, and its calculation formula is as follows:

[0028] in Generate a loss for the candidate boxes in the region proposal network. For the detection loss during the RoI refinement stage, For the auxiliary RoI loss of the point cloud branch, For the auxiliary RoI loss of the pseudo-point cloud branch, To deeply compensate for network losses, As the weighting coefficient, in this invention, we take... .

[0029] By introducing auxiliary RoI loss with point cloud and pseudo-point cloud branches, the gradient is effectively prevented from being dominated by a single branch. This encourages the network to fully utilize the complementary information of the two modalities during training, improving the overall performance and robustness of the model. Simultaneously, the inclusion of depth completion loss ensures the quality of the pseudo-point cloud, providing a reliable foundation for subsequent feature enhancement and multimodal fusion.

[0030] Compared with existing technologies, the beneficial effects of this invention are as follows: By introducing a pseudo-point cloud feature enhancement module and employing deep context modeling and saliency weighting mechanisms of multilayer perceptrons, the invention effectively alleviates the problems of depth estimation errors, structural discontinuities, and semantic drift in pseudo-point clouds, significantly improving the geometric representation ability and semantic stability of pseudo-point clouds, and providing a high-quality feature foundation for subsequent cross-modal feature fusion; by designing a gated multi-head attention fusion module, the invention captures the cross-modal semantic dependencies between LiDAR point clouds and pseudo-point clouds using a multi-head attention mechanism, and adaptively suppresses the feature responses of low-confidence or interfering regions through a gating mechanism, achieving refined multimodal information interaction and fully exploring the complementary advantages of the two modalities; finally, by combining a multi-stage joint loss strategy and introducing auxiliary RoI loss of point cloud branches and pseudo-point cloud branches as well as depth completion loss, the invention effectively prevents the gradient from being dominated by a single branch, ensuring the quality of pseudo-point clouds and effectively improving the target detection accuracy in long-distance or sparse regions. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 is a schematic diagram of the technical process of a multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion; Figure 2 is a schematic diagram of the structure of the pseudo-point cloud feature enhancement module provided by the present invention; Figure 3 is a schematic diagram of the structure of the gated multi-head attention fusion module provided by the present invention; Figure 4 is a schematic diagram of the structure of the multimodal 3D target detection model provided by the present invention. Detailed Implementation

[0033] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0034] Please refer to Figures 1-4. This embodiment of the invention provides a multimodal 3D object detection method based on pseudo-point cloud enhancement and gated attention fusion, including: S1: acquiring LiDAR point cloud data and corresponding camera image data of an autonomous driving scene, preprocessing, labeling, and segmenting the data to obtain training and testing datasets; S2: performing voxelization, feature extraction, and candidate box generation on the LiDAR point cloud data to obtain LiDAR RoI features; S3: performing depth completion on the camera image data to convert the two-dimensional image into a pseudo-point cloud representation, inputting the pseudo-point cloud into a pseudo-point cloud feature enhancement module for feature enhancement to obtain pseudo-point cloud RoI features; S4: inputting the LiDAR RoI features and pseudo-point cloud RoI features into a gated multi-head attention fusion module for feature fusion to obtain fused features; S5: inputting the fused features into a detection head network, outputting the category prediction and 3D bounding box parameters for each candidate box to obtain detection results, comparing the detection results with the real labels using a loss function, and iteratively optimizing the network parameters based on the loss value to obtain a trained multimodal 3D object detection model.

[0035] In this embodiment, LiDAR point cloud data and camera image data collected in a real urban road scene are first acquired. Before model training, the acquired point cloud data and image data are preprocessed, including coordinate system alignment, timestamp synchronization, and data format standardization. Next, a labeling tool is used to annotate the preprocessed data with 3D targets. The annotation information includes the target category, the center coordinates of the 3D bounding box, its size, and rotation angle. Category labels include cars, pedestrians, bicycles, etc. Then, the annotated data is divided according to a preset ratio, such as 1:1, to obtain training and test sets containing point cloud data, image data, and 3D annotation information.

[0036] Next, the training set is input into the constructed multimodal 3D object detection model for training. Specifically, the LiDAR point cloud data is first voxelized, dividing the 3D space into a regular voxel grid. Points in the point cloud are mapped to corresponding voxel units to obtain a voxel representation. Then, a sparse convolutional network is used to encode the voxelized point cloud, extracting voxel-level local features. Subsequently, a height compression operation is performed, compressing the 3D voxel features along the height dimension into a 2D bird's-eye view representation. Next, a 2D backbone network is used to extract multi-scale features from the bird's-eye view features to obtain multi-scale feature representations. Finally, a region proposal network using an anchor-based mechanism generates several candidate bounding boxes and corresponding confidence scores on the multi-scale features. RoI pooling is performed on each candidate box to extract fixed-size region features from the multi-scale features, obtaining LiDAR RoI features.

[0037] For camera image data, the two-dimensional image is first converted into a pseudo-point cloud representation using a depth completion network. The depth completion network estimates the depth value of each pixel using the camera image, and combines this with camera intrinsic parameters and pixel coordinates to project the image into three-dimensional space, generating a pseudo-point cloud. Each pseudo-point cloud contains three-dimensional coordinates, RGB color, image projection coordinates, and depth values, forming a 9-dimensional feature representation. The pseudo-point clouds are then input into a pseudo-point cloud feature enhancement module for feature enhancement.

[0038] The pseudo-point cloud feature enhancement module in this embodiment includes a feature encoder, a saliency weight generator, and a feature fusion unit, employing a multi-level structure to enhance the pseudo-point cloud features layer by layer; wherein, the input feature of each point in the pseudo-point cloud is represented as: .in It is a three-dimensional geometric feature. Image semantic features, For image projection features, For depth features, each pseudo-point cloud Have a neighborhood set , indicating the current pseudo-point cloud of The index of the neighboring points.

[0039] The feature encoder employs a dense perceptual network. Through formula Encode the input features; where Indicates input features, For points, For the input feature dimension, These are the parameters for the first-level linear transformation. To output feature dimensions, for Activation function These are the parameters for the second-level linear transformation. For layer normalization operation, These are the encoded output features.

[0040] Specifically, input features First of all With the first layer linear transformation parameters Multiply, we get Dimensional features, and then through The activation function performs a nonlinear transformation; then the activated features are compared with the parameters of the second-level linear transformation. Multiply, we get The features are then normalized using layer normalization to obtain the encoded features. .

[0041] The saliency weight generator describes the geometric offset relationship by constructing local difference features, using the formula... Generate significance weights; where Representing the geometric and projection features of a pseudo-point cloud. This represents the corresponding characteristics of its neighboring points. This represents the local difference feature between two points. The number of neighboring points. Indicates the first The weight generation network of the layer adopts a dense perceptron network. structure, Indicates the first The significance weight of neighboring points to the center point in the layer.

[0042] Specifically, in layer 1, the current pseudo-point cloud is first calculated. With each of its neighboring points Local difference features between ,get The differential features of the dimensionality are then input into the weight generation network. This network employs a dense sensing network. The structure will take the input 3D difference feature encoding is Significance weight of dimension The same local difference features are used in layers 2 and 3. Generate a network through weights respectively generate and Significance weight of dimension and .

[0043] The feature fusion unit employs a multi-level structure to weight and fuse neighborhood encoded features using a formula. The final enhanced features are obtained; where For the first The output features of the layer (the original features after normalization in the first layer) Subsequent layers are the fused outputs of the previous layer. For the first Encoding features of points in the layer Encoding features of neighboring points, For the first Layer feature encoder, For significance weight, For the first Layer fusion feature encoder, This represents the original features after normalization. For splicing operations, Enhanced features for the final output.

[0044] Specifically, in layer 1, the input features To obtain the first 6 dimensions of the normalized pseudo-point cloud features, the feature encoder first... The input features are encoded to obtain the center point encoded features. Then, the encoded features of neighboring points are obtained through indexing operations. Next, significance weights are used. We obtain the weighted sum of the neighborhood encoded features. The weighted features are calculated using the dimensional weighted features, where M is the number of neighborhood points; finally, the weighted features are input into the fusion feature encoder. ,Will 3D feature fusion Layer 1 fusion output of the dimension In layer 2, the output of layer 1 is merged. As input, through the feature encoder Encode to obtain Encode features in dimension 1; then use saliency weights. We obtain the weighted sum of the neighborhood encoded features. Weighted features of dimension; then fused through a feature encoder ,get Layer 2 fusion output of the dimension In layer 3, the output of layer 2 is merged. As input, through the feature encoder Encode to obtain Encode features in dimension 1; then use saliency weights. We obtain the weighted sum of the neighborhood encoded features. Weighted features of dimension; then fused through a feature encoder ,get Dimensional Layer 3 Fusion Output Finally, the three layers are merged and output. Compared with the original features after normalization The final enhanced features are obtained by concatenating the data along the channel dimension. .

[0045] Throughout the pseudo-point cloud feature enhancement process, layer-by-layer enhancement of pseudo-point cloud features is achieved through deep context modeling and saliency weighting mechanisms using a multilayer perceptron. This design enables the multimodal 3D object detection model of this invention to exhibit higher robustness and accuracy when processing pseudo-point clouds with depth estimation errors. Next, to integrate LiDAR RoI features and pseudo-point cloud RoI features, both are input into a gated multi-head attention fusion module for feature fusion. The gated multi-head attention fusion module includes an intra-modal enhancement branch and a cross-modal fusion branch.

[0046] Among them, the intramodal enhancement branch adopts a multi-head attention mechanism, through the formula Intramodal enhancement is performed; among which As input features, For batch size, For the number of channels, The number of feature points, for Convolution operation, for Convolution operation, These are respectively query, key, and value representation. For normalization operations, In this embodiment, the number of attention heads is set to 4. For each dimension of attention head, The first Each attention head corresponds to a query, key, and value. express transpose, Scaling factor for Activation function For the first The output of each attention head, For splicing operations, This is a feature resulting from enhanced multi-head attention.

[0047] Specifically, input features First through Convolution performs channel mapping, and then... Convolution introduces local context, generating queries separately. ,key Sum All dimensions Then on and Perform normalization; then change the channel dimension. Divided into There are 1 attention head, and each attention head has 1 dimension. For the first One point of attention, and Matrix multiplication yields The attention score matrix, divided by the scaling factor After passing Normalization, and then with Multiply, we get Finally, The outputs of each attention head are concatenated along the channel dimension to obtain 3D features, through Convolution mapping yields intra-modal enhanced features. .

[0048] The cross-modal fusion branch employs a gating mechanism, through formulas. Perform cross-modal fusion; among which The original input features, The weight parameters for the gated convolution are... This represents the convolution operation. for Activation function For gating mapping, These are features of the LiDAR branch after intramodal enhancement. These are features of the pseudo-point cloud branches after intramodal enhancement. and The gating weights are for LiDAR and pseudo point clouds, respectively. This represents element-wise multiplication. and These are the features after gated noise suppression. For splicing operations, Features after splicing To fuse the weight parameters of the convolution, For batch normalization operations, for Activation function This is the final fusion feature.

[0049] Specifically, firstly, the original input features By gated convolution ,go through Activation function generates gating maps Then, intramodal augmentation features of the LiDAR branch were analyzed. Intramodal augmentation features of pseudo-point cloud branches Apply the corresponding gating weights respectively and The gated and denoised features are obtained through element-wise multiplication. and Next, splice the data along the channel dimension. and ,get Finally, by merging convolutions... Normalization and The activation function is used to obtain the final fused features. .

[0050] Throughout the gated multi-head attention fusion process, the complementary information between LiDAR point clouds and pseudo point clouds was fully explored through intra-modal enhancement and cross-modal fusion. This enabled refined multimodal information interaction in spatial location and channel dimensions, effectively avoiding noise propagation and significantly improving the expressive power and robustness of the fused features.

[0051] Next, the fused features are input into the detection head network, which outputs the class prediction and 3D bounding box parameters for each candidate box, thus obtaining the detection results. The detection head includes a classification branch and a regression branch. The classification branch predicts the probability that each candidate box belongs to each class, while the regression branch predicts the refined parameters of the 3D bounding box, including center coordinate offset, size scaling, and rotation angle adjustment.

[0052] Finally, training the multimodal 3D object detection model also includes loss calculation and iterative optimization. Specifically, this is achieved through the formula... The final loss value is calculated, and then backpropagated back to the network to update the model's weights and bias parameters; where... Generate a loss for the candidate boxes in the region proposal network. For the detection loss during the RoI refinement stage, For the auxiliary RoI loss of the point cloud branch, For the auxiliary RoI loss of the pseudo-point cloud branch, To deeply compensate for network losses, For the weighting coefficient, in this embodiment, we take... .

[0053] Simultaneously, the training process also includes evaluating the detection results using real-world annotations to assess the accuracy of the current model's training. Specifically, this is done through the evaluation metric calculation formula. The model evaluation results were obtained; among them Represents average precision, calculated by different recall rates. The precision The average value is used to measure the overall detection performance of the model. This represents the number of sampling points; a higher value indicates better detection accuracy of the model. The intersection-union ratio (IUU) measures the predicted 3D bounding box. With the actual annotation box The overlap ratio between them This represents the volume of the intersection between the predicted bounding box and the ground truth bounding box. This represents the union volume of the predicted and ground truth bounding boxes. A value closer to 1 indicates a higher degree of overlap between the predicted and ground truth bounding boxes, meaning better detection performance. During evaluation, the average accuracy is calculated for different categories such as cars, pedestrians, and cyclists at three difficulty levels: easy, medium, and hard. The accuracy for the car category is... The threshold was set to 0.7, and the IoU threshold for pedestrians and cyclists was set to 0.5. Finally, the model weights with the highest evaluation accuracy during training were saved and used to input point cloud and image data of the autonomous driving scenario, outputting high-precision 3D object detection results. Some data in the above formulas are calculated by removing dimensions and retrieving numerical values. The formulas are derived from software simulations using a large amount of collected data, resulting in a formula that most closely approximates the real-world situation. The preset parameters and thresholds in the formulas were set by those skilled in the art based on actual conditions or obtained through simulations using a large amount of data.

[0054] The working principle of this invention is as follows: S1, Dataset preparation and preprocessing: By acquiring LiDAR point cloud data and camera image data collected in real urban road scenes, preprocessing operations such as coordinate system alignment and timestamp synchronization are performed. Three-dimensional target annotation is performed on the data using annotation tools to obtain training and testing sets containing point cloud data, image data, and 3D annotation information; S2, LiDAR branch processing: Voxelization, sparse convolutional coding, high compression, and multi-scale feature extraction are performed on the LiDAR point cloud data. Candidate boxes are generated through a region proposal network, and LiDAR RoI features are extracted; S3, Pseudo-point cloud branch processing and feature enhancement: Images are converted into pseudo-point clouds through a deep completion network. A pseudo-point cloud feature enhancement module is used to perform layer-by-layer feature enhancement on the pseudo-point clouds to obtain pseudo-point cloud RoI features; S4, Gated multi-head attention fusion: The LiDAR... RoI features and pseudo-point cloud RoI features are input into the gated multi-head attention fusion module. Feature fusion is performed through intra-modal enhancement branch and cross-modal fusion branch to obtain fused features. S5, detection and training: The fused features are input into the detection head to output the detection results. The loss value is calculated through a multi-stage joint loss function, and the model parameters are adjusted iteratively based on the loss value to finally obtain the trained multimodal 3D object detection model.

[0055] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion, characterized in that, include: S1: Collect LiDAR point cloud data and corresponding camera image data for several autonomous driving scenarios. Preprocess and label the point cloud data and image data to obtain a dataset; the dataset includes point cloud data, image data, and ground truth labels. S2: Perform voxelization on the LiDAR point cloud data, extract sparse 3D features using a compressed 2D backbone network, and generate candidate bounding boxes and corresponding LiDAR RoI features using a region proposal network. S3: Perform depth completion on the camera image data, converting the 2D image into a pseudo-point cloud representation. Input the pseudo-point cloud into a pseudo-point cloud feature enhancement module, and enhance the pseudo-point cloud features through saliency modeling and density-aware mechanisms to obtain high-quality pseudo-point cloud RoI features. S4: [The text abruptly ends here, likely due to an incomplete translation or missing information.] RoI features and pseudo-point cloud RoI features are input into a gated multi-head attention fusion module. Feature fusion is performed through intra-modal enhancement and cross-modal attention mechanisms to obtain fused features. S5: The fused features are input into the detection head network, which outputs the category prediction and 3D bounding box parameters for each candidate box to obtain the detection results. The detection results are compared with the real labels through a loss function, and the network parameters are iteratively optimized based on the loss value to obtain the trained multimodal 3D object detection model.

2. The multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion according to claim 1, characterized in that, The point cloud data and image data are preprocessed and labeled, including: S1-1: acquiring LiDAR point cloud data and camera image data collected in real urban road scenes; S1-2: preprocessing the acquired point cloud data and image data, including coordinate system alignment, timestamp synchronization, and data format standardization; S1-3: using a labeling tool to label the preprocessed data with 3D targets, the labeling information including target category, center coordinates of the 3D bounding box, size, and rotation angle; S1-4: dividing the labeled data into training set and test set at a 1:1 ratio, which are used for model training and performance evaluation, respectively.

3. The multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion according to claim 1, characterized in that, The LiDAR point cloud data is voxelized, feature extracted, and candidate bounding boxes generated, including: S2-1: The original LiDAR point cloud data is voxelized, dividing the three-dimensional space into a regular voxel grid, and mapping the points in the point cloud to the corresponding voxel units to obtain a voxelized representation; S2-2: The voxelized point cloud is encoded using a sparse convolutional network to extract local features at the voxel level; then, a height compression operation is performed to compress the three-dimensional voxel features along the height dimension into a two-dimensional bird's-eye view representation; S2-3: Multi-scale feature extraction is performed on the BEV features using a 2D backbone network to obtain a multi-scale feature representation containing rich spatial information; S2-4: Several candidate bounding boxes and corresponding confidence scores are generated on the multi-scale features using an anchor-based mechanism through a region proposal network; RoI pooling is performed on each candidate bounding box to extract fixed-size region features from the multi-scale features to obtain LiDAR RoI features.

4. The multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion according to claim 1, characterized in that, The pseudo-point cloud feature enhancement module includes a feature encoder, a saliency weight generator, and a feature fusion unit; wherein, the input features of each point in the pseudo-point cloud are represented as: ,in It is a three-dimensional geometric feature. Image semantic features, For image projection features, For depth features, each pseudo-point cloud Have a neighborhood set , indicating the current pseudo-point cloud of The index of the neighboring points.

5. The multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion according to claim 1, characterized in that, The feature encoder employs a dense sensing network. Its calculation formula is ;in Indicates input features, For points, For the input feature dimension, These are the parameters for the first-level linear transformation. To output feature dimensions, for Activation function These are the parameters for the second-level linear transformation. For layer normalization operation, These are the encoded output features.

6. The multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion according to claim 4, characterized in that, The saliency weight generator describes the geometric offset relationship by constructing local difference features, and its calculation formula is as follows: Formula ;in Representing the geometric and projection features of a pseudo-point cloud. This represents the corresponding characteristics of its neighboring points. This represents the local difference feature between two points. The number of neighboring points. Indicates the first The weight generation network of the layer adopts a dense perceptron network. structure, Indicates the first The significance weight of neighboring points to the center point in the layer.

7. A multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion according to claim 4, characterized in that, The feature fusion unit employs a multi-level structure to perform weighted fusion of neighborhood encoded features. Its calculation formula is as follows: ;in For the first The output features of the layer (the original features after normalization in the first layer) Subsequent layers are the fused outputs of the previous layer. For the first Encoding features of points in the layer Encoding features of neighboring points, For the first Layer feature encoder, For significance weight, For the first Layer fusion feature encoder, This represents the original features after normalization. For splicing operations, Enhanced features for the final output.

8. The multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion according to claim 1, characterized in that, The gated multi-head attention fusion module includes an intra-modal enhancement branch and a cross-modal fusion branch.

9. A multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion according to claim 8, characterized in that, The intramodal enhancement branch employs a multi-head attention mechanism, and its calculation formula is as follows: in As input features, For batch size, For the number of channels, The number of feature points, for Convolution operation, for Convolution operation, These represent queries, key representations, and value representations, respectively. For normalization operations, For the number of attention heads, For each dimension of attention head, The first Each attention head corresponds to a query, key, and value. express transpose, Scaling factor for Activation function For the first The output of each attention head, For splicing operations, This is a characteristic resulting from enhanced multi-head attention; The cross-modal fusion branch uses a gating mechanism to control cross-modal interactions, and its calculation formula is as follows: .in The original input features, The weight parameters for the gated convolution are... This represents the convolution operation. for Activation function For gating mapping, These are features of the LiDAR branch after intramodal enhancement. These are features of the pseudo-point cloud branches after intramodal enhancement. and The gating weights are for LiDAR and pseudo point clouds, respectively. This represents element-wise multiplication. and These are the features after gated noise suppression. For splicing operations, Features after splicing To fuse the weight parameters of the convolution, For batch normalization operations, for Activation function This is the final fusion feature.

10. A multimodal 3D target detection method based on pseudo-point cloud enhancement and gated attention fusion according to claim 1, characterized in that, The loss function employs a multi-stage joint loss strategy, and its calculation formula is as follows: in Generate a loss for the candidate boxes in the region proposal network. For the detection loss during the RoI refinement stage, For the auxiliary RoI loss of the point cloud branch, For the auxiliary RoI loss of the pseudo-point cloud branch, To deeply compensate for network losses, As the weighting coefficient, in this invention, we take... 。

Images