A three-dimensional target detection model and detection method suitable for shielding and long-distance complex scenes

Abstract

The application relates to the technical field of computer vision, and particularly discloses a three-dimensional target detection model and a detection method suitable for occlusion and long-distance scenes, a three-dimensional target detection framework based on an attention mechanism is designed from the perspective of multi-modal perception feature enhancement and feature fusion, a multi-dimensional point cloud completion module and a depth image generation module are used in a bidirectional perception enhancement network, the multi-dimensional point cloud completion module uses image perception information to complete sparse targets and enhance contour information of point clouds, the depth image generation module uses a coordinate conversion relationship between sensors to enhance depth perception of image data, and effectively reduces feature differences between different modes; in the feature fusion network, the correlation and difference between the two modes are learned based on the attention mechanism, pixel-level image features of each voxel are adaptively aggregated, semantic information of the point cloud is enriched, and the detection precision of small targets and occluded targets is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a three-dimensional target detection model and method suitable for complex scenarios involving occlusion and long distances. Background Technology

[0002] 3D object detection is a key task in computer vision. Its main goal is to accurately locate and classify target objects in an input scene, generating bounding boxes with specific positions, sizes, and orientations to select and classify the targets. In recent years, due to the rapid development of fields such as autonomous driving, machine vision, and augmented reality, accurate and real-time 3D object detection has become crucial. Current 3D object detection methods can be divided into three categories: image-based, point cloud-based, and multimodal fusion-based algorithms.

[0003] Images can provide color and texture information, and image-based 2D object detection methods have achieved great success, proving that images can provide crucial semantic information, thus greatly improving object discrimination. However, images are not suitable for detection in 3D scenes because the lack of depth information makes it impossible to provide accurate coordinate positions. On the other hand, image quality is easily affected by the environment; for example, it is difficult to separate the target from the background in dark environments, and distant or occluded targets suffer from poor detection results due to depth blur.

[0004] Point clouds directly provide accurate location information of targets in three-dimensional space and have strong robustness, making them a preferred data type for detection in many algorithms. Current algorithms are categorized into voxel-based and point-based methods based on their processing of the raw point cloud data. Voxel-based target detection algorithms process the raw point cloud into regularly arranged voxels or grids, then use a 3D convolutional neural network for feature extraction and target bounding box prediction. This method improves detection speed, but voxelization leads to a loss of point cloud precision, reducing detection accuracy. Point-based target detection algorithms, on the other hand, directly extract features from the raw point cloud using a deep neural network to generate target bounding boxes, without any preprocessing of the input point cloud. While voxelization improves the average target detection accuracy compared to point cloud methods, it suffers from lower accuracy when detecting distant or occluded targets due to the sparsity of the point cloud itself, limiting the number of points within the target area. However, point clouds also have limitations. First, point clouds are sparse and disordered, making it difficult to extract structured features. Secondly, point clouds lack texture information, making it difficult to distinguish objects with similar shapes. These limitations restrict the accuracy of point cloud-based algorithms.

[0005] Multimodal fusion-based algorithms can combine data from multiple sensors, leveraging the complementary strengths of each data source to significantly improve detection accuracy. To address the challenges of detecting distant and occluded targets due to the sparsity of point clouds, researchers have explored various methods to locate or enhance object features. In terms of improving detection performance, some methods enhance the density and richness of point cloud data. For example, creating dense virtual points within a 3D truncated cone region and combining them with original LiDAR points can increase the measurement consistency of the point cloud between near and distant objects, thereby enhancing density differences between objects at different distances. This approach helps improve the sparsity problem of point clouds in distant target detection, enabling detection models to more accurately identify and locate distant objects. Furthermore, methods enhance the depth quality of the rising point at each interaction level through depth-aware design and modulate camera-modal-related voxels in a fine-grained manner, summarizing multimodal features into a unified space. This utilization of depth information and fine-grained LiDAR-camera interaction not only improves the detection model's ability to identify distant targets but also enhances its performance in detecting occluded targets. Despite these methods making some progress in improving the accuracy of 3D target detection, they still face the following challenges.

[0006] First, for sparse original point cloud data such as occluded targets and distant targets, the completed virtual point cloud has significant depth errors, while depth information is crucial for generating an effective virtual point cloud. Second, distant and occluded targets account for a smaller proportion of pixels in the image, resulting in sparser point cloud data. Simply fusing and interacting between two modalities using linear interpolation or matrix projection will lose the dense patterns of the image domain due to the sparsity of the fusion points, disrupting the semantic consistency of image features. Therefore, selectively combining image feature information with the point cloud is of paramount importance. Summary of the Invention

[0007] To address the shortcomings and problems of current image-based, point cloud-based, and multimodal fusion-based algorithms, this invention provides a 3D target detection model and method for occluded and distant scenes.

[0008] The solution adopted by this invention to solve its technical problem is: a three-dimensional target detection method suitable for occluded and long-distance scenes, comprising the following steps:

[0009] S1. Multi-dimensional point cloud completion: The original sparse point cloud data and multi-view image data lacking depth are completed in two dimensions and three dimensions to obtain the corresponding point cloud completion data. The three-dimensional point cloud completion data and the two-dimensional point cloud completion data are stitched together to generate dense point cloud data.

[0010] S2. Depth Image Generation: The dense point cloud data is projected onto 6 image planes using the camera's internal and external parameters to obtain point cloud depth features. RGB data is selectively combined to generate image semantic features. The point cloud depth features and image semantic features are then fused to obtain the fused image data.

[0011] S3, Multimodal Feature Extraction Module: Employs a 2D image feature extraction algorithm to extract multi-scale image features from the fused image data, and employs a 3D point cloud feature extraction algorithm to extract voxel features from dense point cloud data;

[0012] S4. Multimodal feature alignment and fusion: Multi-scale image features and voxel features are aligned to obtain voxel image features and voxel point cloud features. The voxel image features and voxel point cloud features are fused to obtain output features, thereby completing target detection.

[0013] The above-mentioned 3D target detection method applicable to occluded and distant scenes includes the following steps in step S1: Multi-dimensional point cloud completion.

[0014] (1) Obtain the original sparse point cloud data P and the corresponding multi-view image data I lacking depth. Process the image data I based on the semantic segmentation network to obtain the instance segmentation mask M related to the detection target. m , The input image is processed based on an object detection network to obtain n instance segmentation sub-masks M for each category instance segmentation mask. mn ,

[0015] (2) Project the point cloud data P onto the instance segmentation sub-mask region M according to the coordinate transformation formula from LiDAR to RGB sensor. mn , Obtain point cloud data P within the segmented region of i*j instances. R ,

[0016]

[0017] In the formula: t1 and t2 are the LiDAR measurement and RGB image capture times, respectively, and T (camera←lidar) It is the transformation matrix from the lidar sensor to the camera coordinate system. It is the transformation matrix of the camera coordinate system between t2 and t1, T (rgb←camera) It is the transformation matrix from the camera coordinate system to the pixel coordinate system; N R x represents the point cloud data within the masked area. i y i , z i Represents the 3D coordinates of the point cloud, used for 3D completion based on voxel lattice sampling; ui and v i It is a pixel coordinate, d i It represents the true depth, used for two-dimensional completion based on mask region sampling;

[0018] (3) The DBSCAN algorithm is used to cluster the point cloud data to obtain new sub-mask regions M′. mn and the point cloud cluster P′ to be completed R ,

[0019]

[0020] (4) 3D point cloud completion of point cloud clusters: The point cloud data is voxelized, and μ non-repeating 3D points are randomly sampled within each non-empty voxel block. For each sampling point F0, the feature r0 is obtained by retrieving the nearest neighbor point cloud from the point cloud data within the acquired voxel grid.

[0021]

[0022] In the formula: P′ M This represents point cloud data within a non-empty voxel lattice. This indicates that the k-nearest neighbor (K-NN) algorithm was used to detect the three-dimensional coordinates P of the nearest neighbor point cloud of the sampling point F0 within the voxel lattice. 3d and the coordinates of the voxel center P c ;

[0023] Based on the distance function, the distance errors between the point and its nearest neighbor, as well as the voxel center, are calculated. An optimization algorithm to minimize these errors is used to adjust the coordinates of the generated points, and this process is iterated multiple times to reach a set minimum value. Sub-masks M are then removed through mapping. mn To avoid the completed point cloud falling outside the masked area, the optimized and corrected 3D completed point cloud data F′ is used.

[0024]

[0025] In the formula: x′ i y′ i , z′ i This indicates the three-dimensional coordinates of the sampling points after correction, which better match the original data.

[0026] (5) Two-dimensional point cloud completion for point cloud clusters: In the submask region M nm Randomly sample τ unique 2D points. For each sampling point S0, from the acquired point cloud data P′ R Depth features d0 are obtained by retrieving the nearest neighbor point cloud from F′.

[0027]

[0028] In the formula: V′ M This represents virtual point cloud data that has been completed in three dimensions. This indicates that the nearest neighbor point cloud depth data of the retrieved sampling point S0 was detected using the k-nearest neighbor algorithm;

[0029] (6) Based on the depth information and two-dimensional coordinate information, generate the true three-dimensional spatial coordinate information S′ based on the transposed coordinate matrix.

[0030] (7) By stitching together the 3D point cloud completion data and the 2D point cloud completion data, the obtained point cloud data is combined with the original point cloud data to generate dense point cloud data P. virtual ,

[0031]

[0032] P virtual ∈R (N×C)

[0033] Where: N v This represents the original point cloud data plus the number of virtual point clouds generated.

[0034] The above-described 3D target detection method applicable to occluded and distant scenes includes the following steps in step S2: Depth image generation:

[0035] (1) Using the camera's internal and external parameters to extract dense point cloud data P virtual Projecting the data onto six image planes yields point cloud depth information I. depth ,

[0036]

[0037] Where: N C The number of cameras is represented by D, H, and W, which represent the depth, height, and width of the point cloud image obtained after projection onto the camera perspective, respectively; i and v represent the pixel coordinates within the projected depth image, and d corresponds to the depth value associated with the pixel at position (u, v); T (camera←lidar) It is the transformation matrix from the lidar sensor to the camera coordinate system; It is the transformation matrix of the camera coordinate system between t2 and t1; T (rgb←camera) It is the transformation moment from the camera coordinate system to the pixel coordinate system; X LiDAR Y LiDAR and Z LiDAR Represents the location information of point cloud data; * represents the prime product of two matrices;

[0038] (2) Fill the global point cloud:

[0039] First, for each target pixel q in the dense mapping, find its k nearest neighbors on the projection point of the image plane. Use a lightweight MLP and a sigmoid activation function to weight the adjacent features according to the coordinate offset to give a score in the range (0, 1). Then, use the adjacent features to perform a symmetric operation to aggregate them based on the weighted scores to obtain the aggregated features.

[0040]

[0041] In the formula: D(q) represents the depth feature of the hole pixel q, N q G represents all neighboring points. i It represents the 3D feature of point i, and MaxPool represents the max pooling operation x. i Represents the two-dimensional image coordinate information of neighboring points;

[0042] Then, a dilation operation is performed to expand the target region in the image. At the same time, a cross kernel is used as the structuring element of the dilation operation to help highlight the target edge during the dilation process. A blur operation is introduced to blur the image to reduce noise and smooth the region to improve the overall image continuity.

[0043] Finally, a 1×1 convolution operation is performed to adjust the alignment and obtain the densely mapped point cloud depth image data I′. depth , Make it have the same features as the input image They have the same shape, meaning D and C have the same number of channels;

[0044] (3) In the mapped point cloud depth image data I′ depth After the input image data I is processed by a ResNet network convolution operation, the image depth features F are obtained. 3D , With image semantic features F 2D , C 2D This represents the number of feature channels after convolution.

[0045] The above-mentioned 3D target detection method applicable to occluded and distant scenes, in step (3), introduces an adaptive feature fusion method based on channel attention to fuse image depth features and semantic features, including the following steps:

[0046] First, the input features F are processed using AvgPool global average pooling. 3D , Generate channel statistics C S ,

[0047]

[0048] Then, the dimension of S is further increased from C through a fully connected layer. 2D Reduce to C 2D / r to create a compact feature C' S This enables more precise and adaptive selection guidance, followed by the use of compact feature C' S Predicted attention score A 2d ,

[0049]

[0050] Through the compact feature descriptor C' S Guided cross-channel soft attention is used to adaptively select different information space scales, and the softmax operator is applied at the channel level to ensure that A holds for any i∈{1,2,...,C}. i，0 +A i，1 =1

[0051]

[0052] In the formula: F 2D and F′ 3D Soft attention vector, i represents channel C 2D The i-th element in; the perceptually enhanced feature image data F^ is derived from F 2D and F′ 3D The weighted sum is obtained,

[0053] The above-described 3D target detection method, applicable to occluded and distant scenes, uses the YOLOv7 2D image feature extraction algorithm in step S3 to extract multi-scale image features F from the perceptually enhanced feature image data F^. R ; and feature extraction algorithms for 3D point clouds based on SECOND and Centerpoint from perceptually enhanced dense point cloud data P virtual Extracting voxel features F P ,

[0054] F R ={f1, f2, ..., f hw},

[0055]

[0056] F P ={p1, p2, ..., p j},

[0057]

[0058] In the formula: f1 represents the image feature at the i-th spatial location, p j This represents the feature of each non-empty voxel extracted from the original point cloud.

[0059] The above-described 3D target detection method applicable to occluded and distant scenes includes the following steps in step S4:

[0060] (1) Based on the obtained image features F R and voxel characteristics F P Each non-empty voxel feature p j As the query, the corresponding image plane reference point P is then calculated using the projected coordinate matrices of the camera and the LiDAR. k As a key, P k =(p jh p jw Furthermore, the image domain features F can be obtained through bilinear interpolation. k Each query obtains the sampling offsets (ΔP) through a Linear. mqk This function is used to query the surrounding location information of a reference point on the Query image plane. An attention score A can be obtained on the aggregated image features using a Linear and a Softmax algorithm. mqk The final deformable cross-attention alignment feature can be obtained through formula (8).

[0061]

[0062] In the formula: W m and W′ m The weights represent the learnable weights; k refers to the weighted average of the image features of each non-empty voxel Query and the k image features surrounding its corresponding image domain reference point; m refers to the m-th head of the multi-head attention.

[0063] (2) Connect the features from each network and use convolutional layers to reduce the number of channels of the fused features to obtain voxel image features F'. R and voxel point cloud features F p The fusion characteristics

[0064] F f =f([F' R ;F p ])

[0065]

[0066] In the formula: F' R F represents the features of a voxel image. prepresents the voxel point cloud features, [·; ·] represents the concatenation operation; f() represents the convolution and activation function processing of the fused features to extract a higher level and more representative representation;

[0067] (3) The fused features are used as a supplement to the original LiDAR features, and the output features F are obtained by fusing them based on the residual structure and combined with the channel attention module. out ,

[0068]

[0069] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))

[0070] Where: M c The convolution operation in the channel attention module of the fusion module is represented by MLP; MLP represents fully connected; Avgpool represents average pooling; Maxpool represents max pooling; σ(x) = 1 / (1+e -x ) represents the sigmoid function.

[0071] This invention also provides a 3D target detection model suitable for occluded and long-distance scenes. The model includes a bidirectional perception enhancement network, a multimodal feature extraction module, and a multimodal feature alignment and fusion module. The bidirectional perception enhancement network processes raw sparse point cloud data and image data lacking depth to obtain dense point cloud data and image data with depth. The multimodal feature extraction module extracts voxel features from the perception-enhanced dense point cloud data and extracts multi-scale image features from the perception-enhanced image data. The multimodal feature alignment and fusion module uses a deformable cross-attention network module to automatically align non-homogeneous features obtained by the multimodal feature extraction module in a dynamic and data-driven manner, and then fuses the aligned features to obtain fused features, thereby completing target detection.

[0072] The aforementioned 3D target detection model applicable to occluded and long-distance scenes includes a bidirectional perception enhancement network comprising a multi-dimensional point cloud completion module and a depth image generation module. The multi-dimensional point cloud completion module performs 2D and 3D completion on the original sparse point cloud data and multi-view image data lacking depth, and stitches the completed data to generate dense point cloud data. The depth image generation module uses the camera's internal and external parameters to project the dense point cloud data onto six image planes to obtain point cloud depth features, and selectively combines RGB data to generate image semantic features. The point cloud depth features and image semantic features are then fused to obtain fused image data.

[0073] Compared with the prior art, the beneficial effects of the present invention are:

[0074] 1. This invention proposes a novel multimodal 3D target detection module suitable for 3D target detection tasks in complex scenes with occlusion and long distances. It overcomes the shortcomings of current mainstream algorithms, improves detection accuracy, and solves the problem of missed or false detections of small, distant, and occluded targets in complex scenes due to the sparsity and disorder of point clouds. Through various ingenious modules and strategies, the MSPE-Fusion framework is extended to enable 3D target detection on multimodal data with sparse point clouds, blurred images, and uneven brightness in urban roads and workshop engineering scenes, improving the detection accuracy and robustness of occluded and long-distance targets.

[0075] 2. The method of the present invention completes point cloud and image data from two dimensions (two-dimensional and three-dimensional) to obtain dense point cloud data, effectively solving the problem of low detection accuracy when sparse point cloud features are fused into dense image features.

[0076] 3. The method of the present invention introduces a learnable interpolation module to fill the global point cloud during the depth image generation process, which can better utilize the depth features of the point cloud, facilitate the fusion of subsequent multimodal features, and improve detection accuracy.

[0077] 4. This invention uses a multi-dimensional point cloud completion module to complete sparse targets by utilizing the perceptual information of the image, thereby enhancing the contour information of the point cloud. The depth image generation module uses the coordinate transformation relationship between sensors to enhance the depth perception of image data, effectively reducing the feature differences between different modalities.

[0078] 5. The method of the present invention uses a residual fusion network based on channel attention when performing feature fusion, which can effectively solve the problem of a large amount of redundant information in the image features extracted by the extraction module and improve the perception ability of different targets. Attached Figure Description

[0079] Figure 1 This is a general framework diagram of the MSPE-Fusion 3D target detection model technical solution.

[0080] Figure 2 Flowchart of MSPE-Fusion multi-dimensional point cloud completion strategy.

[0081] Figure 3 Flowchart of MSPE-Fusion depth image generation strategy.

[0082] Figure 4 This is a detailed flowchart of the learnable interpolation method.

[0083] Figure 5 This is a diagram of the network structure for feature alignment and fusion based on the attention mechanism. Detailed Implementation

[0084] This invention provides a 3D target detection model with multi-sensor perception enhancement and collaborative fusion suitable for occluded and long-distance scenarios. The framework includes a multi-dimensional point cloud completion module, a depth image generation module, a multi-modal feature extraction module, and a multi-modal feature alignment and fusion module. This model overcomes the shortcomings of current mainstream algorithms, improves detection accuracy, and solves the problem of missed or false detections of small, distant, and occluded targets in complex scenes due to the sparsity and disorder of point clouds. The invention is further described below with reference to the accompanying drawings and embodiments.

[0085] Example 1: This example provides a 3D object detection method for occluded and long-distance scenes. From the perspective of multimodal perception feature enhancement and feature fusion, an attention-based 3D object detection framework, MSPE-Fusion, is designed, such as... Figure 1 As shown, this is used to improve the accuracy of 3D object detection in occluded and long-distance scenes. The 3D object detection framework MSPE-Fusion includes a bidirectional perception enhancement network, a multimodal feature extraction module, and a multimodal feature alignment and fusion module.

[0086] First, a bidirectional perceptual enhancement network is used to enhance the original sparse point cloud data and the image data lacking depth. This network achieves multimodal perceptual feature enhancement through two modules: a multi-dimensional point cloud completion module (MPCM) and a depth image generation module (DMGM). The MPCM takes the original sparse point cloud data and the corresponding image data lacking depth as input. This module performs 3D augmentation on the original point cloud at the voxel level and preserves the contour information of the target based on image segmentation data, upscaling 2D image pixels into 3D space. It also explores multiple depths with reference to the neighborhood to generate virtual points with more reliable depth. The depth image generation module adaptively augments the sparse 2D point cloud map to generate a dense 2D point cloud map under the guidance of image semantic features, and selectively combines RGB data to generate semantic feature information containing depth perception, effectively achieving feature separation between foreground and background. Then, a feature extraction module is used to extract convolutional features from the perceptually enhanced point cloud and image features respectively. To address the issue of inaccurate multimodal feature fusion, a Feature Alignment and Fusion (FAFM) module based on an attention mechanism was designed. This module dynamically focuses on pixel-level features in the image, adaptively aggregates the features of each voxel, queries the image features around the target pixel to obtain a pixel-by-pixel semantic feature map, aggregates the image features based on the feature map, and learns the differences between the two modalities based on channel attention contrast learning to achieve effective feature fusion, enrich the semantic information of the point cloud, and improve the detection accuracy of distant small targets and occluded targets.

[0087] The method includes the following steps:

[0088] S1, Multi-dimensional point cloud completion, strategy process as follows: Figure 2 As shown. Acquire sparse point cloud data P and multi-view image data I.

[0089]

[0090] Where: N p N represents the number of point clouds. c x represents the number of camera viewpoints. i y i , z i The coordinates of the point cloud are represented by H, W, and C, which represent the height and width of the image and the RGB image features, respectively.

[0091] Sparse point cloud data P and multi-view image data I are input into a multi-dimensional point cloud completion module. The multi-dimensional point cloud completion module processes the image data I based on a semantic segmentation network (Deeplabev3+) to obtain an instance segmentation mask M related to the detected target. m , The input image is processed using a target detection network (CenterNet2) covering m target categories, including pedestrians, cars, and bicycles, plus a background. This process yields n instance segmentation sub-masks M for each category's instance segmentation mask. mn , To avoid repeatedly completing overlapping areas, detection results with non-zero IoU are merged into one region based on the target detection coordinates.

[0092] Next, the point cloud data P is projected onto the instance segmentation sub-mask region M according to the coordinate transformation formula (1) from laser radar to RGB sensor. mn , Obtain point cloud data P within the segmented region of i*j instances. R ,

[0093]

[0094] In the formula: t1 and t2 are the LiDAR measurement time and the RGB image capture time, respectively. It is the transformation matrix of the camera coordinate system between t2 and t1, T (camera←lidar) It is the transformation matrix from the LiDAR sensor to the camera coordinate system, T (rgb←camera) It is the transformation matrix from the camera coordinate system to the pixel coordinate system; N R x represents the number of point clouds within the masked region. i y i , z i Represents the 3D coordinates of the point cloud, used for 3D completion based on voxel lattice sampling; u i and v iIt is a pixel coordinate, d i It represents the true depth, used for 2D completion based on mask region sampling.

[0095] By analyzing point cloud data P R The selection and density assessment are used to avoid generating a large amount of noisy data and to retain sparse regions with fewer point clouds.

[0096] In addition, the DBSCAN algorithm is used to cluster point cloud data to identify and classify multiple targets in the scene, obtaining new sub-mask regions M′. mn and the point cloud cluster P′ to be completed R , N' R This represents the new submask region M′ mn Number of inner point clouds.

[0097] To achieve 3D point cloud completion for point cloud clusters, the point cloud data was voxelized. The voxel grid size was set to [0.05, 0.05, 0.1] for the Kitti dataset and [0.075, 0.075, 0.2] for the Nuscenes dataset. Within each non-empty voxel block, μ unique 3D points were randomly sampled. For each sampling point F0, the feature r0 is obtained by retrieving the nearest neighbor point cloud from the point cloud data within the acquired voxel grid.

[0098]

[0099] In the formula: P' M This represents point cloud data within a non-empty voxel lattice. This indicates that the k-nearest neighbor (K-NN) algorithm was used to detect the three-dimensional coordinates P of the nearest neighbor point cloud of the sampling point F0 within the voxel lattice. 3d and the coordinates of the voxel center P c .

[0100] Next, based on the distance function, the distance error between the point and its nearest neighbor and the voxel center is calculated. The coordinates of the generated point are adjusted using an optimization algorithm that minimizes the error, and the process is repeated multiple times to reach the set minimum value.

[0101] Finally, the submask M is removed by mapping. mn To avoid the completed point cloud falling outside the masked area, the optimized and corrected 3D completed point cloud data F′ is obtained.

[0102]

[0103] In the formula: x′ i y′ i , z′i This indicates the three-dimensional coordinates of the sampling points after correction, which better match the original data.

[0104] To achieve 2D point cloud completion for point cloud clusters, firstly in the sub-mask region M nm Randomly sample τ unique 2D points. For each sampling point S0, from the acquired point cloud data P′ R Depth features d0 are obtained by retrieving the nearest neighbor point cloud from F′.

[0105]

[0106] In the formula: V' M This represents virtual point cloud data that has been completed in three dimensions. This indicates that the k-nearest neighbor (K-NN) algorithm is used to detect the nearest neighbor point cloud depth data of the retrieved sampling point S0;

[0107] Based on depth information and two-dimensional coordinate information, the true three-dimensional spatial coordinate information S′ is generated based on the transposed coordinate matrix. By combining the point cloud data obtained from the two completion methods with the original point cloud data, dense point cloud data P is generated. virtual ,

[0108]

[0109] Where: N v This represents the original point cloud data plus the number of virtual point clouds generated.

[0110] S2, Depth Image Generation, strategy flow see Figure 3 Using the camera's internal and external parameters according to formula (7), the dense point cloud data P virtual Projected onto 6 image planes, the mapped point cloud depth image data I is obtained. depth ,

[0111]

[0112] Where u and v represent the pixel coordinates within the projected depth image, corresponding to the image's height and width, respectively, and d represents the depth value associated with the pixel at position (u, v), corresponding to T. (camera←lidar) It is the transformation matrix from the lidar sensor to the camera coordinate system. It is the transformation matrix of the camera coordinate system between t2 and t1, T (rgb←camera) X is the transformation moment from the camera coordinate system to the pixel coordinate system. LiDAR Y LiDAR and Z LiDAR This represents the location information of point cloud data; * indicates the prime product of two matrices; N CThe number of cameras is indicated by H and W, which represent the depth, height, and width of the point cloud image that can be obtained after projection onto the camera perspective, respectively. D represents the depth feature.

[0113] Due to the sparse nature of point clouds, most of the information in the image will be lost due to holes caused by projection, and the lack of global information for local targets will hinder the fusion of multimodal features. In order to better utilize the depth features of point clouds to achieve enhanced spatial fusion, a learnable interpolation module is introduced (see...). Figure 4 It is used to fill the global point cloud, solving the problem of fusing sparse point features into dense image features.

[0114] Specifically: the target pixel q is I depth For each target pixel q, the k nearest neighbors of the projection point on the image plane are found. A lightweight MLP and a sigmoid activation function are used to weight the adjacent features according to the coordinate offset to give a score in the range of (0,1). The adjacent features are then aggregated using the weighted score using formula (8) to obtain the depth features.

[0115]

[0116] In the formula: D(q) represents the depth feature of the hole pixel q, N q G represents all neighboring points. i Let x be the 3D feature of point i, MaxPool represents the max pooling operation, and x be the feature of point i. i This represents the two-dimensional image coordinate information of neighboring points.

[0117] Next, in order to expand the target region in the image, effectively enhance the edge information of the target, and reduce the hole region in the image, a dilation operation is performed; at the same time, a cross kernel is used as the structural element of the dilation operation. The design of the cross kernel focuses on the detailed features of the target, which helps to emphasize the edge of the target more prominently during the dilation process, thereby improving the image's discriminability and detection performance; a blur operation is introduced to blur the image, reduce noise and smooth the target region, and improve the continuity of the overall image.

[0118] Finally, a 1×1 convolution operation is performed to adjust the alignment and obtain densely mapped point cloud depth image data. Make it have the same features as the input image They have the same shape, meaning D and C have the same number of channels.

[0119] In the mapped point cloud depth image data I′ depth After the input image data I is processed by a ResNet network convolution operation, the image depth features F are obtained. 3D , With image semantic features F 2D , C 2D This represents the number of feature channels after convolution. When concatenating two types of features, common feature fusion operations include addition and concatenation, but these are both affected by a large number of useless features. Inspired by the selective network SKNet, we introduce an adaptive feature fusion method based on channel attention to fuse image depth features and semantic features, specifically:

[0120] First, the input features F are processed using AvgPool global average pooling. 3D ,F 2D Generate channel statistics C S ,

[0121]

[0122] Then, C is further connected through a fully connected layer. S The dimension from C 2D Reduce to C 2D / r to create a compact feature C' S This enables more precise and adaptive selection guidance, followed by the use of compact feature C' S Predicted attention score A 2d ,

[0123]

[0124] Through the compact feature descriptor C' S Guided cross-channel soft attention is used to adaptively select different information space scales, and the softmax operator is applied at the channel level to ensure that for any i∈{1,2,...,C} 2D A is true in all cases. i，0 +A i，1 =1,,

[0125]

[0126] In the formula: F 2D and F′ 3D Soft attention vector, i represents channel C 2D The i-th element in; the fused perceptually enhanced feature image F^ is composed of F 2D and F′ 3D The weighted sum is obtained,

[0127] S3. Multimodal Feature Extraction: This step aims to extract high-quality feature maps from both 2D images and 3D point clouds using advanced feature extraction algorithms. The core is based on existing deep learning algorithms, such as the YOLOv7 2D image feature extraction algorithm, to extract multi-scale image features F from the fused perceptually enhanced feature image F^. R And feature extraction algorithms for 3D point clouds based on SECOND and Centerpoint from perceptually enhanced point cloud P virtual Extracting voxel features F P ,

[0128]

[0129] In the formula: f1 represents the image feature at the i-th spatial location, p j C represents each non-empty voxel feature extracted from the original point cloud. R and C P These represent the number of channels for image features and point cloud features, respectively.

[0130] S4. Multimodal feature alignment and fusion, network structure see Figure 5 Based on the extracted multimodal features, in order to integrate rich semantic feature information for distant and occluded sparse point cloud targets, a deformable cross-attention network module is introduced. The learningable alignment map is used to model the mapping relationship, enabling the module to automatically align non-homogeneous features in a dynamic and data-driven manner. For 2D convolutional features with depth information, the network's ability to automatically adjust the feature aggregation position can be improved.

[0131] Based on the obtained image features F R and voxel characteristics F P Each non-empty voxel feature p j As the query, the corresponding image plane reference point P is then calculated using the projected coordinate matrices of the camera and the LiDAR. k As a key, P k =(p jh ,p jw And the image domain F can be obtained through bilinear interpolation. R Feature F in k Each query obtains the sampling offsets (ΔP) through a Linear. mqk This function is used to query the surrounding location information of a reference point on the Query image plane. An attention score A can be obtained on the aggregated image features using a Linear and a Softmax algorithm. mqk The final deformable cross-attention alignment feature can be obtained through formula (15).

[0132]

[0133] In the formula: W m and W′ m The weights represent the learnable weights; k refers to the weighted average of the image features of each non-empty voxel Query and the k image features surrounding its corresponding image domain reference point; m refers to the m-th head of the multi-head attention.

[0134] To address the issue of redundant information in image features extracted by 2D convolutional networks, especially since much of this information is useless background information for small targets in discrete point cloud data, which is not conducive to effectively fusing semantic information in point cloud features, a residual fusion network module based on channel attention is designed to adaptively fuse the features and improve the model's ability to perceive different targets, especially in the case of occlusion or long distance.

[0135] The process of multimodal feature fusion is as follows:

[0136] First, the voxel image features F' R and voxel point cloud features F p , F' R , To obtain fused features, features from each network are concatenated, and convolutional layers are used to reduce the number of channels in the fused features. Calculate using formula (16),

[0137] F f =f([F' R ;F p (16)

[0138] Where [·;·] represents the concatenation operation; f() represents the convolution and activation function processing of the fused features to extract a higher level and more representative representation.

[0139] Considering the unreliability of RGB image information in outdoor environments and the susceptibility of cameras to varying lighting and weather conditions, fused features are used as a supplement to the original LiDAR features. A fusion module is designed based on the residual structure and combined with a channel attention module. The output features of the fusion module... This can be expressed by formulas (17) and (18).

[0140]

[0141] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))(18)

[0142] Among them, Mc The convolution operation in the channel attention module of the fusion module is represented by MLP; MLP represents fully connected; Avgpool represents average pooling; Maxpool represents max pooling; σ(x) = 1 / (1+e -x ) represents the sigmoid function.

[0143] This residual-based feature fusion module can optimize the model's ability to perceive different targets, especially in the presence of occlusion or distant targets, and improve the effective semantic information fusion effect of sparse point cloud features of small targets.

[0144] Experimental Example: This experimental example uses the public datasets KITTI and NuScenes as datasets to train a 3D object detection model. The datasets and training parameters are as follows.

[0145] (1) Dataset

[0146] The KITTI dataset contains real-world image data collected from urban, rural, and highway scenes. Each image contains up to 15 vehicles and 30 pedestrians, with varying degrees of occlusion and truncation. The detection difficulty is categorized into three levels: easy, medium, and hard, based on the size of the targets in the scene. This dataset provides 14,999 images and corresponding point clouds for 3D object detection tasks, with 7,481 sets used for training and 7,518 sets for testing. Several performance metrics were selected, including 3D detection frame accuracy at AP40 precision and detection frame accuracy from a bird's-eye view (BEV) perspective. The joint inter-value (IoU) thresholds for vehicle and pedestrian detection were set to 0.7 and 0.5, respectively. In the experiments, detection accuracy was evaluated based on whether the targets in the KITTI data labels were occluded, truncated, and the bounding box extent, categorizing the object detection difficulty into three different levels: easy, medium, and hard.

[0147] The NuScenes dataset comprises 1000 scenes, with 700, 150, and 150 scenes respectively used for training, validation, and testing. Each frame includes six images, a surrounding view, and a LiDAR point cloud. The dataset contains over 14 million annotated 3D bounding boxes across 10 classes. For the 3D detection task, we report several performance metrics, including the nuScenes detection score (NDS), mean average accuracy (mAP), and five true positive (TP) metrics: mean translation error (mATE), mean average scale error (mASE), mean orientation error (mAOE), mean average velocity error (mAVE), and mean attribute error (mAAE).

[0148] (2) Model parameter setting and training

[0149] A dual-stream training strategy was adopted, selecting SECOND or CenterPoint as the 3D detector and CSPNet, a lightweight backbone of YOLOv7, as the image feature extractor. The voxel size was set to (0.05, 0.05, 0.1) for the KITTI dataset and (0.075, 0.075, 0.2) for the NuScenes dataset. To avoid unnecessary computational costs, dynamic voxelization was used to reduce the number of voxel features. For the MPCM module, pre-trained YOLOv7 object detection weights and DeepLabv3+ semantic segmentation weights were used on the KITTI image dataset, and a CenterNet detector was trained on the NuScenes image dataset with cascaded RoI heads added for instance segmentation. The DBSCAN network parameters EPS (specifying the radius parameter) and MinPts (specifying the neighborhood density threshold) were set to 0.1 and 25, respectively, to ensure accurate category querying for occluded point cloud objects. For the FAFM module, the number of heads was set to 4 and the number of deformable points was set to 8. The training and testing were performed on two RTX 3090 GPUs with a batch size of 8 (4 frames per GPU). The entire framework was optimized end-to-end using a hybrid optimizer. The 3D branch was optimized using AdamW, and the 2D branch was optimized using SGD. No data augmentation (i.e., flipping, rotating, or CBGS) was applied when multi-view image input was involved.

[0150] The above description is only a preferred embodiment of the present invention and does not limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A three-dimensional target detection method suitable for occluded and long-distance scenes, characterized in that: Includes the following steps: S1. Multi-dimensional point cloud completion: The original sparse point cloud data and multi-view image data lacking depth are completed in two dimensions and three dimensions to obtain the corresponding point cloud completion data. The three-dimensional point cloud completion data and the two-dimensional point cloud completion data are stitched together to generate dense point cloud data. S2. Depth Image Generation: The dense point cloud data is projected onto 6 image planes using the camera's internal and external parameters to obtain point cloud depth features. RGB data is selectively combined to generate image semantic features. The point cloud depth features and image semantic features are then fused to obtain the fused image data. S3, Multimodal Feature Extraction Module: Employs a 2D image feature extraction algorithm to extract multi-scale image features from the fused image data, and employs a 3D point cloud feature extraction algorithm to extract voxel features from dense point cloud data; S4. Multimodal feature alignment and fusion: Multi-scale image features and voxel features are aligned to obtain voxel image features and voxel point cloud features. The voxel image features and voxel point cloud features are fused to obtain output features, thereby completing target detection.

2. The three-dimensional target detection method applicable to occluded and long-distance scenes according to claim 1, characterized in that: Step S1, multi-dimensional point cloud completion, includes the following steps: (1) Obtain the original sparse point cloud data P and the corresponding multi-view image data I lacking depth. Process the image data I based on the semantic segmentation network to obtain the instance segmentation mask related to the detection target. The input image is processed based on an object detection network to obtain n instance segmentation sub-masks for each category instance segmentation mask. (2) Project the point cloud data P onto the instance segmentation submask region according to the coordinate transformation formula from LiDAR to RGB sensor. Obtain point cloud data P within the segmented region of i*j instances. R , In the formula: t1 and t2 are the LiDAR measurement and RGB image capture times, respectively, and T (camera←lidar) It is the transformation matrix from the lidar sensor to the camera coordinate system. It is the transformation matrix of the camera coordinate system between t2 and t1, T (rgb←camera) It is the transformation matrix from the camera coordinate system to the pixel coordinate system; N R x represents the point cloud data within the masked area. i y i , z i Represents the 3D coordinates of the point cloud, used for 3D completion based on voxel lattice sampling; u i and v i It is a pixel coordinate, d i It represents the true depth, used for two-dimensional completion based on mask region sampling; (3) The DBSCAN algorithm is used to cluster the point cloud data to obtain new sub-mask regions M′. mn and the point cloud cluster P′ to be completed R , (4) 3D point cloud completion of point cloud clusters: The point cloud data is voxelized, and μ non-repeating 3D points are randomly sampled within each non-empty voxel block. For each sampling point F0, the feature r0 is obtained by retrieving the nearest neighbor point cloud from the point cloud data within the acquired voxel grid. In the formula: P' M This represents point cloud data within a non-empty voxel lattice. This indicates that the k-nearest neighbor (K-NN) algorithm was used to detect the three-dimensional coordinates P of the nearest neighbor point cloud of the sampling point F0 within the voxel lattice. 3d and the coordinates of the voxel center P c ; Based on the distance function, the distance error between the point and its nearest neighbor and the center of the voxel is calculated. The coordinates of the generated point are adjusted using an optimization algorithm that minimizes the error, and the process is iterated multiple times to reach the set minimum value. Submask M is removed by mapping. mn To avoid the completed point cloud falling outside the masked area, the optimized and corrected 3D completed point cloud data F′ is used. In the formula: x′ i y′ i , z′ i This indicates the three-dimensional coordinates of the sampling points after correction, which better match the original data. (5) Two-dimensional point cloud completion for point cloud clusters: In the submask region M nm Randomly sample τ unique 2D points. For each sampling point S0, from the acquired point cloud data P′ R Depth features d0 are obtained by retrieving the nearest neighbor point cloud from F′. In the formula: V' M This represents virtual point cloud data that has been completed in three dimensions. This indicates that the nearest neighbor point cloud depth data of the retrieved sampling point S0 was detected using the k-nearest neighbor algorithm; (6) Based on the depth information and two-dimensional coordinate information, generate real three-dimensional spatial coordinate information based on the transposed coordinate matrix. (7) By stitching together the 3D point cloud completion data and the 2D point cloud completion data, the obtained point cloud data is combined with the original point cloud data to generate dense point cloud data P. virtual , P virtual ∈R (N×C) Where: N v This represents the original point cloud data plus the number of virtual point clouds generated.

3. The three-dimensional target detection method applicable to occluded and long-distance scenes according to claim 1, characterized in that: Step S2, depth image generation, includes the following steps: (1) Using the camera's internal and external parameters to extract dense point cloud data P virtual The point cloud depth information is obtained by projecting it onto six image planes. Where: N C The number of cameras is indicated; D, H, and W represent the depth, height, and width of the point cloud image obtained after projection onto the camera perspective, respectively; u and v represent the pixel coordinates within the projected depth image; d corresponds to the depth value associated with the pixel at position (u,v); T (camera←lidar) It is the transformation matrix from the lidar sensor to the camera coordinate system; It is the transformation matrix of the camera coordinate system between t2 and t1; T (rgb←camera) It is the transformation moment from the camera coordinate system to the pixel coordinate system; X LiDAR ,Y LiDAR and Z LiDAR Represents the location information of point cloud data; * represents the prime product of two matrices; (2) Fill the global point cloud: First, for each target pixel q in the dense mapping, find its k nearest neighbors on the projection point of the image plane. Use a lightweight MLP and a sigmoid activation function to weight the adjacent features according to the coordinate offset to give a score in the range (0,1). Then, use the adjacent features weighted by the score to perform a symmetric operation to aggregate them and obtain the aggregated features. In the formula: D(q) represents the depth feature of the hole pixel q, N q G represents all neighboring points. i It represents the 3D feature of point i, and MaxPool represents the max pooling operation x. i Represents the two-dimensional image coordinate information of neighboring points; Then, the dilation operation is performed, using a cross kernel as the structural element of the dilation operation, and a blur operation is introduced to blur the image. Finally, a 1×1 convolution operation is performed to adjust the alignment and obtain the densely mapped point cloud depth image data. Make it have the same features as the input image They have the same shape, meaning D and C have the same number of channels; (3) In the mapped point cloud depth image data I′ depth After the input image data I is processed by a ResNet network convolution operation, the image depth features are obtained. Image semantic features C 2D This represents the number of feature channels after convolution.

4. The three-dimensional target detection method applicable to occluded and long-distance scenes according to claim 3, characterized in that: The fusion process in step (3) involves introducing an adaptive feature fusion method based on channel attention to fuse image depth features and semantic features, including the following steps: First, the input features are processed using AvgPool global average pooling. Generate channel statistics Then, the dimension of S is further increased from C through a fully connected layer. 2D Reduce to C 2D / r to create a compact feature C' S This enables more precise and adaptive selection guidance, followed by the use of compact feature C' S Predicted attention score Through the compact feature descriptor C' S Guided cross-channel soft attention is used to adaptively select different information space scales, and the softmax operator is applied at the channel level to ensure that A holds for any i∈{1,2,...,C}. i,0 +A i,1 =1 In the formula: F 2D and F3' D Soft attention vector, i represents channel C 2D The i-th element in; the perceptually enhanced feature image data F^ is derived from F 2D and F′ 3D The weighted sum is obtained, 5. The three-dimensional target detection method applicable to occluded and long-distance scenes according to claim 1, characterized in that: In step S3, the YOLOv7 2D image feature extraction algorithm is used to extract multi-scale image features F from the perceptually enhanced feature image data F^. R Feature extraction algorithms for 3D point clouds, employing SECOND and Centerpoint, are used to extract features from perceptually enhanced dense point cloud data P. virtual Extracting voxel features F P , F R ={f1,f2,...,f hw }, F P ={p1,p2,...,p j }, In the formula: f1 represents the image feature at the i-th spatial location, p j This represents the feature of each non-empty voxel extracted from the original point cloud.

6. The three-dimensional target detection method applicable to occluded and long-distance scenes according to claim 1, characterized in that: Step S4 includes the following steps: (1) Based on the obtained image features F R and voxel characteristics F P Each non-empty voxel feature p j As the query, the corresponding image plane reference point P is then calculated using the projected coordinate matrices of the camera and the LiDAR. k As a key, P k =(p jh ,p jw Furthermore, the image domain features F can be obtained through bilinear interpolation. k Each query obtains the sampling offsets ΔP through a Linear. mgk This is used to query the surrounding location information of the reference point on the Query image plane. An attention score A can be obtained on the aggregated image features using a Linear and a Softmax algorithm. mqk To obtain the final deformable cross-attention alignment feature. In the formula: W m and W′ m The weights represent the learnable weights; k refers to the weighted average of the image features of each non-empty voxel Query and the k image features surrounding its corresponding image domain reference point; m refers to the m-th head of the multi-head attention. (2) Connect the features from each network and use convolutional layers to reduce the number of channels of the fused features to obtain voxel image features F'. R and voxel point cloud features F p The fusion characteristics F f =f([F‘ R ;F p ]) In the formula: F' R F represents the features of a voxel image. p represents the voxel point cloud features, [·; ·] represents the concatenation operation; f() represents the convolution and activation function processing of the fused features to extract a higher level and more representative representation; (3) The fused features are used as a supplement to the original LiDAR features, and the output features F are obtained by fusing them based on the residual structure and combined with the channel attention module. out , M c (F)<σ(MLP(AvgPool(F))+MLP(MaxPool(F)) Where: M c The convolution operation in the channel attention module of the fusion module is represented by MLP; MLP represents fully connected; Avgpool represents average pooling; Maxpool represents max pooling; σ(x) = 1 / (1+e -x ) represents the sigmoid function.

7. A three-dimensional target detection model suitable for occluded and long-distance scenes, characterized in that: It includes a bidirectional sensing enhancement network, a multimodal feature extraction module, and a multimodal feature alignment and fusion module; The bidirectional sensing enhancement network is used to process the original sparse point cloud data and image data lacking depth to obtain dense point cloud data and image data with depth. The multimodal feature extraction module is used to extract voxel features from perception-enhanced dense point cloud data and to extract multi-scale image features from perception-enhanced image data. The multimodal feature alignment and fusion module uses a deformable cross-attention network module to automatically align non-homogeneous features obtained by the multimodal feature extraction module and voxel features in a dynamic and data-driven manner. Then, the aligned features are fused to obtain fused features, thereby completing target detection.

8. The three-dimensional target detection model applicable to occluded and long-distance scenes according to claim 7, characterized in that: The bidirectional perception enhancement network includes a multi-dimensional point cloud completion module and a depth image generation module. The multi-dimensional point cloud completion module is used to perform two-dimensional and three-dimensional completion on the original sparse point cloud data and multi-view image data lacking depth, and then stitches the completed data to generate dense point cloud data. The depth image generation module uses the camera's internal and external parameters to project the dense point cloud data onto six image planes to obtain point cloud depth features, and selectively combines RGB data to generate image semantic features. The point cloud depth features and image semantic features are then fused to obtain fused image data.

Images