A low-altitude 3D target detection method based on multi-modal information fusion

Abstract

The application relates to a low-altitude 3D target detection method based on multi-modal information fusion, comprising the following steps: respectively performing feature extraction on enhanced point clouds and RGB images after alignment; based on virtual points, extracted image features and camera calibration parameters, completing point cloud features and sampling agent features; uniformly mapping the completed point cloud features and the image features to a bird's eye view to respectively obtain point cloud BEV features and image BEV features; screening potential target areas in the bird's eye view plane based on a preset 2D detection clue, extracting basic features from each potential target area to construct a feature set, performing three-stage cascade optimization on the feature set based on a fusion BEV feature, the agent feature and the completed point cloud feature which are fused with the point cloud BEV feature, the image BEV feature and the 2D detection clue, outputting a region of interest feature, and classifying the region of interest feature to output an accurate 3D detection result.

Description

Technical Field

[0001] This application relates to the technical field of low-altitude 3D target detection methods, and in particular to a low-altitude 3D target detection method based on multimodal information fusion. Background Technology

[0002] Multimodal 3D target detection is a core supporting technology for low-altitude situational awareness systems. Its core task is to accurately identify and locate the 3D spatial state of targets such as drones, birds, buildings, and temporary obstacles in low-altitude scenes by fusing complementary information from LiDAR (airborne lidar) and cameras.

[0003] In practical low-altitude applications, multimodal 3D target detection faces four core challenges:

[0004] 1. Modal heterogeneity conflict: LiDAR point clouds are resistant to illumination and cloud / fog interference but lack texture and semantics; camera RGB images are greatly affected by cloud / fog scattering and backlighting and lack direct depth information. The structural difference between the two, 3D sparse and 2D dense, directly restricts the fusion effect, and existing methods are unable to completely eliminate heterogeneity.

[0005] 2. Incomplete features due to the sparsity of LiDAR point clouds: In low-altitude targets, LiDAR reflective points are extremely few, and the point cloud density often falls below 3 points / cubic meter as the detection distance increases; at the same time, complex terrain also leads to incomplete target structure representation. Existing methods struggle to meet the feature completeness requirements of different scenarios, such as small targets and large obstacles.

[0006] 3. Insufficient hard sample detection capability: Interleaved occlusion can cause point clouds and image features to overlap and become difficult to distinguish. Small targets at a distance result in sparse point clouds and a very small proportion of target pixels in the image. Existing detection solutions are prone to false positives or false negatives and are not deeply integrated with cascaded detection processes, making it difficult to specifically address the feature loss problem of hard samples.

[0007] 4. Weak robustness: In real low-altitude environments, cameras and airborne LiDAR are easily affected by the environment, leading to dynamic inaccuracies in multimodal data. Existing multimodal solutions only design robust training mechanisms for a single modality and lack cross-module multi-loss collaborative constraints, resulting in a significant reduction in detection performance in complex low-altitude environments. Summary of the Invention

[0008] Therefore, it is necessary to provide a low-altitude 3D target detection method based on multimodal information fusion, including:

[0009] S1: Acquire multimodal data containing LiDAR point cloud, RGB image and camera calibration parameters, preprocess the multimodal data to obtain enhanced point cloud and RGB image, and generate virtual points;

[0010] S2: Align the enhanced point cloud and RGB image and extract features respectively to obtain aligned point cloud features and image features; complete the point cloud features based on virtual points, image features and camera calibration parameters and sample surrogate features;

[0011] S3: Map the completed point cloud features and image features to the bird's-eye view to obtain point cloud BEV features and image BEV features respectively; Based on the preset 2D detection cues, screen out potential target areas in the bird's-eye view plane; Within the potential target areas, fuse point cloud BEV features, image BEV features and 2D detection cues to obtain fused BEV features.

[0012] S4: Extract basic features from each potential target region to construct a feature set. Perform three-stage cascade optimization on the feature set based on the fused BEV features, proxy features, and completed point cloud features. Output the region of interest features and classify the region of interest features to output accurate 3D detection results.

[0013] Beneficial effects: This method breaks through the performance bottleneck of single-modal and simple multimodal fusion, and can meet the core requirements of low-altitude situational awareness systems for 3D target detection of "high precision, high robustness and high efficiency". Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a low-altitude 3D target detection method based on multimodal information fusion in an embodiment of this application. Detailed Implementation

[0016] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0017] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0018] like Figure 1 As shown, this embodiment provides a low-altitude 3D target detection method based on multimodal information fusion, including:

[0019] S1: Acquire multimodal data containing LiDAR point cloud, RGB image and camera calibration parameters, preprocess the multimodal data to obtain enhanced point cloud and RGB image, and generate virtual points.

[0020] In this embodiment, the PENet deep completion network, based on deep learning, is used to generate virtual points. The PENet deep completion network employs an encoder-decoder architecture. The methods for generating virtual points include:

[0021] The LiDAR point cloud and RGB image are respectively processed by an encoder to extract the spatial geometric features of the point cloud and the texture semantic features of the image.

[0022] Spatial geometric features and texture semantic features are input into the decoder to obtain a depth map with the same resolution as the RGB image.

[0023] The image domain pixel coordinates of each pixel in the depth map are converted into 3D spatial coordinates using the camera perspective projection formula to obtain the 3D spatial coordinates of each virtual point; the three color channel values ​​of the corresponding pixels in the RGB image are extracted, and semantic labels for each virtual point are generated by combining the foreground target annotation.

[0024] The training process of the PENet depth completion network is as follows: the depth labels provided in the dataset are used as supervision signals, and the deviation between the predicted depth and the real depth is constrained by the L1 loss function, so that the depth error of the generated virtual points is controlled within 0.1 meters, which meets the accuracy requirements of near-range target detection.

[0025] In this embodiment, the camera perspective projection formula is:

[0026] ;

[0027] in, Represents pixel coordinates, , These represent the focal lengths of the camera in the x and y directions, respectively. Indicates the coordinates of the camera's principal point (usually located at the center of the image). Point The coordinates in the camera coordinate system (with the camera optical center as the origin, the x-axis to the right, the y-axis downward, and the z-axis forward). This represents a 3×1 translation vector. Point Coordinates in a 3D spatial coordinate system.

[0028] Optionally, other methods for generating virtual points include:

[0029] The CenterNet object detector is used to segment the foreground of the RGB image and output the binary mask of the object.

[0030] Map the LiDAR point cloud to the image domain using the camera perspective projection formula;

[0031] For pixels in the binary mask that do not have corresponding LiDAR point clouds, the nearest neighbor interpolation method is used to generate the 3D spatial coordinates of each virtual point. The virtual points generated in this way can improve the 3D detection accuracy by 6.6 mAP, especially for the detection of vehicles at a distance greater than 50 meters.

[0032] Both of the above virtual point generation schemes need to undergo spatiotemporal consistency verification: duplicate points are eliminated by calculating the Euclidean distance between the virtual point and the original point cloud (threshold set to 0.3 meters), and time offsets of different modes are corrected using IMU sensor data (≤50ms) to avoid virtual point positioning deviations caused by spatiotemporal misalignment.

[0033] Preprocessing of multimodal data includes:

[0034] The SegFormer model, pre-trained on the Cityscapes dataset, is used to generate pseudo-labels for RGB images;

[0035] This model employs a combined architecture of a Transformer encoder and a lightweight MLP decoder. The encoder outputs feature maps at four different scales (resolutions ranging from 1 / 4 to 1 / 32), achieving multi-scale information fusion without location embedding, effectively avoiding performance degradation caused by resolution variations. The decoder combines local texture features with global semantic features through simple feature concatenation and an MLP layer, achieving 84.0% mIoU on the Cityscapes validation set, with only 1 / 5 the number of parameters compared to traditional models. The class probability of each pixel is calculated using the Softmax function, with the following formula:

[0036] ;

[0037] in, For pixels Category The probability of; Pixels output by the SegFormer model In category The raw scores (unnormalized) on the chart. This represents the total number of categories for semantic segmentation (set to 19 categories based on the low-altitude 3D object detection scenario, including core categories such as roads, buildings, and vehicles). The final pseudo-label is the category with the highest probability. .

[0038] The projection range of the true value of the 3D bounding box onto the RGB image is calculated using the camera perspective projection formula, resulting in a rectangular region containing the complete outline of the target.

[0039] Optionally, the ground truth value of the 3D bounding box is a pre-labeled ground truth value obtained from a standard 3D object detection dataset (such as KITTI, nuScenes).

[0040] The intersection-union ratio (IUGR) is used to remove overlapping bounding boxes from the ground truth values ​​of all 3D bounding boxes. Then, pixels, LiDAR point clouds, and virtual points corresponding to the pseudo-labels of "moving targets" (e.g., vehicles, pedestrians) within the rectangular regions corresponding to the ground truth values ​​of each retained 3D bounding box are extracted. This yields the enhanced point cloud, enhanced RGB image, semantic labels, and the ground truth values ​​of the 3D bounding boxes. The enhanced point cloud includes both LiDAR point clouds and virtual points. The IUGR calculation formula is:

[0041] ;

[0042] ;

[0043] in, The intersection-union ratio of the two 3D bounding boxes; The volume of the intersection; Let the volume be the union volume; , The newly added targets and the original goal 3D volume (calculated from the length, width, and height of the bounding box); , For the goal Maximum and minimum values ​​in the x-axis direction (in world coordinates). , The axis range definition is similar, target The coordinate range is defined consistently. Experiments show that when When the threshold is set to 0.5, all physically unreasonable overlapping targets can be removed while retaining 92% of the effective augmentation targets.

[0044] S2: Align the enhanced point cloud and RGB image and extract features respectively to obtain aligned point cloud features and image features; complete the point cloud features based on virtual points, image features and camera calibration parameters and sample surrogate features.

[0045] In this embodiment, the enhanced point cloud and RGB image further include the following before alignment:

[0046] The sparse depth map of the enhanced point cloud and the enhanced RGB image are processed by a depth map encoder and a depth map decoder to map into a dense depth map. The calculation formula is as follows:

[0047] ;

[0048] in, Represents a dense depth map. This represents a depth map decoder. This represents a depth map encoder. This represents a sparse depth map of the enhanced point cloud. This represents the enhanced RGB image; the sparse depth map is the geometric projection of the enhanced point cloud onto the image domain.

[0049] The enhanced LiDAR point cloud or virtual points are projected onto the image domain pixel coordinates using the following formula:

[0050] ;

[0051] in, Represents the pixel coordinates in the image domain. Represents pixels The depth value represents the camera intrinsic parameter matrix in the camera calibration parameters, and the rotation matrix from the 3D spatial coordinate system to the camera coordinate system in the camera calibration parameters is also represented. This represents the translation vector in the camera calibration parameters. Represents the 3D spatial coordinates of the enhanced LiDAR point cloud or virtual points;

[0052] Set the depth value of the enhanced LiDAR point cloud or virtual point to the depth value of the corresponding pixel in the dense depth map;

[0053] Preliminary features of the enhanced RGB image are extracted using the ResNet-18 model. Fine-grained texture features and medium-grained contour features from these preliminary features are extracted using convolutional kernels of different sizes. The fine-grained texture features and medium-grained contour features are then concatenated to obtain a local multi-scale feature map. The calculation formula is as follows:

[0054] ;

[0055] ;

[0056] in, Indicates adoption Local feature maps extracted by convolution kernels ( Corresponding to fine-grained textures (corresponding to medium-grained profile) For the first Convolution kernel ( The number of input feature channels, (Number of output channels). Indicates the convolution operation; For bias terms; It is the ReLU activation function; Preliminary features of the enhanced RGB image (extracted by ResNet-18, 256 channels); This is a channel-level splicing operation. This is a local multi-scale feature map.

[0057] The dense depth map is linearly transformed into a query matrix, and the local multi-scale feature map is linearly transformed into a key matrix and a value matrix. Attention weights are calculated based on the query matrix and the key matrix. The attention weights are multiplied by the value matrix and then added to the local multi-scale features. The result is then normalized to obtain the globally aligned image semantic features, denoted as . .

[0058] Furthermore, aligned point cloud features and image features are obtained, including:

[0059] The globally aligned image semantic features are processed through ResNet-50, and the output features of the final stage are used as the corresponding basic semantic features. Positional attention enhancement is then applied to the basic semantic features, calculated as follows:

[0060] ;

[0061] ;

[0062] in, This represents the features after enhanced positional attention. , This represents the positional attention weight matrix. Represents basic semantic features. , This represents the softmax function. Indicates transpose. Represents the L2 norm;

[0063] Channel attention enhancement is applied to the features enhanced with positional attention, and the calculation formula is as follows:

[0064] ;

[0065] ;

[0066] ;

[0067] in, This represents the features after channel attention enhancement. This represents the channel attention weight matrix. This represents element-wise multiplication. This represents the Sigmoid activation function. Denotes the first linear transformation matrix. Denotes the second linear transformation matrix. Indicates the first bias term. Indicates the second bias term. Represents the ReLU activation function. Indicates global average pooling. Indicates pooling characteristics;

[0068] The channel attention-enhanced features are mapped to a fixed dimension (256 in this embodiment) to obtain aligned image features.

[0069] The farthest point sampling method is used to select several key points from the enhanced point cloud. For any key point, a neighborhood is defined with a fixed radius, and local geometric features are extracted using the PointNet model. The calculation formula is as follows:

[0070] ;

[0071] in, Indicate key points The local geometric features represent the multilayer perceptron, and the feature concatenation represents the feature stitching. Indicate key points any point in the neighborhood of , Indicate key points The neighborhood;

[0072] Based on local geometric features and dense depth maps, the point cloud features for keypoint alignment are calculated using the following formula:

[0073] ;

[0074] ;

[0075] ;

[0076] in, Indicate key points Aligned point cloud features, Indicate key points Associative deep semantic features Indicates bilinear interpolation. Representing depth map features, Indicate key points The corresponding pixel coordinates Represents a dense depth map. It represents a linear transformation.

[0077] The final output is an aligned point cloud feature. Aligned image features It not only achieves precise matching of spatial location through depth maps and camera parameters, but also complements semantic information through attention mechanisms and feature injection, providing a modally consistent feature foundation for subsequent virtual point completion of point cloud features.

[0078] Furthermore, point cloud features are completed based on virtual points, image features, and camera calibration parameters, including:

[0079] The 3D spatial coordinates of each point in the point cloud feature are converted into pixel coordinates, and a color mapping table for the image domain is constructed based on the color information contained in the virtual points. ,in Represents pixels The corresponding virtual point color;

[0080] If the pixel corresponding to a point is not covered by any virtual point, then the average color of all the virtual points covering the corresponding pixel in the neighborhood is assigned to the corresponding point; otherwise, the color of the virtual point is matched from the color map table and assigned to the corresponding point; the expression is:

[0081] ;

[0082] in, For point The final color information (R, G, B values ​​range from 0 to 255); For point Local neighborhood (with) A set of virtual points within a spherical region (centered at a radius of 0.5m); The number of virtual points within the neighborhood; For a single virtual point within the neighborhood The color value is added. Through VPP operation, a new color channel is added to the point cloud. During subsequent fusion, the color feature can be used to quickly distinguish the target (such as red vehicles, blue road signs) from the background (gray road surface), especially significantly improving the detection accuracy of small targets (such as pedestrians, cyclists).

[0083] Denoise all virtual points to obtain a denoised virtual point set; specifically:

[0084] Based on whether the dataset has ground truth bounding boxes, a differentiated denoising strategy is adopted to ensure denoising accuracy and scene adaptability.

[0085] (1) KITTI dataset (with GT bounding boxes): The first step is to filter foreground virtual points using GT bounding boxes—removing background virtual points (such as virtual points in the sky, trees, etc., which are not part of the target area), and retaining only virtual points in the target area. Let the range of a certain GT bounding box in the world coordinate system be... , , Then only virtual points that meet the following conditions will be retained. :

[0086] ;

[0087] in, , The minimum and maximum coordinates of the ground truth bounding box in each axis direction are given (provided by the dataset annotations); after filtering, the foreground virtual point set is obtained. ( (The original virtual point set).

[0088] The second step is... Denoising is achieved using DBSCAN (Density-Based Noise Spatial Clustering), the core of which is to remove isolated noisy virtual points (points whose positions are off-center due to depth errors). DBSCAN determines clusters based on "core points": for any virtual point... If its Within the neighborhood (KITTI dataset setting) The number of virtual points included is not less than (KITTI dataset setup) If a point is identified as a core point, the formula is:

[0089] ;

[0090] in, For virtual points and The Euclidean distance; The number of elements in the set; The neighborhood radius (controls the density sensitivity of clustering); To determine the minimum number of neighborhood points for a core point, all core points and "density-reachable" virtual points (points within the same cluster) are retained, while isolated points (noise points) are removed, resulting in a denoised set of virtual points. .

[0091] (2) nuScenes dataset (without ground truth bounding boxes): Since there are no ground truth bounding boxes to filter the foreground, the original virtual point set is directly used. DBSCAN is used for noise reduction, with parameters adapted to scene complexity: neighborhood radius. (Greater than KITTI, because nuScenes scenarios are more complex and virtual points are more dispersed), minimum number of neighborhood points (Greater than KITTI to avoid misjudging the foreground as noise), the denoising logic is consistent with KITTI, ultimately yielding a denoised virtual point set. .

[0092] The space containing the dense depth map is divided into several cubic sub-regions with a side length of 1m. Calculate the point cloud density in each cubic sub-region. The calculation formula is:

[0093] ;

[0094] Wherein, the molecule is a cubic subregion Number of point clouds after inner alignment; denominator The volume of the sub-region is fixed at 1 m³.

[0095] When the point cloud density is less than a set threshold (5 points / cubic meter), the corresponding cubic sub-region is determined to be a sparse region, and the virtual point requirement of the sparse region is calculated based on the difference between the point cloud density and the set threshold. The calculation formula is:

[0096] ;

[0097] in, The function is an up-rounding function (ensuring that the density after supplementation is not lower than the threshold). This represents the density difference. Therefore .

[0098] Select several denoised virtual points from the set of denoised virtual points that are closest to the center of the corresponding cubic sub-region and fill them into the sparse region; the calculation formula is:

[0099] ;

[0100] in, The features of the completed point cloud; For the denoised virtual point set (KITTI uses) nuScenes uses ); sparse sub-region The center coordinates; These are the selected virtual points to be added; This indicates that the number of filters equals the number of requirements.

[0101] Traverse all sparse regions and fill them to obtain the completed point cloud features.

[0102] Through the completion operation, the density of the point cloud in distant (50~100m) and occluded areas (such as a truck obscuring a car) is significantly improved, and the features of the completed point cloud are enhanced. It can more accurately reflect the 3D structure of the target, while surrogate features This provides a semantic association basis for subsequent BEV fusion.

[0103] In this embodiment, the agent features are sampled, including:

[0104] Points with depth values ​​less than the depth threshold (0.5m) in the dense depth map are removed to obtain point cloud proxy points, thus avoiding ground clutter interference.

[0105] The dense depth map is voxelized, and the center coordinates of each non-empty voxel are taken as voxel surrogate points; based on the characteristics of the dataset scene, the voxel size of the car class target in the KITTI dataset is set to (X-axis side length) (Y-axis side length) (Z-axis side length), the boundary range of the voxel mesh is (100m forward) (30m on each side) (From 2m below ground to 4m above ground). For any non-empty voxel, first calculate its index in the X, Y, and Z axes. Then, deduce the coordinates of the conductor element center:

[0106] ;

[0107] ;

[0108] in, , , y represents the 3D spatial coordinates of any point within the voxel; This is a floor function (ensure the index is an integer); , , The minimum boundary of the voxel mesh; , , These are the indices of the voxels on the X, Y, and Z axes, respectively. The world coordinates of the proxy point of this voxel; The length is half the length of the voxel, ensuring that the coordinates are located at the center of the voxel.

[0109] A fixed-size grid is defined on the bird's-eye view plane, and the coordinates of the grid nodes are used as surrogate points. Specifically, a fixed-size grid is defined on the BEV plane (the bird's-eye view plane, which is the XY-axis plane, ignoring the Z-axis height, adapted to the BEV perspective), and the coordinates of the grid nodes are used as surrogate points to cover global view features. Let the BEV grid size be... (128 grids horizontally, 128 grids vertically), the grid boundary is consistent with the voxel grid. , Then the grid step sizes on the X and Y axes are respectively , ( (Grid dimension), grid nodes The coordinates are:

[0110] ;

[0111] Note: , For the index of the grid node; , The grid step size; The default height of the BEV plane (take the average height of the target in the scene to ensure that the proxy point is located in the effective area of ​​the target); These are the 3D spatial coordinates of the grid proxy point.

[0112] For any point cloud proxy point, voxel proxy point, or mesh proxy point, obtain the corresponding pixel coordinates using the camera perspective projection formula, and take the four nearest integer pixels around the pixel coordinates. Perform bilinear interpolation based on the elements corresponding to the four integer pixels in the image features to obtain the proxy features of the corresponding proxy point.

[0113] S3: Map the completed point cloud features and image features to the bird's-eye view to obtain point cloud BEV features and image BEV features respectively; based on preset 2D detection cues, screen out potential target regions in the bird's-eye view plane; within the potential target regions, fuse point cloud BEV features, image BEV features and 2D detection cues to obtain fused BEV features.

[0114] Point cloud BEV feature generation is achieved through three steps: “voxarization - 3D sparse convolution extraction - BEV projection”, which compresses 3D point cloud features into 2D BEV features while preserving the target’s height (Z-axis) information and spatial structure.

[0115] The first step is voxelization: converting the completed point cloud features ( For point The world coordinates are divided into a fixed-size 3D voxel mesh. The spatial range of the voxel mesh is adapted to the requirements of low-altitude 3D target detection scenarios. (Forward Coverage) Therefore , ), (Cover to the left and right) Therefore , ), (Vertical coverage) Therefore , ); voxel resolution set to direction Each grid With 32 grid points in each direction, the side lengths of individual voxels along each axis are calculated:

[0116] ;

[0117] in, These are the side lengths of the voxel along the X, Y, and Z axes (unit: m). These represent the number of voxel meshes for each axis; substituting the parameters yields... , , This ensures that voxels can cover the entire scene while preserving the local structure of the target.

[0118] Then, for each point Calculate the index of its voxel. To achieve the mapping from point clouds to voxels:

[0119] ;

[0120] in, This is a floor function (ensure the index is a non-negative integer); If the coordinates of a point are outside the range of the voxel mesh, it is discarded directly (considered as noise outside the scene).

[0121] The second step is 3D sparse convolution extraction: Four layers of 3D sparse convolution are used to extract voxel features from non-empty voxels (voxels containing point clouds), with the kernel size being [missing information]. The stride along the Z-axis is alternately set to 2 and 1 (2 stride for layers 1 and 3, and 1 stride for layers 2 and 4), which compresses the feature dimension while preserving Z-axis information. After each convolutional layer, ReLU activation function and batch normalization (BN) are used to optimize training stability, and finally, 3D voxel features are output. ( This represents the number of voxel grids after convolution. (Number of feature channels).

[0122] The third step is BEV projection: mapping 3D voxel features along the Z-axis. Max pooling is performed to compress the multi-channel features along the Z-axis into a single channel, resulting in the initial BEV features; then, two more layers are applied. 2D convolution (stride 1, padding 1) aggregates spatial features to ultimately generate point cloud BEV features. ( (where the final number of channels is ), the mathematical expressions for pooling and convolution are:

[0123] ;

[0124] ;

[0125] in, For the initial BEV features in the mesh Feature vector (dimension) at ); Indicates indexing along the Z-axis Take the maximum value (preserve the most prominent features of the target in the vertical direction, such as the top profile of a vehicle). These are the first and second layers of 2D convolution (convolution kernels) (The number of output channels is 256). For batch normalization, This is an activation function used to enhance the nonlinear representation of features.

[0126] Image BEV feature generation is achieved through a three-step process of "lifting-tiling-projection". The semantic features of the 2D image are mapped to the 3D space and then compressed into BEV features, ensuring that the viewpoint and coordinates are completely aligned with the point cloud BEV features.

[0127] The first step is "enhancing": aligning the image features based on camera intrinsics. ( For image height and width, (For the number of feature channels) is elevated from 2D pixel coordinates to 3D camera coordinates. Let any pixel in the image feature map... ( For horizontal indexing, (Using vertical indexing), its corresponding 3D camera coordinates The density depth map is calculated by inverse operation of camera perspective projection and combined with the obtained density map. (pixels) The corresponding depth value, i.e. The formula is:

[0128] ;

[0129] Note: The focal length of the camera in the X and Y directions (unit: pixels, provided by camera calibration parameters); These are the pixel coordinates of the camera's principal point (image center); pixels The X, Y, and Z coordinates in the camera coordinate system (unit: m), where This is the depth value (a positive number indicates that the camera is in front of you).

[0130] Then, pixels Features By binding the features to the corresponding 3D camera coordinates, the 2D features are "upgraded" to 3D features, resulting in a 3D image feature set. .

[0131] The second step is "tiling": assembling the 3D image feature set. Mapped to the same 3D voxel mesh as the point cloud voxelization (i.e., the same... (Range and voxel resolution), for each voxel, if multiple 3D image feature points exist, the features are aggregated through average pooling to generate a 3D image feature volume. The aggregation formula is:

[0132] ;

[0133] Note: voxels The set of all 3D image feature points within the image; The number of feature points in the set (set to 1 if empty, and the feature value to 0); average pooling ensures that the features of each voxel can represent the semantics of the image in that spatial region (such as the semantic information of "vehicle" and "road surface").

[0134] The third step is "projection": projecting the 3D image features along the Z-axis. Perform average pooling (different from max pooling for point cloud BEV, because the image semantics are more uniform in the vertical direction), compress into BEV-2D features; then pass through 1 layer The 2D convolution (stride 1, padding 1) adjusts the number of channels, ultimately generating an image BEV feature with the same size as the point cloud BEV feature. ( The formula is:

[0135] ;

[0136] ;

[0137] Note: BEV features (dimensions) of the initial image ); Average pooling is performed along the Z-axis (preserving the overall semantics in the vertical direction, such as the height semantics of "pedestrian"); subsequent convolution and normalization operations ensure that the dimensions of the image BEV features are completely matched with the point cloud BEV features, laying the foundation for subsequent fusion.

[0138] Specifically, the fused BEV features include:

[0139] The center coordinates of each non-empty voxel in the bird's-eye view plane are taken as the mean of the Gaussian distribution of the corresponding voxel; let the coordinates of its center in the BEV plane (XY plane) be... ( , , (using voxel indexes), with the mean of the Gaussian distribution.

[0140] The covariance is calculated based on the size and orientation of the target in the preset 2D detection cues; the covariance matrix is ​​represented as:

[0141] ;

[0142] Wherein, rotation matrix Note: It is a 2×2 covariance matrix (describing the spatial distribution variance of the target in the XY plane); This is used to rotate the target's axis alignment variance to the actual orientation, ensuring that the Gaussian distribution matches the target's true spatial shape. These are the variances in the length and width directions of the target, respectively (based on the statistical assumption that "when the center of the target is within a voxel, there is a 95% probability that it falls within the length / width range").

[0143] Based on the Gaussian distribution mean and covariance of voxels, calculate the Gaussian score of the target contained in the corresponding voxel; the calculation formula is:

[0144] ;

[0145] in, voxels Gaussian score (probability density value, range) ); BEV coordinates of the voxel center (i.e. (Here, the voxel center coordinates are substituted to calculate the probability of itself). This is the determinant of the covariance matrix (used for normalizing probabilities); It is the inverse of the covariance matrix; the exponent term describes the spatial distance correlation between the voxel center and the target center. The closer the distance (or the more the voxel matches the target size distribution), the higher the probability density value.

[0146] Voxels with Gaussian scores greater than the Gaussian score threshold (0.6) are merged into continuous regions to obtain potential target regions; the merging rule is as follows:

[0147] ;

[0148] in, A set of high-probability voxels; For the first One potential target area; Represents the calculation of voxels With existing areas Minimum BEV distance of the inner voxel (in terms of actual spatial distance) As the merging threshold, approximately The merged potential target region can accurately cover the spatial range of the potential target, thus defining the effective calculation area for subsequent fine fusion.

[0149] Within the potential target region, the point cloud BEV features and the image BEV features are concatenated along the channel dimension to obtain the concatenated features; the number of channels of the concatenated features is compressed to a fixed dimension (256) by a 3×3 2D convolution layer (stride of 1, padding of 1) to obtain the compressed features.

[0150] Global average pooling is performed on each channel of the compressed feature to obtain the channel descriptor for each channel; the channel descriptor is then passed through two layers of multilayer perceptron (the input dimension of the first layer MLP) Output dimension The second-layer MLP has an input dimension of 64 and an output dimension of... (used to learn the non-linear correlation between channels) and the Sigmoid activation function to generate corresponding channel weights;

[0151] The channel weights of each channel are multiplied one by one with the channel weights of the compressed feature to obtain the dynamic fusion feature;

[0152] The pseudo-labels (such as “vehicle=1”, “pedestrian=2”, “background=0”) are converted into category probability vectors through one-hot encoding, with the number of categories K=19. The category probability vectors are then mapped onto the bird’s-eye view to obtain the BEV semantic probability features.

[0153] The pixel coordinates of the 2D bounding boxes in the 2D retrieval cues are converted into the 3D bounding box range of the bird's-eye view, and the bounding box features of each 2D bounding box (output by Faster R-CNN) are mapped to the corresponding region in the bird's-eye view to obtain BEV-2D detection features.

[0154] The dimensions of both the BEV semantic probability feature and the BEV-2D detection feature were adjusted to a fixed dimension (256). The dynamically fused feature was then added element-wise to the BEV semantic probability feature and the BEV-2D detection feature with the adjusted dimensions to generate the fused BEV feature.

[0155] S4: Extract basic features from each potential target region to construct a feature set. Perform three-stage cascade optimization on the feature set based on the fused BEV features, proxy features, and completed point cloud features. Output the region of interest features and classify the region of interest features to output accurate 3D detection results.

[0156] Optionally, basic features are extracted from each potential target region to construct a feature set, including:

[0157] Any potential target region is divided into several grids of equal size. The spatial extent of the bird's-eye view corresponding to each grid is calculated based on its side length (the x-axis side length is calculated by dividing the potential target region's x-axis length by the number of x-axis divisions, and the y-axis side length is calculated by dividing the potential target region's y-axis length by the number of y-axis divisions). The calculation formula is:

[0158] ;

[0159] in, Potential target area Inner line, number Column grid; , This is the starting offset of the grid, ensuring that the 7×7 grid completely covers the potential target area. BEV range, , These are the potential target areas. The side length of a single grid cell along the X and Y axes (unit: m); , Potential target area Minimum and maximum coordinates on the X-axis of the BEV plane; , These are the minimum and maximum coordinates of the Y-axis; The grid dimension is fixed and its value is set to ensure that the feature dimensions of different RoIs are consistent.

[0160] Define the neighborhood range based on the edge length of any grid cell (with the grid center as the origin and the edge length as the radius). The circular region is aggregated using a PointNet structure to fuse BEV features within its neighborhood, resulting in the aggregated features of the corresponding grid. The calculation formula is:

[0161] ;

[0162] in, For grid Aggregation features ( (Number of grid feature channels); To integrate BEV features in BEV coordinates Feature vector (dimension) at ); For grid The neighborhood range (all BEV coordinates within the circular area) ); This is a max pooling operation (preserving the most salient features in the neighborhood, such as the target edge); It is a multilayer perceptron (containing two fully connected layers, with ReLU activation function, which maps the pooled features to 64 dimensions).

[0163] The aggregated features of all grids are spliced ​​along the channel dimension to obtain the basic features of the corresponding potential target regions; a feature set is constructed based on the basic features of all potential target regions.

[0164] Furthermore, it outputs accurate 3D detection results, including:

[0165] Each basic feature in the feature set is passed through two fully connected layers, outputting the corresponding target classification confidence and initial bounding box correction; the calculation formula is:

[0166] ;

[0167] ;

[0168] in, For the classification confidence vector ( (The target category is 3 categories, such as vehicles, pedestrians, and cyclists). Indicates that RoI belongs to the first The probability of a class; , A fully connected layer for the classification branch (input dimension 3136, intermediate dimension 1024, output dimension...) ); To perform the normalization operation, ensure that the sum of the confidence levels is 1; This refers to the bounding box correction (corresponding to the 6 parameters of the 3D bounding box: center coordinates). Length, width and height Orientation Angle ); , The fully connected layer for the regression branch (input dimension 3136, intermediate dimension 1024, output dimension 6). It serves as an activation function to enhance the nonlinear representation of features.

[0169] Basic features with target classification confidence scores below the confidence threshold are removed, and the initial bounding boxes are corrected based on the initial bounding box correction amount to obtain a coarsely optimized feature set;

[0170] The point cloud density within any retained basic feature in the coarsely optimized feature set is calculated by kernel density estimation. Each point cloud density is then mapped to a position code that matches the dimension of the corresponding basic feature. The position code is then multiplied element-wise with the corresponding basic feature to obtain the density-aware basic feature. The density-aware basic feature is then passed through a self-attention layer to output the density-aware region of interest feature.

[0171] Based on the spatial extent of the potential target region corresponding to the density-aware region of interest features and the number of point clouds contained therein, the corresponding second point cloud density is calculated. The calculation formula is: , The density-aware region of interest features correspond to the number of point clouds contained in the potential target region. , , The spatial range of the potential target region corresponding to the density-aware region of interest features is represented; the density-aware region of interest features with cloud density less than the density threshold at the second point are removed to obtain the density-aware optimized feature set.

[0172] Extract the surrogate features within the potential target region corresponding to the density-aware region of interest features in the density-aware optimized feature set, and aggregate the surrogate features into a single vector by average pooling. Adjust the dimension of the single vector to obtain the dimension-adjusted surrogate features corresponding to the density-aware region of interest features.

[0173] The dimension-adjusted proxy features are concatenated with the corresponding density-aware region of interest features along the channel dimension to obtain the region of interest features with fused semantics.

[0174] The region of interest features of the fused semantics are classified by two fully connected layers to output 3D detection results. The 3D detection results are then processed by a non-maximum suppression algorithm to remove duplicate 3D detection results, resulting in accurate 3D detection results, which include accurate classification confidence and 3D bounding boxes.

[0175] This embodiment also provides robust training and loss functions to ensure the generalization ability and optimization stability of the low-altitude 3D target detection method, specifically including:

[0176] Randomly generate rotation angle around the Z-axis The rotation operation is applied only to the aligned point cloud features, while the image features remain unchanged, thereby simulating the modal space misalignment caused by calibration error.

[0177] With probability A horizontal flip operation is applied to the aligned image features while the point cloud features remain unchanged, thereby simulating modal semantic misalignment caused by data synchronization errors.

[0178] The two types of degradation operations mentioned above are applied only during the forward propagation phase of training. During backpropagation, the loss is still calculated using the original ground truth labels to ensure that the model learns the ability to "recover the true target from inaccurate data". No degradation operations are applied during the testing phase to avoid noise affecting inference accuracy. This design ensures both the robustness of training and the performance of testing.

[0179] The core of the multi-loss function design is to constrain the model's performance in four dimensions—object classification, bounding box localization, cross-modal alignment, and point cloud density perception—through a weighted combination of four types of losses: classification, regression, alignment, and density. This avoids module imbalance caused by a single loss dominating optimization (e.g., optimizing only the classification loss would ignore the accuracy of bounding box localization). The weights of all losses are determined through experimental verification to ensure coordinated convergence of each process. The final total loss is used for backpropagation to update all model parameters (such as parameters in the alignment process, virtual point completion process, and cascaded detection process).

[0180] 1. Classification Loss: Constraining RoI classification accuracy:

[0181] The classification loss uses the cross-entropy loss function to optimize classification confidence, ensuring the model can accurately distinguish target categories (such as vehicles, pedestrians, and cyclists) from the background. The smaller the classification loss value, the higher the classification accuracy.

[0182] 2. Regression Loss: Constrained Bounding Box Positioning Accuracy:

[0183] The regression loss uses a 3D IoU loss function to optimize the 3D bounding box localization accuracy, addressing the issue of traditional L1 / L2 loss being sensitive to bounding box scale (e.g., small target bounding boxes have a larger error proportion). The smaller the regression loss value, the more accurate the bounding box localization.

[0184] 3. Alignment Loss: Constraining cross-modal feature consistency.

[0185] The alignment loss employs the mean squared error (MSE) loss function to optimize cross-modal alignment, ensuring spatial-semantic consistency between point cloud features and image features (avoiding feature misalignment due to modal heterogeneity). The smaller the alignment loss value, the stronger the cross-modal feature consistency.

[0186] 4. Density loss: Constraining the accuracy of the density sensing module:

[0187] The density loss uses the mean squared error (MSE) loss function to optimize the virtual point completion and density perception effects, ensuring that the model's estimation of the point cloud density within the RoI (Region of Interest) is consistent with the true density (avoiding false detections of hard samples due to density estimation bias).

[0188] 5. Total Loss: Weighted combination achieves collaborative optimization:

[0189] The total loss is a weighted sum of the losses from the four classes. By setting appropriate weights, we avoid a single loss class dominating the optimization (e.g., over-optimizing the regression loss can lead to a decrease in classification accuracy). Referring to the optimal weights verified in experiments, the total loss... The formula for calculation is:

[0190] ;

[0191] The weight values ​​are... , , , . This represents the total loss used for backpropagation (the smaller the value, the better the overall model performance). The weights are for classification, regression, alignment, and density loss, respectively. The weights are set according to the principle of "localization accuracy first (regression weight is the highest), basic alignment second (classification and alignment weights are moderate), and density assistance (density weight is the lowest)". The total loss updates all trainable parameters of the model (such as attention weights, virtual point completion network parameters, and fully connected network parameters) through the backpropagation algorithm to achieve coordinated convergence of each process.

[0192] The low-altitude 3D target detection method based on multimodal information fusion provided in this embodiment has the following beneficial effects:

[0193] This method constructs a "full-process, multi-dimensional" solution through technological fusion: It uses a "LiDAR-led, camera-assisted" paradigm as its core, combining spatial alignment and semantic enhancement to eliminate modal heterogeneity (adapting to offsets caused by low-altitude vibrations). It addresses point cloud sparsity through color supplementation, small target adaptation and completion, and density mitigation (targeting small drones and birds). It collaboratively improves hard sample accuracy using cascaded detection and density perception (addressing overlapping occlusions and distant targets), and strengthens robustness through degradation simulation (such as simulating vibration offsets and cloud / fog noise) and multi-loss constraints (adapting to airborne sensor characteristics). This fusion is not a simple superposition, but follows a process logic of "data augmentation → alignment extraction → enhancement completion → fusion detection → robust training," allowing various technologies to complement and enhance each other. Ultimately, it overcomes the performance bottlenecks of single-modal and simple multi-modal fusion, meeting the core requirements of low-altitude situational awareness systems for 3D target detection: "high precision, high robustness, and high efficiency."

[0194] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0195] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A low-altitude 3D target detection method based on multimodal information fusion, characterized in that, include: S1: Acquire multimodal data containing LiDAR point cloud, RGB image and camera calibration parameters, preprocess the multimodal data to obtain enhanced point cloud and RGB image, and generate virtual points; S2: Align the enhanced point cloud and RGB image and extract features from them to obtain aligned point cloud features and image features; Point cloud features are completed based on virtual points, image features, and camera calibration parameters, and surrogate features are sampled. Sampling agent features includes: Points with depth values ​​less than the depth threshold in the dense depth map are removed to obtain point cloud proxy points; The dense depth map is voxelized, and the center coordinates of each non-empty voxel are taken as voxel surrogate points. A fixed-size grid is divided into the bird's-eye view plane, and the coordinates of the grid nodes are taken as grid proxy points; For any point cloud proxy point, voxel proxy point, or mesh proxy point, obtain the corresponding pixel coordinates through the camera perspective projection formula, and take the four nearest integer pixels around the pixel coordinates. Perform bilinear interpolation based on the elements corresponding to the four integer pixels in the image features to obtain the proxy features of the corresponding proxy point. S3: Map the completed point cloud features and image features onto the bird's-eye view to obtain point cloud BEV features and image BEV features respectively; based on preset 2D detection cues, filter out potential target regions within the bird's-eye view plane; within the potential target regions, fuse point cloud BEV features, image BEV features, and 2D detection cues to obtain fused BEV features, including: The center coordinates of each non-empty voxel in the bird's-eye view plane are used as the mean of the Gaussian distribution of the corresponding voxel. The covariance is calculated based on the size and orientation of the target in the preset 2D detection cues; Based on the mean and covariance of the Gaussian distribution of voxels, calculate the Gaussian score of the target contained in the corresponding voxel; Voxels with Gaussian scores greater than the Gaussian score threshold are merged into continuous regions to obtain potential target regions. Within the potential target region, the point cloud BEV features and the image BEV features are concatenated along the channel dimension to obtain the concatenated features; the number of channels of the concatenated features is compressed to a fixed dimension through 2D convolution to obtain the compressed features. Global average pooling is performed on each channel of the compressed feature to obtain the channel descriptor for each channel; the channel descriptor is then passed through two layers of multilayer perceptron and a sigmoid activation function to generate the corresponding channel weights. The channel weights of each channel are multiplied one by one with the channel weights of the compressed feature to obtain the dynamic fusion feature; The pseudo-labels are converted into category probability vectors through one-hot encoding, and the category probability vectors are mapped onto the bird's-eye view to obtain the BEV semantic probability features; The pixel coordinates of the 2D bounding boxes in the 2D detection cues are converted into the 3D bounding box range of the bird's-eye view, and the bounding box features of each 2D bounding box are mapped to the corresponding region in the bird's-eye view to obtain the BEV-2D detection features. The dimensions of both the BEV semantic probability feature and the BEV-2D detection feature are adjusted to a fixed dimension. The dynamically fused feature is then added element-wise to the BEV semantic probability feature and the BEV-2D detection feature with the adjusted dimensions to generate the fused BEV feature. S4: Extract basic features from each potential target region to construct a feature set. Perform a three-stage cascaded optimization on the feature set based on the fused BEV features, proxy features, and the completed point cloud features, outputting the region of interest features, including: Each basic feature in the feature set is passed through two fully connected layers to output the corresponding target classification confidence and initial bounding box correction. Basic features with target classification confidence scores below the confidence threshold are removed, and the initial bounding boxes are corrected based on the initial bounding box correction amount to obtain a coarsely optimized feature set; The point cloud density within any retained basic feature in the coarsely optimized feature set is calculated by kernel density estimation. Each point cloud density is then mapped to a position code that matches the dimension of the corresponding basic feature. The position code is then multiplied element-wise with the corresponding basic feature to obtain the density-aware basic feature. The density-aware basic feature is then passed through a self-attention layer to output the density-aware region of interest feature. Based on the spatial range and number of point clouds of the potential target region corresponding to the density-aware region of interest features, the corresponding second point cloud density is calculated, and the density-aware region of interest features with a second point cloud density less than the density threshold are removed to obtain the density-aware optimized feature set. Extract the surrogate features within the potential target region corresponding to the density-aware region of interest features in the density-aware optimized feature set, and aggregate the surrogate features into a single vector by average pooling. Adjust the dimension of the single vector to obtain the dimension-adjusted surrogate features corresponding to the density-aware region of interest features. The dimension-adjusted proxy features are concatenated with the corresponding density-aware region of interest features along the channel dimension to obtain the region of interest features with fused semantics. The system classifies the features of the region of interest and outputs accurate 3D detection results.

2. The low-altitude 3D target detection method according to claim 1, characterized in that, Methods for generating virtual points include: The LiDAR point cloud and RGB image are respectively processed by an encoder to extract the spatial geometric features of the point cloud and the texture semantic features of the image. Spatial geometric features and texture semantic features are input into the decoder to obtain a depth map with the same resolution as the RGB image. The image domain pixel coordinates of each pixel in the depth map are converted into 3D spatial coordinates using the camera perspective projection formula to obtain the 3D spatial coordinates of each virtual point; the three color channel values ​​of the corresponding pixels in the RGB image are extracted, and semantic labels for each virtual point are generated by combining the foreground target annotation.

3. The low-altitude 3D target detection method according to claim 2, characterized in that, Preprocessing of multimodal data includes: The SegFormer model, pre-trained on the Cityscapes dataset, is used to generate pseudo-labels for RGB images; The projection range of the true value of the 3D bounding box onto the RGB image is calculated using the camera perspective projection formula, resulting in a rectangular region containing the complete outline of the target. Extract the pixels, LiDAR point cloud, and virtual points corresponding to the pseudo-label category "moving target" within the rectangular area; The intersection-union ratio (IUU) is used to remove overlapping bounding boxes from the ground truth values ​​of all 3D bounding boxes. Then, the pixels, LiDAR point cloud, and virtual points corresponding to the pseudo-label category "moving target" within the rectangular area corresponding to the ground truth value of each retained 3D bounding box are extracted to obtain the enhanced point cloud, the enhanced RGB image, the semantic label, and the ground truth values ​​of the 3D bounding boxes. The enhanced point cloud includes LiDAR point cloud and virtual points.

4. The low-altitude 3D target detection method according to claim 3, characterized in that, The enhanced point cloud and RGB image, before alignment, also include: The sparse depth map of the enhanced point cloud and the enhanced RGB image are processed by a depth map encoder and a depth map decoder to map into a dense depth map. The calculation formula is as follows: ； in, Represents a dense depth map. This represents a depth map decoder. This represents a depth map encoder. This represents a sparse depth map of the enhanced point cloud. This represents the enhanced RGB image; the sparse depth map is the geometric projection of the enhanced point cloud onto the image domain. The enhanced LiDAR point cloud or virtual points are projected onto the image domain pixel coordinates using the following formula: ； in, Represents the pixel coordinates in the image domain. Represents pixels The depth value represents the camera intrinsic parameter matrix in the camera calibration parameters, and the rotation matrix from the 3D spatial coordinate system to the camera coordinate system in the camera calibration parameters is also represented. This represents the translation vector in the camera calibration parameters. Represents the 3D spatial coordinates of the enhanced LiDAR point cloud or virtual points; Set the depth value of the enhanced LiDAR point cloud or virtual point to the depth value of the corresponding pixel in the dense depth map; Preliminary features of the enhanced RGB image are extracted based on the ResNet-18 model, and fine-grained texture features and medium-grained contour features are extracted from the preliminary features using convolution kernels of different sizes. The fine-grained texture features and medium-grained contour features are then concatenated to obtain a local multi-scale feature map. The dense depth map is linearly transformed into a query matrix, and the local multi-scale feature map is linearly transformed into a key matrix and a value matrix. Attention weights are calculated based on the query matrix and the key matrix. The attention weights are multiplied by the value matrix and then added to the local multi-scale features. The result of the addition is normalized by layers to obtain the globally aligned semantic features of the image.

5. The low-altitude 3D target detection method according to claim 4, characterized in that, The aligned point cloud features and image features are obtained, including: The globally aligned image semantic features are processed through ResNet-50, and the output features of the final stage are used as the corresponding basic semantic features. Positional attention enhancement is then applied to the basic semantic features, calculated as follows: ； ； in, This represents the features after enhanced positional attention. This represents the positional attention weight matrix. Represents basic semantic features. This represents the softmax function. Indicates transpose. Represents the L2 norm; Channel attention enhancement is applied to the features enhanced with positional attention, and the calculation formula is as follows: ； ； ； in, This represents the features after channel attention enhancement. This represents the channel attention weight matrix. This represents element-wise multiplication. This represents the Sigmoid activation function. Denotes the first linear transformation matrix. Denotes the second linear transformation matrix. Indicates the first bias term. Indicates the second bias term. Represents the ReLU activation function. Indicates global average pooling. Indicates pooling characteristics; The channel attention-enhanced features are mapped to a fixed dimension to obtain aligned image features; The farthest point sampling method is used to select several key points from the enhanced point cloud. For any key point, a neighborhood is defined with a fixed radius, and local geometric features are extracted using the PointNet model. The calculation formula is as follows: ； in, Indicate key points The local geometric features represent the multilayer perceptron, and the feature concatenation represents the feature stitching. Indicate key points any point in the neighborhood of , Indicate key points The neighborhood; Based on local geometric features and dense depth maps, the point cloud features for keypoint alignment are calculated using the following formula: ； ； ； in, Indicate key points Aligned point cloud features, Indicate key points Associative deep semantic features Indicates bilinear interpolation. Representing depth map features, Indicate key points The corresponding pixel coordinates Represents a dense depth map. It represents a linear transformation.

6. The low-altitude 3D target detection method according to claim 1, characterized in that, Point cloud features are completed based on virtual points, image features, and camera calibration parameters, including: The 3D spatial coordinates of each point in the point cloud feature are converted into pixel coordinates, and a color mapping table for the image domain is constructed based on the color information contained in the virtual points. If the pixel corresponding to a point is not covered by any virtual point, then the average color of all the virtual points covering the pixel in the neighborhood of the point is assigned to the corresponding point; otherwise, the color of the virtual point is matched from the color map table and assigned to the corresponding point. Denoise all virtual points to obtain a denoised set of virtual points; The space containing the dense depth map is divided into several cubic sub-regions with a side length of 1m, and the point cloud density in each cubic sub-region is calculated. When the point cloud density is less than the set threshold, the corresponding cube sub-region is determined to be a sparse region. The virtual point requirement of the sparse region is calculated based on the difference between the point cloud density and the set threshold. Several denoised virtual points closest to the center of the corresponding cube sub-region are selected from the set of denoised virtual points to fill the sparse region. Traverse all sparse regions and fill them to obtain the completed point cloud features.

7. The low-altitude 3D target detection method according to claim 1, characterized in that, Basic features are extracted from each potential target region to construct a feature set, including: Divide any potential target area into several grids of the same size, and calculate the spatial range of the bird's-eye view corresponding to the grid based on the side length of the grid. The neighborhood range is defined based on the side length of any grid, and the fused BEV features within the neighborhood range are aggregated through the PointNet structure to obtain the aggregated features of the corresponding grid. The aggregated features of all grids are spliced ​​along the channel dimension to obtain the basic features of the corresponding potential target regions; a feature set is constructed based on the basic features of all potential target regions.

8. The low-altitude 3D target detection method according to claim 1, characterized in that, Output accurate 3D detection results, including: The region of interest features of the fused semantics are classified by two fully connected layers to output 3D detection results. The 3D detection results are then processed by a non-maximum suppression algorithm to remove duplicate 3D detection results, resulting in accurate 3D detection results, which include accurate classification confidence and 3D bounding boxes.

Images