Airborne laser radar point cloud three-dimensional target detection method based on terrain guidance

Abstract

The invention discloses an airborne laser radar point cloud three-dimensional target detection method based on terrain guidance, and belongs to the technical field of three-dimensional target detection. The objective of the invention is to solve the problem that the existing three-dimensional target detection does not consider the influence caused by terrain complexity and target-background similarity, so that the detection effect needs to be improved. According to the invention, based on the VoxelNext network, a multi-stage terrain guidance module is introduced to predict terrain distribution and guide the processes of height compression, region-of-interest generation and proposal prediction, and effective combination of terrain modeling and pipeline detection is realized; according to the method, voxels are randomly sampled and grouped through a sparse feature enhancement module, middle layer features are aggregated, and two auxiliary tasks of foreground segmentation and voxel up-sampling are introduced to enhance the feature expression ability of the network.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional target detection, and in particular relates to a three-dimensional target detection method for airborne laser radar point clouds. Background Art

[0002] 3D object detection is a key downstream task of LiDAR (LiDAR), aiming to identify and locate objects of interest in a 3D scene. Airborne LiDAR (ALS) can penetrate tree canopies and acquire high-precision 3D spatial information. 3D object detection based on this data has broad potential in areas such as terrain mapping, personnel rescue, and battlefield reconnaissance. Existing 3D object detection algorithms are designed for vehicle-based scenarios and fail to consider the impact of terrain complexity and object-background similarity. Consequently, direct application of ALS point cloud detection has limited effectiveness. Summary of the Invention

[0003] The present invention aims to solve the problem that existing three-dimensional target detection does not consider the influence brought by terrain complexity and target-background similarity, so that the detection effect needs to be improved.

[0004] A terrain-guided three-dimensional target detection method based on airborne laser radar point cloud, comprising: First, airborne LiDAR point cloud data is acquired and a digital elevation model (DEM) is generated based on the point cloud data. The ground truth of each detection scene is then obtained based on the geographic coordinates. Detection is then performed using a detection network, which includes an encoder and detection head, as well as a multi-level terrain guidance module. The encoder and detection head adopt the VoxelNext design. The encoder uses sparse residual blocks for feature extraction, and the detection head is used for object detection. The multi-level terrain guidance module predicts the scene's terrain distribution and guides the multi-stage detection process, and inputs the prediction results into the detection head. The specific process is as follows: The multi-level terrain guidance module includes a terrain height prediction process and a multi-level guidance process based on the prediction results; Terrain height prediction is achieved by accessing a branch sub-network in the middle layer of the encoder, and the middle layer features extracted by the encoder are Separate from the computation graph, and then Extracting intermediate layer features between input and encoder The sparse residual block with the same structure as the subsequent sparse residual block extracts Features of the same size are denoted as ; is the feature in the encoder 、 Expand to The features obtained by matching the spatial dimensions and summing them at the corresponding positions, features 、 、 The middle layer features The features obtained by the three subsequent sparse residual blocks; The features The non-zero parts are accumulated along the height dimension and mapped to the BEV feature map; finally, the BEV feature map is processed by a two-dimensional sparse residual block to generate the terrain prediction result With index ; Get the terrain prediction for the scene Afterwards, As the target distribution center, the distribution offset m is a priori, giving the location feature the maximum weight, and using a Gaussian distribution to calculate the elevation weight: in, is the variance, represent The weight of the i-th voxel in , represent The index of the i-th feature; Based on the elevation weight, the elevation attention Weighted accumulation along the vertical direction to obtain the terrain-weighted BEV feature map ;Will After a layer of sparse convolution and Connect in series along the channel dimension, feed into sub-manifold convolution to complete feature fusion, and output fusion bird's-eye view features , Bird's-Eye View Features Send it into the detection head to complete target detection.

[0005] Furthermore, the process of obtaining a digital elevation model (DEM) based on point cloud data includes: During the point cloud data acquisition process by the airborne lidar, all point clouds are aggregated and the ground points are classified using the IPTD method: first, a sparse triangulated network is constructed using the initial seed points. Then, in an iterative process, the point cloud is encrypted layer by layer. The addition of new points to the triangulated network is determined based on the distance and angle thresholds until the ground point classification is complete. The ground point cloud is then processed using the irregular triangulated network interpolation method, and grid cell values are extracted from multiple triangular surfaces composed of adjacent points to generate a digital elevation model (DEM).

[0006] Furthermore, the detection network also includes an SFE module; the SFE module includes a feature aggregation process and an auxiliary task prediction process; The feature aggregation process includes: Features extracted by the encoder 、 、 ,feature The middle layer features The features of the first sparse residual block corresponding to the sparse residual block are obtained based on the reverse distance weighted average method. 、 、 Corresponding voxel feature coordinates, interpolate them to the point set , and interpolate features in series along the channel dimension, and generate aggregate features through a linear layer ; The auxiliary task prediction process includes two auxiliary tasks: foreground segmentation and point cloud upsampling: The foreground segmentation task is processed using a multi-layer perceptron , output voxel-level segmentation results ; The point cloud upsampling task is to convert It is connected in series with its maximum pooling feature and input into the multi-layer perceptron to achieve feature expansion; then the feature is transformed into Point sets of the same size, output upsampling results ; When adding samples to the network training process, the sampling point is taken as the center, and its neighborhood voxels are queried and grouped to obtain a voxel set.

[0007] Furthermore, the aggregation feature as follows: in, Represents the set of points before interpolation A coordinate point in For The coordinates of the nearest k sparse voxels, Belongs to The elements in the feature set represented by the nearest k sparse voxels; Indicates the Layer feature pairs The interpolation result of represents a linear layer, Represents concatenation.

[0008] Furthermore, the detection network also includes a refinement network, which is the refinement network part in Voxel-RCNN, including the step of voxel region of interest pooling processing; the SFE module is integrated with the refinement network, the SFE module outputs the segmentation result and the upsampling prediction, and the segmentation result is mapped to the upsampled voxel set by interpolation to form a feature set ; followed by The segmentation results and voxel coordinates are used as features, and the block encoding of the region of interest is realized according to the detection head proposal information, that is, point pooling, to obtain the The pooled features of the detection head are as follows; the detection head proposal information is the detection result of the detection head; The features generated by pooling the voxel region of interest are then concatenated with the pooled features along the channel dimension and input into the subsequent detection head for refined prediction.

[0009] Furthermore, the block encoding of the region of interest is implemented according to the detection head proposal information, and then the features generated by pooling the voxel region of interest are concatenated with the pooled features along the channel dimension, including the following steps: After receiving the proposal information, the refinement network divides the three-dimensional cube space circled by the proposal into k*k*k small cube spaces. According to the center of each small cube space, the neighborhood features are queried and encoded from the features extracted by the encoder to obtain the feature A of each small space; at the same time, the center point of the proposal is used as the sampling point, and the SFE module is used to obtain the segmentation and upsampling results of its neighborhood. Then, according to the center of each small cube space, the upsampled neighborhood voxels containing the segmentation information are queried and encoded to obtain the feature B of each small space. After A and B are concatenated along the feature dimension, they are input into the prediction head for processing.

[0010] Furthermore, the detection network is pre-trained, and during the training process, the auxiliary tasks participate in the training process, but the auxiliary tasks do not participate in the processing during the actual three-dimensional object detection process.

[0011] Furthermore, when the detection network is not provided with a refinement network, feature aggregation of the SFE module does not participate in target detection during the actual three-dimensional target detection process.

[0012] Furthermore, the total loss function of the detection network during training includes: The loss function corresponding to the branch sub-network corresponding to terrain height prediction : in 、 Represents two horizontal coordinate axes, 、 represents the index of the i-th feature of the DEM, 、 represents the index of the i-th feature prediction, 、 represents the feature step size, 、 represents the resolution of the voxel, 、 represents the resolution of the DEM, 、 Represents the size of the detection space, 、 Represents the center index of DEM, Represents the number of features in the bird's-eye view; In the detection proposal generation phase, we use For reference, the difference between the center of the candidate box and the corresponding ground height prediction is integrated into the loss function to explicitly constrain the vertical height of the candidate box; the loss function is defined as follows; in and Respectively The center height and box height of the candidate box predicted by the i-th sparse voxel in; During the network training process, the loss function of the foreground segmentation task adopts focal loss, and the supervision label Provided by the three-dimensional bounding box, the loss function of the point cloud upsampling task adopts the chamfer distance loss, using The coordinate part of is used as supervision; the joint loss function of the two auxiliary tasks is defined as follows: in, represents the focal loss, represents the chamfer distance loss, stands for rejection loss.

[0013] Furthermore, the training data samples of the detection network during training are samples added using the AADA strategy. The specific process includes: For the airborne lidar point cloud data sample library, select samples from the sample library and perform random offset, rotation and collision detection to obtain the sample point cloud and bounding box Based on the digital elevation model, the point cloud elevation is located to determine the ground height of the target placement area. The elevation of the sample center point is then determined in combination with the bounding box height, and the ground points below are removed. The segmented point cloud data is recorded as a scene, and the digital elevation model corresponding to the scene is determined. Adaptive rotation based on position: Divide the bounding box sample into four equal-sized target blocks along the length and width of the bounding box. Rotate the samples by an integer multiple of , so that the block with the largest point cloud density faces the center of the scene; Then take a sample : Construct a spherical polar coordinate system with the drone's location as the origin , where r is the distance from a point to the origin, is the azimuth from the x-axis in the xy plane, Is the polar angle starting from the positive z axis; and transform the background point cloud and the upper four corner points of each target block into the spherical polar coordinate system; query the background point cloud The point cloud set that falls within the area enclosed by the corner points and the origin of each target block : in, Represents the points in the background point cloud set, is its spherical polar coordinate; and are the minimum and maximum azimuths of the corner points, and are the minimum and maximum polar angles of the corner points, is the minimum distance from the origin to the corner point; Calculation point set The mean distance And the area of the sphere at that position in the spherical coordinate system : Further obtain the point cloud density And the sampling rate of the blocks ,in Representative Set Number of internal point clouds; It is a constant, the default value is 1000; Get point cloud density And the sampling rate of the blocks for sampling.

[0014] Beneficial effects: The present invention establishes a terrain-guided airborne lidar point cloud 3D target detection network, which can effectively reduce the impact of terrain complexity and target-background similarity, thereby improving the detection effect. Both the lightweight TG-ADet and the TG-ADet-R with the added refinement stage achieved the best detection performance under the three difficulty indicators. In the most challenging difficult sample detection, the former and The corresponding indicators of the latter are 69.46% and 47.93% respectively. At easy difficulty, they increased by 7.6% and 12.59% respectively; at medium difficulty, they increased by 5.71% and 9.8% respectively; at hard difficulty, they increased by 5.38% and 8.16% respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of the integrated data augmentation method architecture for adaptive sample generation and enhancing terrain diversity, where (a) is the adaptive ALS data augmentation strategy and (b) is the terrain distribution adjustment strategy.

[0016] Figure 2 This is a diagram of the TG-Adet (TG-Adet-R when seamless integration is performed) network structure.

[0017] Figure 3 Schematic diagram of the sparse feature enhancement module structure.

[0018] Figure 4 Schematic diagram of the integrated structure of the SFE module and the refining network.

[0019] Figure 5 This is a schematic diagram of the multi-level terrain guidance module structure.

[0020] Figure 6 Visualization of the predicted 3D bounding box in a certain L2 dataset, where (a) is the true value, (b) VoxelNext (baseline) result, and (c) is the TG-ADet result. DETAILED DESCRIPTION

[0021] To address the challenges presented in the prior art, we designed the first 3D object detection network, TG-Adet (TG-Adet-R), for ALS point clouds. To adapt to complex terrain distributions and suppress background interference, a multi-level terrain guidance (MTG) module predicts terrain distribution and guides height compression, region of interest generation, and proposal prediction, effectively integrating terrain modeling with the detection pipeline. To improve object-background differentiation, a sparse feature enhancement (SFE) module randomly samples and groups voxels, aggregates intermediate features, and introduces two auxiliary tasks, foreground segmentation and voxel upsampling, to enhance the network's feature representation capabilities. To generate training samples that better reflect the true distribution and verify the algorithm's terrain robustness, an integrated data augmentation method combining the adaptive ALS data augmentation (AADA) strategy and the terrain distribution adjustment (TDA) strategy was designed. This is explained below with reference to specific implementations.

[0022] Specific implementation method 1: Combination Figure 2 To explain this embodiment, This embodiment is a method for detecting three-dimensional targets using airborne lidar point clouds based on terrain guidance, including: Step 1: During the network training phase, ground elevation positioning and dynamic sample optimization are used to add samples that conform to the real distribution to the training data. During the network testing phase, a terrain distribution adjustment strategy is used to introduce Perlin noise to the terrain to verify the terrain robustness of the algorithm (this is an optional operation). The specific process is as follows: During the training process, the AADA strategy is used to add samples that conform to the true distribution to the training data. The AADA strategy is as follows: Compared with MLS point clouds, ALS point clouds present complex terrain distribution and object occlusion relationships, making it difficult for traditional methods (such as GT-PASTE) to add samples that conform to the real distribution in the scene. Therefore, the present invention proposes Figure 1 The AADA strategy shown in (a) aims to construct an adaptive sample generation strategy based on ALS sampling features. In order to ensure the representativeness of the enhanced samples, objects with insufficient points are removed from the sample library, and only samples with rich information are retained. First, the GT-PASTE process is used to select samples from the sample library and perform random offset, rotation, and collision detection to obtain a sample point cloud. and bounding box (i.e. the true bounding box value). Then, in order to match the relative relationship between the target and the scene, an elevation positioning method based on the digital elevation model (DEM) is designed. For point cloud elevation positioning, specifically, the offset coordinates of the sample are used to query the DEM to determine the ground height of the target placement area, and then the elevation of the sample center point is determined in combination with the bounding box height, and the ground points below are removed.

[0023] The sample library contains the point clouds (samples) of all targets in the training set and the bounding box labels of the samples (length, width, height, direction, and category of the target), and the height of the bounding box is known. This implementation method considers that all point clouds below the bounding box are ground points. The reason is that the elevation positioning was previously performed based on the DEM, and the ground height of the added sample area was estimated. Occlusion will cause holes in the ground point cloud, affecting the estimation of the ground height, but because the scene is repeatedly collected by drones in multiple strips, even if the ground is obscured, the point cloud will not be missing too seriously, and a more accurate DEM can still be fitted. The data targeted by the subsequent algorithm is the lidar point cloud in a short period of time (although we have multiple strips of point clouds, only a short period of time is captured each time to simulate real-time processing conditions). The point cloud data divided by time is recorded as a scene to determine the digital elevation model corresponding to the scene.

[0024] Next, to address the self-occlusion problem of the target, a position-based adaptive rotation strategy was further designed. Specifically, the bounding box sample is divided into four target blocks of equal size along the length and width of the bounding box. The samples are rotated by an integer multiple of , and the block with the largest point cloud density is oriented toward the center of the scene (i.e., the position of the drone), thereby more realistically simulating the actual observation characteristics of the lidar.

[0025] In addition, considering the negative correlation between point cloud density and occlusion rate, a block sampling strategy based on occlusion rate is designed. The specific process includes: According to the horizontal size of the scene And the lateral field of view of the lidar Can estimate the flight altitude of the drone : The vertical dimension is the flight direction of the drone, and the horizontal dimension reflects the field of view of the lidar point cloud. Since the field of view of the device is known, the flight altitude at this time can be calculated. With the location of the drone as the origin, a spherical polar coordinate system is constructed. , where r is the distance from a point to the origin, is the azimuth from the x-axis in the xy plane (horizontal plane), It is the polar angle starting from the positive z-axis (perpendicular to the xy plane); and the background point cloud (point cloud outside the bounding box) and the upper four corner points of each target block are transformed into the spherical polar coordinate system. The point cloud set that falls within the area enclosed by the corner points and the origin of each target block : in, Represents the points in the background point cloud set, is its spherical polar coordinate; and are the minimum and maximum azimuths of the corner points, and are the minimum and maximum polar angles of the corner points, is the minimum distance from the origin to the corner point.

[0026] Calculation point set The mean distance And the area of the sphere at that position in the spherical coordinate system : Further obtain the point cloud density And the sampling rate of the blocks ,in Representative Set Number of internal point clouds; It is a constant, and the default value is 1000.

[0027] Get point cloud density And the sampling rate of the blocks The strategy is the block sampling strategy .

[0028] In summary, the complete process of AADA can be expressed as the following composite transformation: On the left are the digital elevation model of the scene obtained by AADA, the 3D bounding box label, the added sample point set, and the background point set. These four parts correspond to the input. The final result of AADA is to add samples that conform to the real distribution to the scene.

[0029] After training is complete, the TDA strategy is used to add samples that match the true distribution during testing. The TDA strategy is as follows: Due to the limitations of experimental conditions, the terrain changes in the scene are relatively gentle, which makes it difficult to reflect the adaptability of the algorithm to complex terrain and environmental changes. Therefore, this embodiment also designs a TDA strategy based on Perlin noise to test the anti-interference ability of the algorithm. Figure 1 As shown in (b) of the figure. Specifically, based on the Perlin noise generation algorithm, a noise map with the same size as the scene's digital elevation model (DEM) is generated. The scene's DEM is then overlaid with the generated Perlin noise, and the noise value is used as an offset to adjust the elevation of the point cloud and the center of the bounding box within 5 meters of the ground surface. This random, smooth noise can generate a more natural and continuous terrain surface, effectively simulating complex and varied terrain distributions. This method can significantly increase the diversity and complexity of scenes and can be used to verify the robustness of the algorithm in complex terrain environments and evaluate its detection performance under irregular surface conditions.

[0030] Step 2: For samples that have been added by AADA in step 1 and conform to the real distribution, the detection network is used for detection. During this process, the sparse feature enhancement module is used to obtain voxel sets, aggregate intermediate layer features, and introduce auxiliary tasks. The specific process is as follows: For targets of interest, especially those that are severely occluded or small in size, their point clouds are not only sparse but also similar to background features. This places higher demands on the network's feature extraction capabilities, and it is necessary to capture the fine-grained features of sparse targets to improve the detection performance of the algorithm.

[0031] Design as Figure 3 The SFE module, shown here, addresses these issues. By extracting features from intermediate layers and combining them with auxiliary tasks, it helps the network obtain richer and more fine-grained shared feature representations. Local semantic and upsampling auxiliary tasks enhance the network's ability to express features for sparse objects. The SFE module can also be seamlessly integrated with the network refinement process to further improve detection accuracy. The SFE module's functionality includes enhancing the encoder's feature representation (auxiliary tasks) and integrating with the refined network.

[0032] The SFE module can be divided into two parts: feature aggregation and auxiliary task prediction: Feature aggregation is used to sample and group scene voxels, aggregate the intermediate layer features and then input them into the task prediction head. Compared with the method of using all voxels to aggregate features, the advantages of this sampling-grouping strategy are: 1) It can better highlight local target features and provide more representative training samples for the network. 2) It can effectively reduce the data size and reduce the training cost. 3) It balances the proportion of foreground and background, helping the network to better learn the difference between the target and background. Specifically, before the voxels are input into the network, they are first selected. sampling points, including foreground points, and the rest are background points (default Because the scene contains a large number of sparse objects, to ensure effective training for subsequent upsampling tasks, foreground points are randomly sampled from the center of the bounding box of the samples added by the AADA method. If there are insufficient foreground points, they are supplemented by background points, which are randomly selected from scene voxels using the farthest point sampling algorithm. The default number of foreground and background points is 6, but during training, fewer than 6 samples may be added. In this case, the number of background points must be expanded to ensure a total of K.

[0033] Next, with the sampling point as the center, query its neighboring voxels and group them (the default query radius is 4, the neighboring sampling point The default is 1024), get the voxel set Then do the following to the collection times random downsampling ( The default value is 4), get the sampling set ,in Finally use Replace the original voxel set in the scene , and input the down-sampling result into the detection network.

[0034] The detection network first uses an encoder to extract features. The encoder and detection head follow the design of VoxelNext and obtain the feature step size of (That is, the convolution step size is 2, and the feature obtained by the next layer of SRB is 1 / 2 of the previous layer) sparse voxel features ,in Represents the encoding stage, and Represent the number of non-empty voxels and feature dimensions respectively. 、 Expand to Match the spatial dimensions and add them at the corresponding positions to get .

[0035] Sparse voxel features extracted by the encoder 、 、 , using the inverse distance weighted average method, according to the coordinates of these voxel features, they are interpolated to the point set , and interpolate features in series along the channel dimension, and generate aggregate features through a linear layer , the calculation process is as follows: in, Represents the set of points before interpolation A coordinate point in For The coordinates of the nearest k sparse voxels (k defaults to 3), Belongs to The elements in the nearest k sparse voxel feature set; Indicates the Layer feature pairs The interpolation result of represents a linear layer, Represents concatenation.

[0036] Based on aggregated features , designed two auxiliary tasks: foreground segmentation and point cloud upsampling. The auxiliary tasks are designed to enhance the feature mining capability of the encoder, so their structure is designed to be as simple as possible to avoid overly complex architectures that reduce the guidance effect on the encoder. The foreground segmentation task is processed using a multi-layer perceptron , output voxel-level segmentation results The loss function uses focal loss and supervised labels. It is provided by a three-dimensional bounding box, that is, the voxels inside it are considered as foreground, and the rest are background.

[0037] Although the segmentation task can obtain fine-grained feature guidance, the background similar to the target often interferes with the accurate candidate box generation. Therefore, the point cloud upsampling task is introduced. By reconstructing the scene outline, the encoder can obtain implicit perception of the ground structure. In order to ensure adaptability to different scenes and better guide the encoder learning, the trained upsampling network weights are not used. Instead, this task is integrated into the detection network to achieve end-to-end training. Specifically, along the channel dimension, It is connected in series with its maximum pooling feature and input into the multi-layer perceptron to achieve feature expansion. Next, the feature is transformed into Point sets of the same size, output upsampling results This task uses The coordinate part of is used as supervision, and the loss function adopts chamfer distance loss.

[0038] The joint loss function of the two auxiliary tasks is defined as follows: in, represents the focal loss, represents the chamfer distance loss, stands for rejection loss.

[0039] It should be noted that the auxiliary task is common to both networks in the TG-ADet model, TG-ADet-R. This task is provided by the SFE (which is its core functionality). While the SFE module provides auxiliary tasks, it can also be combined (i.e., integrated) with the refinement network from the existing Voxel-RCNN method. This integration of the SFE module and the refinement network is known as the "improved refinement network," used to refine detection results. TG-ADet with the refinement network is effectively TG-ADet-R. During training, the auxiliary tasks in both TG-ADet and TG-ADet-R participate in training to enhance network model capabilities. During actual detection, feature aggregation and auxiliary tasks are not involved in TG-ADet processing, while auxiliary tasks are not involved in TG-ADet processing. Feature aggregation from the SFE module is also involved in inference, while the refinement network in TG-ADet-R participates in detection processing.

[0040] In general, TG-ADet is a detection network without a refined network, while TG-ADet-R is a detection network with an “improved refined network”.

[0041] In addition to helping the encoder better express sparse features, the SFE module can also be further integrated with the voxel region of interest pooling in the refinement network (the refinement network of the 3D object detection network Voxel-RCNN) to improve detection accuracy due to its ability to predict the semantics and structure of local regions. It is worth noting that during training, the refinement network needs to cut off the gradient propagation between it and the features obtained by the encoder to prevent interference with the learning of the backbone features. Specifically, if Figure 4 As shown, the network inputs the center point of the proposed 3D bounding box into the SFE module, outputs the semantics (segmentation results) and upsampled predictions of its neighborhood, and maps the semantic predictions to the upsampled voxel set by interpolation to form a feature set. ; Then, the predicted segmentation results and voxel coordinates are used as features to implement block coding (point pooling) of the region of interest according to the proposal information, and the The pooled features are connected in series with the features generated by pooling the voxel region of interest (the block parameters of the two are consistent, the default is 6) along the channel dimension and input into the subsequent detection head for refined prediction. The network with the above-mentioned refinement stage added is named TG-ADet-R to distinguish it from TG-ADet. The detection network corresponding to TG-Adet includes two parts: encoder and detection head (the design of the detection head follows VoxelNext, and the "neck" in VoxelNext is divided into the detection head in this invention). In fact, the detection network in TG-ADet-R is also composed of two parts: encoder and detection head, but it becomes TG-ADet-R after the refinement network is added.

[0042] Voxel region of interest pooling is part of the refinement network designed by Voxel-RCNN (the refinement network includes two parts: "voxel region of interest pooling" and "proposal refinement prediction"). During the network design process, based on the three-dimensional bounding box (proposal) output by the detection head, the designed refinement network further refines these proposals. After receiving the proposal prediction, the refinement network divides the three-dimensional cube space circled by the proposal into k*k*k small cube spaces. According to the center of each small cube space, the neighborhood features are queried and encoded from the features extracted by the multi-layer encoder to obtain the feature A of each small space; and the additional operation of the present invention is to first use the center point of the proposal as the sampling point, and use the SFE module (query-aggregate feature-prediction) to obtain the segmentation and upsampling results of its neighborhood. Then, according to the center of each small cube space, the upsampled neighborhood voxels containing segmentation information are queried and encoded to obtain the feature B of each small space. We concatenate A and B along the feature dimension and input them into the second half of the refinement network (classification head and regression head) to predict the refinement result of the proposal.

[0043] Step 3: The multi-level terrain guidance module predicts the scene's terrain distribution and guides the multi-stage detection process, which is then input into the detection head to obtain the prediction results. The specific process is as follows: Since most targets of interest are located on the ground, good ground perception helps the method better suppress background interference and detect targets. However, the application scenario of vehicle-mounted lidar is flat and regular urban roads, with less background interference in the height direction. Therefore, existing three-dimensional target detection methods often lack targeted terrain modeling strategies. In comparison, the terrain distribution of airborne scenes is more complex, and ground point clouds are often missing due to occlusion, further weakening the conventional detection method's ability to perceive the ground and making it difficult to focus on targets of interest. Therefore, predicting the scene terrain and integrating it into the detection process is a clear and effective solution.

[0044] The first task in predicting terrain is to obtain accurate ground truth. When collecting data in the experiment, the scene was repeatedly scanned by flying a drone in multiple strips to obtain a relatively dense ground point cloud. The present invention summarizes the point clouds of all strips and uses an improved progressive irregular triangulated network encryption (IPTD) method to classify ground points. This method first constructs a sparse triangulated network through the initial seed point, and then encrypts the point cloud layer by layer in an iterative process, and determines whether to add new points to the triangulated network based on the threshold of distance and angle until the ground point classification is completed. Next, the ground point cloud is processed by the irregular triangulated network interpolation method, and the grid cell values are extracted from multiple triangular surfaces composed of neighboring points to generate a resolution of (Default is ) of a large scene digital elevation model (DEM), and then the terrain truth of each detection scene is obtained according to the geographic coordinates.

[0045] Using the ground truth as supervision, the design Figure 5 The MTG module (Multi-Level Terrain Guidance) shown here explicitly predicts terrain and guides the multi-stage detection process. This module helps the network better distinguish valid targets from background clutter in complex airborne scenarios, and to some extent compensates for information loss caused by occlusion. By tightly integrating terrain prediction with the network architecture, the method can more accurately focus on ground areas, highlighting sparse target features and improving detection accuracy.

[0046] The MTG module consists of two parts: terrain height prediction and multi-level guidance based on the prediction results.

[0047] Terrain height prediction is achieved by accessing a branch sub-network in the middle layer of the encoder, such as Figure 5 In order to better adapt the SFE module and avoid the interference of the terrain prediction branch on the encoder optimization process, the encoder needs to extract the Separate from the computation graph. Input the sparse residual block with the same structure as the fourth sparse residual block of the encoder, and extract Features of the same size are denoted as ; Then the feature The non-zero parts are accumulated along the height dimension and mapped to the BEV feature map. Finally, the BEV feature map is processed by a two-dimensional SRB to generate the terrain prediction result. With index The loss function of the terrain prediction branch is defined as follows: in 、 Represents two horizontal coordinate axes, 、 represents the index of the i-th feature of the DEM, 、 represents the index of the i-th feature prediction, 、 represents the feature step size, 、 represents the resolution of the voxel, 、 represents the resolution of the DEM, 、 Represents the size of the detection space, 、 Represents the center index of DEM, Represents the number of bird's-eye view features.

[0048] Get the terrain prediction for the scene After that, it is coupled with multiple stages of the algorithm. First, a height compression strategy based on elevation attention is designed. Traditional height compression methods usually directly accumulate or concatenate features along the vertical direction, ignoring the importance of different height features for the target detection task. To this end, an attention map is constructed based on the relative distance between the voxel and the ground elevation (e.g. Figure 5 As shown in ), the voxels close to the ground occupy a larger proportion in the subsequent stages and are easier to be paid attention to and learned by the network. Specifically, As the target distribution center (the distribution offset m is a priori, assuming 1 as the most common target distribution location), the location feature is given the greatest weight, and then a Gaussian distribution is used to design elevation weights, that is, weights for different locations. Considering the smoothness and distance sensitivity of the Gaussian distribution, it can effectively suppress background interference. The designed elevation weights are as follows: The variance Set to 6, 0.5 represents the center offset of the index, represent The weight of the i-th voxel in , represent The index of the i-th feature.

[0049] Based on the elevation weight, the elevation attention Weighted accumulation along the vertical direction to obtain the terrain-weighted BEV feature map .

[0050] Next will After a layer of sparse convolution and Connect in series along the channel dimension, feed into sub-manifold convolution to complete feature fusion, and output fusion bird's-eye view features ,like Figure 5 Through the above simple and effective operations, the terrain height prediction results are directly integrated into the feature map to support the subsequent detection head detection task.

[0051] In both TG-Adet and TG-Adet-R, bird's-eye view features are fed into the detection head (neck portion). In TG-Adet, after the bird's-eye view image passes through the neck, it is passed through the classification and regression heads to output the first-stage predictions. If the refinement network TG-Adet-R is added, features are extracted based on the first-stage predictions to further refine these predictions.

[0052] Finally, in the proposal generation phase, For reference, the difference between the center of the candidate box and the corresponding ground height prediction is integrated into the loss function to explicitly constrain the vertical height of the candidate box. The loss function is defined as follows: in and Respectively The center height and box height of the candidate box predicted by the i-th sparse voxel in .

[0053] The encoder encodes 3D voxel features, compresses them vertically into 2D bird's-eye view features, and then feeds them into the detection head to predict the target bounding boxes, or proposals. The process here refers to the detection head predicting proposals, generating candidate boxes by predicting the category and regressing the bounding box for each non-empty bird's-eye view feature.

[0054] This method, based on the prior knowledge of the relative position of the target and the terrain, can enhance the semantic connection between the target and the ground, preventing excessive vertical deviation of the candidate boxes. Especially in scenes with complex terrain, it allows the network to better adapt to terrain changes and ensures that the predicted boxes are more closely aligned with the ground.

[0055] Compared with the prior art, the present invention has the following advantages: This method invents the first 3D target detection algorithm for airborne lidar point clouds, which can solve the challenges brought by terrain complexity and target-background similarity problems, thereby achieving accurate positioning and identification of objects of interest in the scene. The performance of the present invention is verified by constructing a data set for experiments, and the experimental results verify the effectiveness of the proposed method.

[0056] 1. Experimental dataset: Harbin L2 Dataset: This dataset was collected by a Matrice 350 RTK drone equipped with a Zenmuse L2 LiDAR from June 26 to 28, 2024, covering campuses and parks in Harbin (collected over two days at different flight altitudes and speeds). The scenes contain five types of objects of interest: vehicles, tents, canopies, large cubes, and small cubes. The drones flew at altitudes ranging from 70 to 150 meters, and the areas of the scenes collected were: , as well as Considering the high real-time and low computational complexity requirements of the airborne platform detection task, the data was segmented according to an accumulation time of 1 second and an inter-frame interval of 0.2 seconds, generating a training set of 3044 samples and a test set of 3024 samples.

[0057] 2. Experimental setup: 1) Evaluation index: In the experiment of this invention, the evaluation index proposed by the KITTI dataset is followed, and the average precision of the bird's-eye view is calculated by 40 values on the precision-recall (PR) curve ( ) and 3D average precision ( The two methods measure the detection performance of the algorithm from a bird's-eye view and in three-dimensional space, respectively. For the Harbin L2 dataset, the intersection-over-union ratio threshold for vehicle targets is set to 0.7, and the thresholds for the other four targets are all set to 0.5.

[0058] To more precisely evaluate the algorithm's detection performance, thresholds were set based on the number of target point clouds, and the algorithm was categorized into three difficulty levels: "Easy," "Medium," and "Difficult." Furthermore, to assess the algorithm's ability to detect occluded targets and verify its adaptability to complex scenarios, targets were categorized into "Unobstructed," "Partially Obstructed," and "Highly Obstructed" based on the canopy density of the target's neighborhood. The canopy density is calculated by calculating the ratio of the number of point clouds above the target's neighborhood to the total number of neighboring point clouds.

[0059] 2) Implementation Details: The method of the present invention is based on the open-source 3D object detection code library OpenPCDet and the baseline method VoxelNext, using CUDA version 10.2 and the deep learning framework PyTorch version 1.12.1-cu102. The network of the present invention was trained using two V100 32GB graphics cards using the DistributeDataParallel strategy. The batch size per card was set to 2, the Adam optimizer was used, the learning rate and weight decay were set to 0.003 and 0.05, respectively, and training was performed for 20 epochs. For the Harbin L2 dataset, the input voxel size was set to [0.1m, 0.1m, 0.1m], the maximum number of voxels and the maximum number of point clouds per voxel were set to 40,000 and 20, and the point range on the X, Y, and Z axes was set to [-114.5, -14.5, 0, 114.5, 14.5, 16] (m).

[0060] 3. Result analysis: 1) Comparative analysis with other methods: As shown in Table 1, to demonstrate the effectiveness of TG-ADet, the experiment compared the detection results of the proposed method, the baseline method VoxelNext, and a variety of excellent LiDAR-based 3D target detection algorithms on the Harbin L2 dataset. According to Table 1, both the lightweight TG-ADet and the TG-ADet-R with the added refinement stage achieved the best detection performance under the three difficulty indicators. In the most challenging difficult sample detection, the former and The corresponding indicators of the latter are 69.46% and 47.93% respectively. At the easy difficulty level, the improvements were 7.6% and 12.59% respectively, at the medium difficulty level, they were 5.71% and 9.8% respectively, and at the hard difficulty level, they were 5.38% and 8.16% respectively. This shows that the average precision of the two methods is significantly higher than that of other methods.

[0061] Table 1 Regarding detection performance for specific categories, although neither algorithm achieves optimal detection accuracy across all categories, they lack significant weaknesses and demonstrate adaptability to diverse object types. In particular, for small, difficult-to-identify objects such as small cubes and tents, the algorithms achieve accuracy far exceeding that of other methods. This outstanding performance is attributed to the synergistic effect of the MTG module, the SFE module, and the AADA data augmentation strategy. The MTG module predicts scene terrain and guides multi-stage detection, enabling the network to better adapt to the complex terrain of airborne scenes. The SFE module introduces auxiliary tasks, enhancing the algorithm's ability to represent sparse features. The AAAS strategy generates high-quality training samples that align with the real-world data distribution, improving detection accuracy while enhancing the algorithm's robustness. Furthermore, compared to TG-ADet, TG-ADet-R further improves the regression accuracy of prediction proposals in three-dimensional space by introducing a refinement stage that integrates auxiliary tasks.

[0062] Figure 6 Comparative visualizations of the ground truth annotations with those of TG-ADet and the baseline method VoxelNext in five scenarios are provided. (a) represents the ground truth, (b) represents the VoxelNext (baseline) result, and (c) represents TG-ADet. Red represents missed detections, and yellow represents false detections. The terrain complexity and target-background similarity in the ALS point cloud lead to a large number of false detections and missed detections in the baseline method. In contrast, our method demonstrates stronger anti-interference capabilities, and the detection results are more consistent with the ground truth annotations.

[0063] 2) Comparative analysis under different canopy densities: Table 2 shows the detection accuracy of the proposed method under different canopy density conditions on the Harbin L2 dataset. It should be noted that some categories do not have corresponding targets under certain canopy density levels. By analyzing the results in the table, it can be found that the canopy density of the target area does not show a significant correlation with the detection accuracy of the algorithm. On the Harbin L2 dataset, TG-ADet In the "partial occlusion" and "high occlusion" scenarios, the percentages are 59.70% and 54.87% respectively. 38.44% and 37.99% respectively; in comparison, TG-ADet-R They are 62.55% and 59.86% respectively. The results are 37.99% and 41.96% respectively. Although the overall index of the "partial occlusion" scene is slightly higher than that of the "high occlusion" scene, the situation varies in different categories. For example, in the vehicle and tent categories, the algorithm performs better in the "partial occlusion" scene; while in the large cube, small cube and sky dome categories, it performs better in the "high occlusion" scene. Based on these results, two conclusions can be drawn. First, the auxiliary tasks introduced by the SFE module effectively enhance the feature representation ability of the algorithm and help overcome the impact of occlusion. In addition, the inherent penetration ability and three-dimensional imaging characteristics of the lidar provide a natural advantage in detecting obscured targets, enabling the algorithm of the present invention to maintain robust performance even under challenging occlusion conditions. This is particularly important in airborne scenarios, because occlusion caused by tree canopies and other objects is very common in such scenes.

[0064] Table 2 3) Terrain enhancement and comparative analysis: The MTG module is used to predict the terrain distribution of the scene, and then guide multiple stages of the detection network based on the predicted values. Through this module, the network can focus on the area close to the ground and better adapt to airborne scenes with complex and changeable terrain. In order to verify the algorithm's ability to model and focus on terrain, the present invention designs a TDA method based on Perlin noise to further increase the terrain complexity of the scene. The experiment was based on the Harbin L2 dataset, keeping the weights of the network unchanged and adjusting the terrain distribution of the test set. The performance of the excellent SAFDNet, the baseline method VoxelNext and the proposed TG-ADet on the test set after terrain enhancement is compared, and the results are shown in Table 3. The experimental results show that terrain enhancement causes the detection accuracy of each algorithm to decline. However, the method proposed in the present invention still has the best detection performance. Under difficult difficulty conditions, TG-ADet's The index reached 67.18%. Compared with the original detection accuracy, TG-ADet's performance dropped the least. Under the three levels of "easy", "medium" and "difficult", Only decreased by 0.88%, 2.58% and 3.19%, while The decreases were 0.32%, 0.67%, and 0.9%, respectively. This demonstrates the excellent generalization ability of the proposed method for diverse terrain distributions. These results indicate that TG-ADet has good generalization ability and robustness to different terrain distributions.

[0065] Table 3 The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. A terrain-guided airborne laser radar point cloud 3D target detection method, characterized in that: include: First, obtain the airborne lidar point cloud data, and then obtain the digital elevation model (DEM) based on the point cloud data, and then obtain the true terrain value of each detection scene according to the geographic coordinates; Then, detection is performed using a detection network, which includes an encoder and a detection head part, as well as a multi-level terrain guidance module; The encoder and detection head adopt the VoxelNext design. The encoder uses sparse residual blocks for feature extraction, and the detection head is used for object detection. The multi-level terrain guidance module predicts the scene's terrain distribution and guides the multi-stage detection process, and inputs the prediction results into the detection head. The specific process is as follows: The multi-level terrain guidance module includes a terrain height prediction process and a multi-level guidance process based on the prediction results; Terrain height prediction is achieved by accessing a branch sub-network in the middle layer of the encoder, and the middle layer features extracted by the encoder are Separate from the computation graph, and then Extracting intermediate layer features between input and encoder The sparse residual block with the same structure as the subsequent sparse residual block extracts Features of the same size are denoted as ; is the feature in the encoder 、 Expand to The features obtained by matching the spatial dimensions and summing them at the corresponding positions, features 、 、 The middle layer features The features obtained by the three subsequent sparse residual blocks; The features The non-zero parts are accumulated along the height dimension and mapped to the BEV feature map; finally, the BEV feature map is processed by a two-dimensional sparse residual block to generate the terrain prediction result With index ; Get the terrain prediction for the scene Afterwards, As the target distribution center, the distribution offset m is a priori, giving the location feature the maximum weight, and using a Gaussian distribution to calculate the elevation weight: in, is the variance, represent The weight of the i-th voxel in , represent The index of the i-th feature; Based on the elevation weight, the elevation attention Weighted accumulation along the vertical direction to obtain the terrain-weighted BEV feature map ;Will After a layer of sparse convolution and Connect in series along the channel dimension, feed into sub-manifold convolution to complete feature fusion, and output fusion bird's-eye view features , Bird's-Eye View Features Send it into the detection head to complete target detection.

2. The method for three-dimensional target detection based on terrain-guided airborne laser radar point cloud according to claim 1, characterized in that: The process of obtaining a digital elevation model (DEM) based on point cloud data includes: During the point cloud data acquisition process by the airborne lidar, all point clouds are aggregated and the ground points are classified using the IPTD method: first, a sparse triangulated network is constructed using the initial seed points. Then, in an iterative process, the point cloud is encrypted layer by layer. The addition of new points to the triangulated network is determined based on the distance and angle thresholds until the ground point classification is complete. The ground point cloud is then processed using the irregular triangulated network interpolation method, and grid cell values are extracted from multiple triangular surfaces composed of adjacent points to generate a digital elevation model (DEM).

3. The method for three-dimensional target detection based on terrain-guided airborne laser radar point cloud according to claim 1, characterized in that: The detection network also includes an SFE module; the SFE module includes a feature aggregation process and an auxiliary task prediction process; The feature aggregation process includes: Features extracted by the encoder 、 、 ,feature The middle layer features The features of the first sparse residual block corresponding to the sparse residual block are obtained based on the reverse distance weighted average method. 、 、 Corresponding voxel feature coordinates, interpolate them to the point set , and interpolate features in series along the channel dimension, and generate aggregate features through a linear layer ; The auxiliary task prediction process includes two auxiliary tasks: foreground segmentation and point cloud upsampling: The foreground segmentation task is processed using a multi-layer perceptron , output voxel-level segmentation results ; The point cloud upsampling task is to convert It is connected in series with its maximum pooling feature and input into the multi-layer perceptron to achieve feature expansion; then the feature is transformed into Point sets of the same size, output upsampling results ; When adding samples to the network training process, the sampling point is taken as the center, and its neighborhood voxels are queried and grouped to obtain a voxel set.

4. The method for three-dimensional target detection based on terrain-guided airborne laser radar point cloud according to claim 3, characterized in that: The aggregation characteristics as follows: in, Represents the set of points before interpolation A coordinate point in For The coordinates of the nearest k sparse voxels, Belongs to The elements in the feature set represented by the nearest k sparse voxels; Indicates the Layer feature pairs The interpolation result of represents a linear layer, Represents concatenation.

5. The method for three-dimensional target detection based on terrain-guided airborne laser radar point cloud according to claim 3, characterized in that: The detection network also includes a refinement network, which is the refinement network part in Voxel-RCNN, including the step of voxel region of interest pooling processing; the SFE module is integrated with the refinement network, the SFE module outputs the segmentation result and the upsampling prediction, and the segmentation result is mapped to the upsampled voxel set by interpolation to form a feature set ; followed by The segmentation results and voxel coordinates are used as features, and the block encoding of the region of interest is realized according to the detection head proposal information, that is, point pooling, to obtain the The pooled features of the detection head; the detection head proposal information is the detection result of the detection head; The features generated by pooling the voxel region of interest are then concatenated with the pooled features along the channel dimension and input into the subsequent detection head for refined prediction.

6. The method for three-dimensional target detection based on terrain-guided airborne laser radar point cloud according to claim 5, characterized in that: The process of implementing block encoding of the region of interest based on the detection head proposal information and then concatenating the features generated by pooling the voxel region of interest with the pooled features along the channel dimension includes the following steps: After receiving the proposal information, the refinement network divides the three-dimensional cube space circled by the proposal into k*k*k small cube spaces. According to the center of each small cube space, the neighborhood features are queried and encoded from the features extracted by the encoder to obtain the feature A of each small space; at the same time, the center point of the proposal is used as the sampling point, and the SFE module is used to obtain the segmentation and upsampling results of its neighborhood. Then, according to the center of each small cube space, the upsampled neighborhood voxels containing the segmentation information are queried and encoded to obtain the feature B of each small space. After A and B are concatenated along the feature dimension, they are input into the prediction head for processing.

7. The method for three-dimensional target detection based on terrain-guided airborne laser radar point cloud according to any one of claims 3 to 6, characterized in that: The detection network is pre-trained. During the training process, the auxiliary tasks participate in the training process, but the auxiliary tasks do not participate in the actual three-dimensional object detection process.

8. The method for three-dimensional target detection based on terrain-guided airborne laser radar point cloud according to claim 7, characterized in that: When the detection network is not provided with a refinement network, the feature aggregation of the SFE module does not participate in the target detection during the actual three-dimensional target detection process.

9. The method for three-dimensional target detection based on terrain-guided airborne laser radar point cloud according to claim 7, characterized in that: The total loss function of the detection network during training includes: The loss function corresponding to the branch sub-network corresponding to terrain height prediction : in 、 Represents two horizontal coordinate axes, 、 represents the index of the i-th feature of the DEM, 、 represents the index of the i-th feature prediction, 、 represents the feature step size, 、 represents the resolution of the voxel, 、 represents the resolution of the DEM, 、 Represents the size of the detection space, 、 Represents the center index of DEM, Represents the number of features in the bird's-eye view; In the detection proposal generation phase, we use For reference, the difference between the center of the candidate box and the corresponding ground height prediction is integrated into the loss function to explicitly constrain the vertical height of the candidate box; the loss function is defined as follows: in and Respectively The center height and box height of the candidate box predicted by the i-th sparse voxel in; During the network training process, the loss function of the foreground segmentation task adopts focal loss, and the supervision label Provided by the three-dimensional bounding box, the loss function of the point cloud upsampling task adopts the chamfer distance loss, using The coordinate part of is used as supervision; the joint loss function of the two auxiliary tasks is defined as follows: in, represents the focal loss, represents the chamfer distance loss, stands for rejection loss.

10. The method for three-dimensional target detection based on terrain-guided airborne laser radar point cloud according to claim 9, characterized in that: The training data samples of the detection network during training are samples added using the AADA strategy. The specific process includes: For the airborne lidar point cloud data sample library, select samples from the sample library and perform random offset, rotation and collision detection to obtain the sample point cloud and bounding box Based on the digital elevation model, the point cloud elevation is located to determine the ground height of the target placement area. The elevation of the sample center point is then determined in combination with the bounding box height, and the ground points below are removed. The segmented point cloud data is recorded as a scene, and the digital elevation model corresponding to the scene is determined. Adaptive rotation based on position: Divide the bounding box sample into four equal-sized target blocks along the length and width of the bounding box. Rotate the samples by an integer multiple of , so that the block with the largest point cloud density faces the center of the scene; Then take a sample : Construct a spherical polar coordinate system with the drone's location as the origin , where r is the distance from a point to the origin, is the azimuth from the x-axis in the xy plane, Is the polar angle starting from the positive z axis; and transform the background point cloud and the upper four corner points of each target block into the spherical polar coordinate system; query the background point cloud The point cloud set that falls within the area enclosed by the corner points and the origin of each target block : in, Represents the points in the background point cloud set, is its spherical polar coordinate; and are the minimum and maximum azimuths of the corner points, and are the minimum and maximum polar angles of the corner points, is the minimum distance from the origin to the corner point; Calculation point set The mean distance And the area of the sphere at that position in the spherical coordinate system : Further obtain the point cloud density And the sampling rate of the blocks ,in Representative Set Number of internal point clouds; It is a constant, the default value is 1000; Get point cloud density And the sampling rate of the blocks for sampling.

Images