A semantic segmentation method for ground targets based on helicopter-borne laser radar

Abstract

The present invention relates to a method for semantic segmentation of ground targets based on helicopter-mounted laser radar, and relates to the technical field of real-time processing of helicopter-mounted laser radar, comprising the following steps: acquiring real-time point cloud through laser radar, and collecting point cloud data; entering the collected point cloud data into a downsampling module, retaining point cloud features while reducing the amount of point cloud data, and obtaining valid data; pre-processing the valid data obtained after downsampling into an input tensor of a deep learning network; dividing the point cloud into point cloud clusters of each object by means of a deep learning network; and inferring the classification of each point of the original point cloud through KNN, and outputting the point cloud segmentation result. The method has the advantages of being able to simultaneously meet the requirements of real-time and low power consumption, having better detection performance, and ensuring the safety needs of helicopter flight.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time processing of helicopter-borne laser radar, and more specifically, to a semantic segmentation method of ground targets based on helicopter-borne laser radar. Background Art

[0002] LiDAR has the advantages of high measurement accuracy, fine time and space resolution, and long measurement distance, while helicopters can perform low-altitude, low-speed and head-unchanged flight and various complex movements such as lifting, hovering, pitching, and deflection. They are flexible and maneuverable, and are very suitable for measuring various complex terrains. Therefore, helicopter-borne LiDAR has been increasingly widely used in many fields such as geodesy, forest exploration, urban modeling, and disaster assessment.

[0003] With the rapid development of artificial intelligence and helicopter fields, scene understanding is crucial to the safety and effectiveness of machine automatic perception in complex dynamic scenes. Helicopters are usually equipped with various sensors, especially lidar sensors, which play an important role in understanding the visual environment. Lidar systems are used to collect sparse 3D point clouds to reconstruct the environment in the actual scene and help the automatic system make decisions to better understand the scene. Therefore, the scene semantic understanding of point clouds is crucial for helicopter flight; at the same time, research has found that the ground can provide useful information and effectively eliminate the ambiguity caused by data sparsity. The relationship between objects and the ground is conducive to semantic segmentation prediction. Therefore, how to effectively segment the semantics of point clouds in helicopter flight scenes and how to segment the ground and obtain the relationship between objects and the ground are the focus of research.

[0004] Existing methods are divided into two categories: deep learning-based and non-deep learning-based. Deep learning-based methods have high computing power requirements and are difficult to meet both real-time and low power consumption requirements under airborne conditions; non-deep learning-based methods have poor detection performance and are difficult to ensure helicopter flight safety needs.

[0005] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the invention

[0006] The purpose of the present invention is to provide a ground target semantic segmentation method based on helicopter-mounted laser radar, which can simultaneously meet the requirements of real-time and low power consumption, has better detection performance, and ensures the safety needs of helicopter flight.

[0007] The present invention provides a ground target semantic segmentation method based on helicopter-mounted laser radar, comprising the following steps:

[0008] S1: Obtain real-time point cloud through laser radar, collect point cloud data, and enter S2 step;

[0009] S2: Feed the collected point cloud data into the downsampling module to reduce the amount of point cloud data while retaining the point cloud features, obtaining the effective data, and then proceed to step S3;

[0010] S3: Preprocess the effective data obtained after downsampling into the input tensor of the deep learning network, and then proceed to step S4;

[0011] S4: Use the deep learning network method to divide the point cloud into point cloud clusters of each object, and then proceed to step S5;

[0012] S5: Infer the classification of each point in the original point cloud through KNN and output the point cloud segmentation result.

[0013] Furthermore, step S1 includes the following steps:

[0014] S11: The lidar sends udp packets;

[0015] S12: The server side receives and unpacks the packets, discards the bad points, and performs coordinate transformation;

[0016] S13: Complete the acquisition of point cloud data.

[0017] Furthermore, step S2 includes the following steps:

[0018] S21: Use range filtering with a range of 256m in both the x and y directions;

[0019] S22: Use voxel filtering with a voxel size of 0.5m.

[0020] Furthermore, step S3 includes the following steps:

[0021] S31: Project the three-dimensional point cloud information onto a two-dimensional plane using the bird's-eye view method;

[0022] S32: Obtain an input two-dimensional feature map for the deep learning network for training and inference.

[0023] Furthermore, in step S31: Project the three-dimensional point cloud information onto a two-dimensional plane using the front view method.

[0024] Furthermore, step S3 also includes:

[0025] Set up a roi. In the lidar coordinate system, -128m < x ≤ 128m, 0 < y ≤ 256m. Corresponding to a statistical unit in a 0.5m X 0.5m space, use the highest height value of the point cloud falling into the same grid and the median height of the point cloud falling into the same grid to encode the height information. The converted three-channel values are (x, y, z) to obtain a three-channel two-dimensional feature map as the input of the network.

[0026] Furthermore, the step S4 further includes:

[0027] Input the three-channel two-dimensional feature map, pass through the 1×1 convolution kernel, and output the channel to 32 channels to obtain the feature map; use the 32-channel point cloud three-dimensional coordinate tensor feature map, pass through the feature extraction network, pass through the 1×1 convolution kernel, the output channel is 64 channels, then pass through the feature extraction network, pass through the 1×1 convolution kernel, the output channel is 128 channels, then pass through the feature extraction network, pass through the 1×1 convolution kernel, the output channel is 256 channels, then pass through the feature extraction network, output the void space convolution pooling pyramid, and obtain the feature map again; predict the feature map through bilinear interpolation, use the 1-channel point cloud three-dimensional coordinate tensor feature map, through the nearest neighbor classification algorithm, random conditional field, to obtain the point cloud segmentation result.

[0028] Furthermore, the atrous spatial convolution pooling pyramid includes outputting 1×1 convolution kernel 1 atrous convolution, outputting 3×3 convolution kernel 3 atrous convolution, outputting 3×3 convolution kernel 6 atrous convolution, outputting 3×3 convolution kernel 12 atrous convolution, and outputting maximum pooling, and through feature map splicing, 1×1 convolution kernel 1 atrous convolution is obtained again.

[0029] Furthermore, the bad points in step S12 include points with unreasonable distance or reflection intensity and invalid points.

[0030] The method for semantic segmentation of ground targets based on helicopter airborne laser radar provided by the present invention is optimized based on image network so that it can process point cloud data. Downsampling and preprocessing can greatly reduce the amount of data to be processed. When computing resources are limited, non-deep learning methods can be used for processing, which reduces computing power overhead and ensures real-time and low power consumption requirements. Accurate point cloud segmentation can adapt to different environments and landforms, and has better detection performance, thereby ensuring the safety needs of helicopter flight. Considering that in airborne perception applications, the perception range of helicopters is tens to hundreds of times that of autonomous driving vehicles, in order to improve the perception range of the model without increasing the computational complexity, network structures such as ASPP are added to the mobilenet model. For semantic segmentation of ground targets, three-dimensional point clouds are encoded into two-dimensional bird's-eye views, and the highest height value of point clouds falling into the same grid and the median height of point clouds falling into the same grid are used to encode height information, thereby maintaining the simplicity and improving the ability to distinguish false noise points. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 An algorithm module diagram of a ground target semantic segmentation method based on helicopter-mounted laser radar provided in an embodiment of the present invention.

[0032] Figure 2 for Figure 1Flowchart of the semantic segmentation method of ground targets based on helicopter-borne lidar.

[0033] Figure 3 for Figure 1 FIG. 4 is a processing module diagram of step S4 of the ground target semantic segmentation method based on helicopter-borne lidar. DETAILED DESCRIPTION

[0034] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0035] The terms "first", "second", "third", "fourth" and the like in the description and claims of the present invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence.

[0036] Example 1

[0037] Figure 1 This is an algorithm module diagram of the ground target semantic segmentation method based on helicopter-mounted laser radar provided by an embodiment of the present invention. Figure 1 The method for semantic segmentation of ground targets based on helicopter-borne laser radar provided by an embodiment of the present invention comprises the following steps:

[0038] S1: Obtain real-time point cloud through laser radar, collect point cloud data, and enter S2 step;

[0039] S2: The collected point cloud data enters the downsampling module, retains the point cloud features while reducing the amount of point cloud data to obtain valid data. It should be noted that the downsampling module can reduce the processing time and enter the S3 step;

[0040] S3: The valid data obtained after downsampling is preprocessed into the input tensor of the deep learning network. It should be noted that the projection process reduces the data dimension and data volume, and enhances the computing power dependence and timeliness compared to the three-dimensional data processing method, and then enters the S4 step;

[0041] S4: Use deep learning network to divide the point cloud into point cloud clusters of each object;

[0042] S5: Use KNN (nearest neighbor classification algorithm) to infer the classification of each point in the original point cloud and output the point cloud segmentation result.

[0043] The algorithm of the ground target semantic segmentation method based on helicopter airborne laser radar of the present invention is optimized based on image network so that it can process point cloud data. Downsampling and preprocessing can greatly reduce the amount of data to be processed. When computing resources are limited, non-deep learning methods can be used for processing, which reduces computing power overhead and ensures real-time and low power consumption requirements. Accurate point cloud segmentation can adapt to different environments and landforms, has better detection performance, and ensures the safety needs of helicopter flight.

[0044] Figure 2 for Figure 1 Flowchart of the semantic segmentation method of ground targets based on helicopter-borne laser radar. Please refer to Figure 2 , step S1 comprises the following steps:

[0045] S11: The laser radar sends a UDP (User Datagram Protocol) packet;

[0046] S12: The server receives and decompresses the packets, discards bad pixels, and performs coordinate system conversion;

[0047] Specifically, bad pixels include pixels with unreasonable distance or reflection intensity and invalid pixels.

[0048] S13: Complete the collection of point cloud data.

[0049] Step S2 includes the following steps:

[0050] S21: Use range filtering of 256m in x and y directions;

[0051] S22: Use voxel filtering with a voxel size of 0.5m.

[0052] It should be noted that the downsampling process can effectively retain the point cloud features while reducing the amount of point cloud data and shortening the processing time.

[0053] Step S3 includes the following steps:

[0054] S31: projecting the three-dimensional point cloud information onto a two-dimensional plane using a bird's-eye view or projecting the three-dimensional point cloud information onto a two-dimensional plane using a front view;

[0055] S32: Obtain an input 2D feature map of a deep learning network for training and inference.

[0056] It should be noted that the projection process reduces the data dimension and data volume, and enhances the computing power dependence and timeliness compared to the three-dimensional data processing method.

[0057] The step S3 further comprises:

[0058] Set up a ROI (region of interest), in the lidar coordinate system, -128m < x ≤ 128m, 0 < y ≤ 256m. Only convert the point cloud within this range. To convert the point cloud into the pixel coordinate system, there must be a process of generating statistical values. According to a 0.5m X 0.5m space corresponding to a statistical unit (such as the pixels of an image), use the highest height value of the point cloud falling into the same grid and the median height of the point cloud falling into the same grid to encode the height information (analogous to the three channels of a color image being the values in the color space, such as the blue, green, and red components in the BGR color space). The converted three-channel values are (x, y, z), and a two-dimensional feature map with three channels is obtained as the input of the network.

[0059] Figure 3 For Figure 1 Figure of the processing module in step S4 of the ground target semantic segmentation method based on helicopter-borne lidar. Please refer to Figure 3 , and step S4 includes the following steps:

[0060] Input the two-dimensional feature map with three channels, pass through a 1×1 convolutional kernel, and the output channel is 32 channels to obtain a feature map; use the three-dimensional coordinate tensor feature map of the 32-channel point cloud, pass through the feature extraction network, pass through a 1×1 convolutional kernel, and the output channel is 64 channels. Then pass through the feature extraction network again, pass through a 1×1 convolutional kernel, and the output channel is 128 channels. Then pass through the feature extraction network again, pass through a 1×1 convolutional kernel, and the output channel is 256 channels. Then pass through the feature extraction network again to output the atrous spatial pyramid pooling, and obtain the feature map again; through bilinear interpolation, predict the feature map, use the three-dimensional coordinate tensor feature map of 1 channel, and through the nearest neighbor classification algorithm and random conditional field, obtain the point cloud segmentation result.

[0061] Specifically, the atrous spatial pyramid pooling includes outputting a 1×1 convolutional kernel with 1 dilation, outputting a 3×3 convolutional kernel with 3 dilations, outputting a 3×3 convolutional kernel with 6 dilations, outputting a 3×3 convolutional kernel with 12 dilations, outputting max pooling, and obtaining a 1×1 convolutional kernel with 1 dilation again through feature map splicing.

[0062] Based on the above description, the advantages of the present invention are as follows:

[0063] 1. The deep learning point cloud perception network model based on mobilenet+ASPP+InterP+Knn is the first of its kind; considering that in airborne perception applications, the perception range of a helicopter is dozens to hundreds of times that of an autonomous vehicle. To improve the perception range of the model without increasing the computational complexity, network structures such as ASPP are added to the mobilenet model;

[0064] 2. Encode the three-dimensional point cloud into a two-dimensional bird's-eye view. The encoding method usually takes the highest height value of the point cloud falling into the same grid, but this encoding method lacks the ability to distinguish false noise points, or encodes the three-dimensional point cloud into a three-dimensional grid map, but this method will produce a large number of empty grids, resulting in a waste of computing resources. In the present invention, the highest height value of the point cloud falling into the same grid and the median height of the point cloud falling into the same grid are used to encode the height information, which maintains the simplicity while improving the ability to distinguish false noise points.

[0065] 3. Accurate point cloud segmentation can adapt to different environments and terrains, with better detection performance, ensuring the safety needs of helicopter flight.

[0066] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for semantic segmentation of ground targets based on helicopter-borne laser radar, characterized in that: It includes the following steps: S1: Obtain real-time point cloud through lidar, collect point cloud data, and enter step S2; S2: Input the collected point cloud data into the downsampling module, reduce the amount of point cloud data while retaining the point cloud features to obtain valid data, and enter step S3; S3: Preprocess the valid data obtained after downsampling into the input tensor of the deep learning network, and enter step S4; S4: Use the deep learning network method to divide the point cloud into point cloud clusters of each object, and enter step S5; S5: Infer the classification of each point of the original point cloud through KNN, and output the point cloud segmentation result; Step S1 includes the following steps: S11: The lidar sends udp packets; S12: The server side receives and unpacks the packets, discards the bad points, and performs coordinate system conversion; S13: Complete the collection of point cloud data; Step S3 includes the following steps: S31: Project the three-dimensional point cloud information onto a two-dimensional plane in the way of a bird's-eye view; S32: Obtain an input two-dimensional feature map of the deep learning network for training and inference; Step S3 also includes: Set up a roi, in the lidar coordinate system, -128m < x ≤ 128m, 0 < y ≤ 256m, corresponding to a statistical unit for every 0.5mX0.5m space. Use the highest height value of the point cloud falling into the same grid and the median height of the point cloud falling into the same grid to encode the height information. The converted three-channel value is (x, y, z) to obtain a three-channel two-dimensional feature map as the input of the network.

2. The method for semantic segmentation of ground targets based on helicopter-borne laser radar according to claim 1, characterized in that: Step S2 includes the following steps: S21: Use range filtering with a range of 256m in both the x and y directions; S22: Use voxel filtering with a voxel size of 0.5m.

3. The method for semantic segmentation of ground targets based on helicopter-borne laser radar according to claim 1, characterized in that: In step S31, project the three-dimensional point cloud information onto a two-dimensional plane in the way of a front view.

4. The method for semantic segmentation of ground targets based on helicopter-borne laser radar according to claim 1, characterized in that: Step S4 also includes: Input the three-channel two-dimensional feature map, pass through a 1x1 convolution kernel, with an output channel of 32 channels to obtain a feature map; use the 32-channel point cloud three-dimensional coordinate tensor feature map, pass through the feature extraction network, through a 1x1 convolution kernel, with an output channel of 64 channels, then through the feature extraction network, through a 1x1 convolution kernel, with an output channel of 128 channels, then through the feature extraction network, through a 1x1 convolution kernel, with an output channel of 256 channels, then through the feature extraction network to output the atrous spatial pyramid pooling, and obtain a feature map again; through bilinear interpolation, predict to obtain a feature map, use the 1-channel point cloud three-dimensional coordinate tensor feature map, through the nearest neighbor classification algorithm and random conditional field, to obtain the point cloud segmentation result.

5. The method for semantic segmentation of ground targets based on helicopter-borne laser radar according to claim 4 is characterized in that: The atrous spatial pyramid pooling includes outputting a 1x1 convolution kernel 1 atrous convolution, outputting a 3x3 convolution kernel 3 atrous convolution, outputting a 3x3 convolution kernel 6 atrous convolution, outputting a 3x3 convolution kernel 12 atrous convolution, outputting a max pooling, and obtaining a 1x1 convolution kernel 1 atrous convolution again through feature map splicing.

6. The method for semantic segmentation of ground targets based on helicopter-borne laser radar according to claim 1, characterized in that: In step S12, the bad points include points with unreasonable distance or reflection intensity and invalid points.

Images