Laser radar signal processing method and system based on deep learning

Abstract

The invention discloses a laser radar signal processing method and system based on deep learning, and belongs to the technical field of laser radars. According to the method, a data set for training is obtained through simulation of a laser radar detection principle, and a time window sliding module, a self-attention mechanism module and a soft threshold denoising module are introduced for the data set for training to construct a laser radar signal processing and three-dimensional structure reconstruction network. And finally, based on the constructed laser radar signal processing and three-dimensional structure reconstruction network, outputting a corresponding denoised three-dimensional structure reconstruction result according to the input to-be-processed laser radar original data. Compared with a traditional histogram technical method, the method has the advantages that weak echo and low signal-to-noise ratio scene laser radar signal processing and three-dimensional structure reconstruction can be achieved, the detection speed for a target scene is increased, and the requirement for high-precision reconstruction of a detection scene in a laser radar is met.

Description

Technical Field

[0001] The present invention belongs to the field of laser radar technology, and in particular relates to a laser radar signal processing method and system based on deep learning. Background Art

[0002] As an active detection technology with high precision and high temporal resolution, LiDAR can obtain high-precision three-dimensional structural information of the detection scene and is often used in fields such as satellite-borne remote sensing and autonomous driving. Traditional LiDAR needs to receive enough echo signals to suppress the inherent background noise in the detection process, and is not applicable in scenarios with limited integration time and light flux. In order to improve hardware performance, single-photon LiDAR systems have been widely used. However, there is a problem of long accumulation time in the detection process, and it is necessary to optimize the algorithm design to improve the detection performance indicators.

[0003] There are currently two major challenges that limit the decomposition of the distance information of the detected target from the photon sequence histogram. The first is the weak echo scenario. In the scenario where the light flux and integration time are limited, the signal photons corresponding to the pulse echo received by the lidar are weaker than the ambient light intensity background noise; the second is the low signal-to-noise ratio scenario: in actual application scenarios, due to the strong background light noise and detector dark count noise, the recorded photon sequence histogram data contains a large number of noise counts.

[0004] In order to achieve efficient three-dimensional reconstruction counting, it is necessary to solve the two major difficulties of signal feature extraction and signal noise separation in weak echo and low signal-to-noise ratio scenarios. The deep reconstruction method based on statistical learning achieves efficient photon imaging through constrained convex optimization heuristic iteration, but has the disadvantages of requiring manual setting of hyperparameters and poor performance in low signal-to-noise ratio scenarios. The deep learning-based method can achieve robust and efficient three-dimensional structure reconstruction in multiple signal-to-noise ratio scenarios, but it has the disadvantages of lack of interpretability and specific design for the unique properties of single-photon lidar raw data, and there is room for further improvement. Summary of the invention

[0005] To solve the above technical problems, the present invention provides a lidar signal processing method and system based on deep learning, which solves the technical problem that the statistical learning method in the prior art is not suitable for weak echo and low signal-to-noise ratio scenarios, and improves processing efficiency and accuracy.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] A laser radar signal processing method based on deep learning, the method comprising:

[0008] Acquire a plurality of detection scene depth maps marked with pixel true distances and a plurality of detection scene intensity maps marked with reflection intensities;

[0009] Performing simulation processing on the acquired depth map and intensity map to obtain a training data set;

[0010] Based on the training data set, an ADMN network architecture is trained to obtain a lidar signal processing and three-dimensional structure reconstruction network; wherein the ADMN network architecture introduces a time window sliding module for preliminarily extracting signal features, a self-attention mechanism module for fully extracting signal features, and a soft threshold denoising module for adaptively separating signal photons and noise photons;

[0011] Based on the laser radar signal processing and three-dimensional structure reconstruction network, a three-dimensional structure diagram of the detected target is reconstructed after denoising corresponding to the laser radar raw data to be processed.

[0012] On the other hand, the present invention provides a laser radar signal processing system based on deep learning, the system comprising:

[0013] A data acquisition module, used to acquire a plurality of detection scene depth maps marked with pixel real distances and a plurality of detection scene intensity maps marked with reflection intensities;

[0014] A simulation module is used to simulate the acquired depth map and intensity map to obtain a training data set;

[0015] A model building module, used to train the ADMN network architecture based on the training data set to obtain a lidar signal processing and three-dimensional structure reconstruction network; wherein the ADMN network architecture introduces a time window sliding module for preliminary extraction of signal features, a self-attention mechanism module for fully extracting signal features, and a soft threshold denoising module for adaptively separating signal photons and noise photons;

[0016] The data processing module is used to reconstruct the three-dimensional structure diagram of the detected target after denoising based on the laser radar signal processing and three-dimensional structure reconstruction network model according to the laser radar raw data to be processed.

[0017] In a third aspect, the present invention provides an electronic device comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned deep learning-based lidar signal processing method.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned deep learning-based lidar signal processing method.

[0019] The beneficial effects of the present invention are:

[0020] The present invention introduces a time window sliding module, which facilitates the subsequent denoising module to more intuitively calculate the denoising threshold, and realizes feature extraction and enhancement without significantly increasing the number of parameters and computing costs; introduces a soft threshold denoising module based on residual connection, which does not require manual setting of the threshold, and sets an adaptive threshold through a deep learning method to better filter out noise photons; introduces a self-attention mechanism to learn the long-range correlation of the photon sequence histogram, which is helpful for feature extraction and distance calculation; introduces multiple loss function joint constraints to improve the network convergence speed and calculation accuracy. In general, the speed of laser radar three-dimensional structure reconstruction is improved, the accuracy of the three-dimensional structure reconstruction of the detection scene is improved, and the demand for improving the efficiency of photon utilization in laser radar detection is met. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the network structure of a laser radar signal processing method based on deep learning according to the present invention;

[0022] Figure 2 The schematic diagram of the algorithm of the soft threshold denoising module of the present invention;

[0023] Figure 3 This is one of the implementation effect diagrams based on the method of the present invention;

[0024] Figure 4 This is the second effect diagram of the implementation of the method of the present invention;

[0025] Figure 5 It is a structural schematic diagram of the laser radar signal processing system based on deep learning of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0027] The present invention provides a laser radar signal processing method based on deep learning, such as Figure 1 As shown, the method includes: obtaining a depth map and an intensity map of a detection scene of a PNG format image that marks the depth information and intensity information of a detection target; performing data enhancement and simulation on the obtained registered and aligned depth map and intensity map according to the laser radar detection principle, obtaining three-dimensional laser radar raw data, and obtaining a training data set; based on the training data set, training an ADMN network architecture to obtain a laser radar signal processing and three-dimensional structure reconstruction model; introducing a time window sliding module for preliminarily extracting signal features, a self-attention mechanism module for fully extracting signal features, and a soft threshold denoising module for adaptively separating signal photons and noise photons into the ADMN network architecture; based on the laser radar signal processing and three-dimensional structure reconstruction model, the three-dimensional structure is reconstructed according to the denoising solution corresponding to the laser radar raw data to be processed.

[0028] Furthermore, the acquired depth map and intensity map are simulated to obtain a training data set, including: randomly rotating the depth map and intensity map, randomly flipping the image, and adjusting the image size to increase the distribution diversity of the training data, enhancing the depth map and intensity map based on the matched aligned data, setting different signal-to-noise ratios, simulating each pixel to obtain the corresponding photon sequence histogram, and obtaining the lidar raw data as a training data set.

[0029] Furthermore, the ADMN network architecture includes a time window sliding module for preliminarily extracting signal features, a self-attention mechanism module for fully extracting signal features, and a soft threshold denoising module for adaptively separating signal photons and noise photons; wherein the time window sliding module accepts the original laser radar data input by the user and outputs the preliminarily processed feature data; then the preliminarily features are extracted through a series of convolutional layers, Relu activation function layers, and dilated convolutional layers, and the preliminarily features are input to the self-attention mechanism module; the self-attention mechanism module performs global correlation processing and global feature extraction on the feature data of the input preliminary features, and outputs the global correlation feature data considering the global correlation to the soft threshold denoising module; the soft threshold denoising module performs adaptive soft threshold generation and noise photon removal on the global correlation feature data of the fully extracted feature information, and outputs the target feature data containing the echo signal feature information after the noise photons are removed; the feature data is processed through a series of deconvolution layers (i.e., transposed convolution shown in the figure), Relu activation function layers, and fully connected layers, and finally weighted sum calculation is performed on the depth dimension according to the centroid method to obtain the detection scene depth map distribution obtained after denoising.

[0030] Furthermore, the ADMN network architecture contains a time window sliding module for preliminarily extracting signal features, including: the value of the convolution kernel in the time window sliding module used is related to the value of the system laser pulse width, the convolution kernel value is set to a Gaussian convolution kernel of the same size as the laser pulse width, and is not modified during the network training process.

[0031] Furthermore, the ADMN network architecture contains a self-attention mechanism module for fully extracting signal features, including first deforming and rearranging the input preliminary feature map, which is reflected in merging the feature dimension and depth dimension of the feature map, performing global correlation calculation between feature maps at different pixel positions, and then calculating the corresponding self-attention features and performing residual connections, and finally deforming the self-attention features again to restore the same dimension as the input.

[0032] like Figure 2As shown, the ADMN network architecture contains a soft threshold denoising module for adaptively separating signal photons and noise photons, and the soft threshold denoising module includes a first shaping operator, an absolute value operator, a normalization module, a three-dimensional feature extraction layer, an element-by-element multiplication operator, a soft threshold calculation operator, a second shaping operator, an element-by-element summation operator and an activation function output unit; wherein the first shaping operator inputs feature data considering global correlation output by a self-attention mechanism module processed by a series of convolutional layers and Relu activation function layers, and is used to perform feature extraction on the feature data considering global correlation. The data is rearranged in dimension to obtain a feature map corresponding to the three-dimensional space dimension; the absolute value operator is connected to the first shaping operator and is used to take the absolute value of the feature map corresponding to the three-dimensional space dimension to obtain an absolute value feature map; the normalization module is connected to the absolute value operator and is used to normalize the absolute value feature map along the depth dimension to obtain a normalized feature map; the three-dimensional feature extraction layer is connected to the normalization module and is used to extract and fuse the input normalized feature map to obtain signal distribution characteristics, wherein the three-dimensional feature extraction layer includes a series of cross The convolution layer and the Relu activation function layer connected alternately, and the final sigmoid activation function layer; the element-by-element multiplication operator is connected to the three-dimensional feature extraction layer and the absolute value feature output by the absolute value operator, and is used to perform element-by-element multiplication of the signal distribution feature and the absolute value feature map to obtain the soft threshold feature; the soft threshold calculation operator is connected to the element-by-element multiplication operator and the first shaping operator, and is used to fuse the feature map corresponding to the three-dimensional space dimension and the soft threshold feature, separate the signal photon and the noise photon features, and obtain the feature map after soft threshold denoising; the second shaping operator is connected to the soft threshold calculation operator, and is used to rearrange the dimension of the feature map after soft threshold denoising to obtain the final feature map; the element-by-element summation operator is connected to the output feature of the self-attention mechanism module and the second shaping operator, and is used to add the feature data considering global correlation and the final feature map; the activation function output unit is used to receive the feature data considering global correlation and the final feature map after addition, use the Relu activation function for processing, and output the target feature data containing the echo signal feature information after removing the noise photons to the subsequent processing network. Finally, a series of deconvolution layers, Relu activation function layers and fully connected layers are connected for subsequent feature integration and depth solving.

[0033] Furthermore, the loss functions for training the LiDAR signal processing and 3D structure reconstruction networks include: cross entropy loss, mean square error loss, and ordinal regression loss;

[0034] Furthermore, the ordinal regression loss is:

[0035]

[0036] in, is the one-hot encoding of the real data detection distance distribution time bin, is the echo waveform distribution after network solution, is the first one-hot encoding of the real data detection distance distribution The corresponding value of each time bin is It is the first time that the echo waveform distribution is solved through the network. The corresponding value of each time bin is is the entire detection cycle, It is the value of the time bin corresponding to the real data detection distance, and cumsum is the cumulative summation operation.

[0037] The NYUv2 dataset contains 1449 pairs of intensity and depth images from 27 different scenes. The average signal-to-noise ratios of the simulations are set to 2:2 / 2:10 / 2:50 / 5:2 / 5:10 / 5:50 / 10:2 / 10:10 / 10:50, and the training, validation, and test sets are allocated in a ratio of 7:2:1. After network training, the network is tested on 8 scenes in Middlebury with a signal-to-noise ratio of 2:100, which does not appear in the training set. The quantitative results are shown in the figure below. Figure 3 As shown. The network structures used for comparison include U-Net++, PRSNet, Non-local and the network proposed in the present invention. The best results are marked in bold and the suboptimal results are marked with underlines. The quantitative results in the figure show that compared with other deep learning methods, the method proposed in the present invention achieves better solution results in terms of root mean square error and threshold accuracy. In addition, the raw data of the weak echo scene laser radar collected in reality is processed using the method proposed in the present invention to obtain the solved depth map as shown in the figure. Figure 4 As shown in the figure, for low signal-to-noise ratio scenarios (SNR of 2:100) and weak echo scenarios (the average number of echo photons per pixel is 3.48), the method proposed in the present invention has relatively fewer noise points in the solved depth map, and the edge solution is clearer, which is more similar to the distribution of the detected target. The overall solution result is better, which proves the effectiveness of the network structure proposed in the present invention.

[0038] like Figure 5 As shown, in order to achieve the above-mentioned purpose and other related purposes, the present invention provides a laser radar signal processing system based on deep learning, and each module included in it can implement each step of the above-mentioned method. The system includes: a data acquisition module, which is used to acquire a plurality of detection scene depth maps marked with pixel real distances and a plurality of detection scene intensity maps marked with reflection intensities;

[0039] A simulation module is used to simulate the acquired depth map and intensity map to obtain a training data set;

[0040] A model building module, used to train the ADMN network architecture based on the training data set to obtain a lidar signal processing and three-dimensional structure reconstruction network; wherein the ADMN network architecture introduces a time window sliding module for preliminary extraction of signal features, a self-attention mechanism module for fully extracting signal features, and a soft threshold denoising module for adaptively separating signal photons and noise photons;

[0041] The data processing module is used to reconstruct the three-dimensional structure diagram of the detected target after denoising based on the laser radar signal processing and three-dimensional structure reconstruction network model according to the laser radar raw data to be processed.

[0042] In a third aspect, the present invention provides an electronic device comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned lidar signal processing method based on the deep learning method.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned lidar signal processing method based on the deep learning method.

[0044] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A laser radar signal processing method based on deep learning, characterized in that: The method comprises: Acquire a plurality of detection scene depth maps marked with pixel true distances and a plurality of detection scene intensity maps marked with reflection intensities; Performing simulation processing on the acquired depth map and intensity map to obtain a training data set; Based on the training data set, an ADMN network architecture is trained to obtain a lidar signal processing and three-dimensional structure reconstruction network; wherein the ADMN network architecture introduces a time window sliding module for preliminarily extracting signal features, a self-attention mechanism module for fully extracting signal features, and a soft threshold denoising module for adaptively separating signal photons and noise photons; Based on the laser radar signal processing and three-dimensional structure reconstruction network, a three-dimensional structure diagram of the detected target is reconstructed after denoising corresponding to the laser radar raw data to be processed.

2. According to the deep learning-based laser radar signal processing method described in claim 1, it is characterized in that: The step of performing simulation processing on the acquired depth map and intensity map to obtain a training data set includes: The collected depth map and intensity map are randomly rotated in a random direction, randomly flipped, and the image size is adjusted to obtain the depth map and intensity map after data enhancement; Based on the depth map and intensity map enhanced by matching and aligning data, different signal-to-noise ratios are set, and each pixel is simulated to obtain the corresponding photon sequence histogram to obtain the original lidar data as a training data set.

3. According to the deep learning-based laser radar signal processing method described in claim 1, it is characterized in that: The value of the convolution kernel in the time window sliding module is set to a Gaussian convolution kernel of the same size as the laser pulse width and is not modified during the network training process.

4. The laser radar signal processing method based on deep learning according to claim 1, characterized in that: The self-attention mechanism module is used to receive the preliminary features output by the time window sliding module, perform global correlation processing and secondary feature extraction on the preliminary features, output feature data considering global correlation and input it into the soft threshold denoising module.

5. The laser radar signal processing method based on deep learning according to claim 1, characterized in that: The soft threshold denoising module includes a first shaping operator, an absolute value operator, a normalization module, a three-dimensional feature extraction layer, an element-by-element multiplication operator, a soft threshold calculation operator, a second shaping operator, an element-by-element summation operator and an activation function output unit which are sequentially connected in communication; wherein, The first shaping operator receives the feature data considering global correlation output by the self-attention mechanism module, and is used to rearrange the dimension of the feature data considering global correlation to obtain a feature map corresponding to the three-dimensional space dimension; The absolute value operator is used to take the absolute value of the feature map corresponding to the three-dimensional space dimension to obtain an absolute value feature map; The normalization module is used to normalize the absolute value feature map along the depth dimension to obtain a normalized feature map; The three-dimensional feature extraction layer is used to extract and fuse the normalized feature map to obtain signal distribution features; The element-by-element multiplication operator is used to perform element-by-element multiplication of the signal distribution feature and the absolute value feature map to obtain a soft threshold feature; The soft threshold calculation operator is used to fuse the feature map corresponding to the three-dimensional space dimension and the soft threshold feature, separate the signal photon and noise photon features, and obtain the feature map after soft threshold denoising; The second shaping operator is used to rearrange the dimension of the feature map after the soft threshold denoising to obtain a final feature map; The element-by-element summation operator is used to add the feature data considering global correlation and the final feature map; The activation function output unit is used to receive the added feature data considering global correlation and the final feature map, process them using the Relu activation function, and output the target feature data containing the echo signal feature information after removing the noise photons to a subsequent processing network.

6. The laser radar signal processing method based on deep learning according to claim 1, characterized in that: The loss functions for training the LiDAR signal processing and 3D structure reconstruction networks include cross entropy loss, mean square error loss, and ordinal regression loss.

7. A laser radar signal processing method based on deep learning according to claim 6, characterized in that: The ordinal regression loss is: ； in, is the one-hot encoding of the real data detection distance distribution time bin, is the echo waveform distribution after network solution, is the first one-hot encoding of the real data detection distance distribution The corresponding value of each time bin is It is the first time that the echo waveform distribution is solved through the network. The corresponding value of each time bin is is the entire detection cycle, It is the value of the time bin corresponding to the real data detection distance, and cumsum is the cumulative summation operation.

8. A laser radar signal processing system based on deep learning, characterized in that: The system comprises: A data acquisition module, used to acquire a plurality of detection scene depth maps marked with pixel real distances and a plurality of detection scene intensity maps marked with reflection intensities; A simulation module is used to simulate the acquired depth map and intensity map to obtain a training data set; A model building module, used to train the ADMN network architecture based on the training data set to obtain a lidar signal processing and three-dimensional structure reconstruction network; wherein the ADMN network architecture introduces a time window sliding module for preliminary extraction of signal features, a self-attention mechanism module for fully extracting signal features, and a soft threshold denoising module for adaptively separating signal photons and noise photons; The data processing module is used to reconstruct the three-dimensional structure diagram of the detected target after denoising based on the laser radar signal processing and three-dimensional structure reconstruction network output according to the original laser radar data to be processed.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; Wherein, when one or more programs are executed by the one or more processors, the one or more processors implement a lidar signal processing method based on deep learning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, which, when executed by a processor, enable the processor to implement a deep learning-based lidar signal processing method as described in any one of claims 1-7.

Images