A method for reconstructing saturated waveforms of a lidar based on a hybrid neural network

Abstract

This application provides a method for reconstructing saturated waveforms of lidar based on a hybrid neural network, comprising: normalizing the full waveform data of the lidar; traversing the data using a fixed-length sliding window to extract sub-waveform segments with larger sums of amplitudes within the sliding window; extracting local abrupt change features and global morphological features of each sub-waveform segment using a dual-branch convolutional network; fusing the extracted features using a multi-head attention fusion network to detect abnormal sub-waveform segments and corresponding saturated sampling points; establishing a reconstruction network based on an encoder structure and a residual convolutional structure to reconstruct the amplitude values ​​of the saturated sampling points, obtaining reconstructed sub-waveform segments; inserting the reconstructed sub-waveform segments into the corresponding positions of the normalized full waveform data of the lidar, and performing an inverse normalization operation to obtain the reconstructed full waveform data. This method has high recognition accuracy and small reconstruction error, and can significantly improve the ranging accuracy and waveform data usability of lidar.

Description

Technical Field

[0001] This application belongs to the field of lidar data processing technology, and in particular relates to a lidar saturation waveform reconstruction method based on a hybrid neural network. Background Technology

[0002] Full-waveform lidar systems, as an important tool for high-precision 3D mapping, provide continuous information on the distribution of echo energy over time and are widely used in remote sensing tasks such as terrain modeling, vegetation structure inversion, and urban scene analysis. Compared to traditional point cloud products, full-waveform data contains richer echo features of ground objects and has higher application value. However, in actual measurements, when the laser beam encounters high-reflectivity targets (such as metal roofs, water surfaces, and highly reflective vegetation), the returned signal energy may exceed the dynamic range of the receiving system, leading to saturation distortion in the lidar waveform. Saturated waveforms often exhibit characteristics such as spike distortion and flat-top truncation, resulting in the loss of key physical features and affecting the accuracy of subsequent target identification and quantitative estimation. Currently, some studies have attempted to recover saturated waveforms through signal interpolation, empirical modeling, and regression reconstruction, but most methods are not ideal when dealing with complex distributions of saturation points and large variations in the morphology of saturation regions, and are difficult to adapt to waveform distortion patterns from multiple sources and multiple scenes. Therefore, how to design a waveform reconstruction method with structural adaptability and capable of handling multiple types of saturation modes has become a key problem that urgently needs to be solved in the field of intelligent processing of full-waveform lidar. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a method for reconstructing saturated waveforms of lidar based on hybrid neural networks. This method has high accuracy in identifying saturated sampling points and small reconstruction error, which can significantly improve the ranging accuracy and waveform data availability of lidar.

[0004] This application provides a method for reconstructing saturated waveforms of lidar based on hybrid neural networks, including:

[0005] After normalizing the full waveform data of the lidar, a fixed-length sliding window is used to traverse the data to extract the sub-waveform segments with larger total amplitude within the sliding window.

[0006] Local mutation features and global morphological features of each sub-waveform segment are extracted using a dual-branch convolutional network. Then, a multi-head attention fusion network is used to fuse the extracted features to detect abnormal sub-waveform segments and corresponding saturation sampling points.

[0007] A reconstruction network is established based on the Transformer encoder structure and the residual convolution structure, and the amplitude values ​​of the saturated sampling points are reconstructed to obtain the reconstructed sub-waveform segments.

[0008] The reconstructed sub-waveform segments are inserted into the corresponding positions of the normalized full waveform data of the lidar, and then the inverse normalization operation is performed to obtain the reconstructed full waveform data.

[0009] Furthermore, the full waveform data of the lidar is normalized using the following method:

[0010] To standardize the amplitude scale of different waveforms, normalization is performed on each waveform using the following formula:

[0011] ;

[0012] In the formula, This is the current waveform number. The amplitude value at each sampling point This is the current waveform number. The normalized amplitude value of each sampling point and These are the minimum and maximum amplitude values ​​in the current waveform.

[0013] Furthermore, the sub-waveform segments with larger sums of amplitude within the sliding window are extracted using the following method:

[0014] Assume the normalized full waveform is Where N is the number of sampling points in the full waveform data, and the length of the sliding window is L, then at each starting position... The total amplitude within the sliding window is calculated as follows:

[0015] ;

[0016] Select to make Largest index The sub-waveform segment most likely to have a saturation sampling point is obtained as follows:

[0017] ;

[0018] In the formula, This indicates the amplitude value at the beginning of the sub-wavelength segment. This indicates the amplitude value at the end position of the sub-waveform segment.

[0019] Furthermore, the extraction of local abrupt change features and global morphological features of each sub-waveform segment using a dual-branch convolutional network refers to:

[0020] Convolutional networks using 3×1 small kernels can extract local abrupt changes in peak-distorted waveforms, while convolutional networks using 5×1 large kernels can extract global morphological features of flat-topped truncated waveforms.

[0021] Furthermore, the step of fusing the extracted features using a multi-head attention fusion network to detect abnormal sub-waveform segments and corresponding saturation sampling points includes:

[0022] After the extracted local mutation features and global morphological features are concatenated along the channel dimension, they are weighted and fused using learnable weight coefficients to obtain preliminary fused features.

[0023] Each attention head performs a linear projection on the preliminary fused features to obtain a query vector, a key vector, and a value vector;

[0024] The query vector, key vector, and value vector of multiple attention heads are concatenated and then linearly mapped to obtain fused features, which are used to determine whether each sub-waveform segment is abnormal.

[0025] The saturation sampling point is determined by identifying the sub-waveform segment that is deemed abnormal.

[0026] Furthermore, the method also includes:

[0027] During the training phase of the dual-branch convolutional network and the multi-head attention fusion network, to address the extreme imbalance problem where the proportion of saturated sampling points in the full waveform sample data of LiDAR is less than 5%, the loss function is constructed in the following way:

[0028] The proportion of saturated sampling points in the full waveform sample data of the LiDAR in the current training round is used to obtain the category weighting factor. :

[0029] ;

[0030] In the formula, Parameters are adjusted for dynamic categories. This represents the number of saturated sampling points in the full waveform sample data of the lidar. This represents the total number of sampling points in the full waveform sample data of the lidar.

[0031] A focus factor for easy and difficult samples is introduced, and this factor is used in conjunction with the category weighting factor to adjust the loss function. :

[0032] ;

[0033] ;

[0034] In the formula, As a prediction probability adjustment factor, For focus adjustment parameters, For the true labels of the samples, Predict probabilities for the model.

[0035] Furthermore, before reconstructing the amplitude values ​​of the saturated sampling points, the method further includes:

[0036] Based on the number of detected saturated sampling points, a suitable reconstruction mechanism is selected using the following method:

[0037] When the number of saturated sampling points is greater than 1 and less than or equal to 6, the amplitude values ​​of the saturated sampling points are reconstructed using a parallel reconstruction network.

[0038] When the number of saturated sampling points is greater than 6, the saturated sampling points are downsampled and then reconstructed using a parallel reconstruction network.

[0039] Specifically, when the number of saturated sampling points is 7 or 9, downsampling is performed by sampling at intervals with a step size of 2. When the number of saturated sampling points is 8 or 10, non-uniform sampling is performed according to predefined rules.

[0040] Furthermore, the reconstruction network reconstructs the amplitude values ​​of the saturated sampling points in the following manner:

[0041] The Transformer encoder is used to model the long-range dependency of the original sub-waveform segments, and residual convolution is combined to enhance the local detail restoration capability. The amplitude value of the sampling point is reconstructed using the Wave block decoder. At the same time, a gating fusion mechanism is introduced to dynamically balance the features of the reconstructed sub-waveform segments and the original sub-waveform segments. The original sub-waveform segments refer to the sub-waveform segments remaining after excluding saturated sampling points in the abnormal sub-waveform segments.

[0042] Furthermore, after reconstructing the amplitude values ​​of the saturated sampling points, the method further includes:

[0043] If the saturated sampling points are obtained through downsampling, the sampling density of the reconstructed sub-waveform segment is restored using a cubic spline interpolation algorithm based on the function fitting between consecutive sampling points, so as to ensure the continuity of the wave shape.

[0044] The lidar saturated waveform reconstruction method based on hybrid neural networks provided in this application has high accuracy in identifying saturated sampling points and small reconstruction error, which can significantly improve lidar ranging accuracy and waveform data availability. Attached Figure Description

[0045] Figure 1 A flowchart of the lidar saturation waveform reconstruction method based on a hybrid neural network provided in this application is shown.

[0046] Figure 2 The diagram shows a comparison of the real-world lidar waveform before and after reconstruction, as provided in the embodiments of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this technical solution clearer, the following detailed description, in conjunction with specific embodiments, further illustrates this technical solution. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of this technical solution.

[0048] Please see as follows Figure 1 The flowchart shown is for a lidar saturation waveform reconstruction method based on a hybrid neural network. Figure 1 As shown, the method includes:

[0049] S101. After normalizing the full waveform data of the lidar, a fixed-length sliding window is used to traverse the data to extract the sub-waveform segments with larger total amplitude within the sliding window.

[0050] In this step, since a saturation sampling point can appear continuously in multiple sub-waveform segments, it is necessary to select multiple sub-waveform segments with larger sums of amplitude within the sliding window.

[0051] In practical implementation, the full waveform data of the lidar is normalized using the following methods:

[0052] To standardize the amplitude scale of different waveforms, normalization is performed on each waveform using the following formula:

[0053] ; (1)

[0054] In the formula, This is the current waveform number. The amplitude value at each sampling point This is the current waveform number. The normalized amplitude value of each sampling point and These are the minimum and maximum amplitude values ​​in the current waveform.

[0055] Furthermore, the sub-waveform segments with larger sums of amplitudes within the sliding window are extracted using the following method:

[0056] Assume the normalized full waveform is Where N is the number of sampling points in the full waveform data, and the length of the sliding window is L, then at each starting position... The total amplitude within the sliding window is calculated as follows:

[0057] ; (2)

[0058] Select to make Largest index The sub-waveform segment most likely to have a saturation sampling point is obtained as follows:

[0059] ; (3)

[0060] In the formula, This indicates the amplitude value at the beginning of the sub-wavelength segment. This indicates the amplitude value at the end position of the sub-waveform segment.

[0061] As an example, L is set to 60, but other values ​​can be used depending on the actual situation. This application does not impose any restrictions on this value.

[0062] S102. The local mutation features and global morphological features of each sub-waveform segment are extracted using a dual-branch convolutional network. Then, the extracted features are fused using a multi-head attention fusion network to detect abnormal sub-waveform segments and corresponding saturation sampling points.

[0063] The use of a dual-branch convolutional network to extract the local abrupt change features and global morphological features of each sub-waveform segment means that a convolutional network with a 3×1 small convolutional kernel is used to extract the local abrupt change features that can reflect the peak distortion waveform, and a convolutional network with a 5×1 large convolutional kernel is used to extract the global morphological features that can reflect the flat-top truncated waveform.

[0064] Here, one branch of the dual-branch convolutional network uses a one-dimensional convolutional kernel with a receptive field of 1×3 to enhance local information such as edge mutations; the other branch uses a large receptive field convolutional kernel of 1×5 to obtain global features such as broadening and plateauing.

[0065] In practical implementation, abnormal sub-waveform segments and corresponding saturation sampling points are detected using the following methods:

[0066] Step 1021: After concatenating the extracted local mutation features and global morphological features along the channel dimension, the extracted features are weighted and fused using learnable weight coefficients to obtain preliminary fused features.

[0067] Step 1022: Each attention head performs a linear projection on the preliminary fused features to obtain a query vector, a key vector, and a value vector.

[0068] Step 1023: After concatenating the query vector, key vector and value vector of multiple attention heads, perform linear mapping to obtain fused features to determine whether each sub-waveform segment is abnormal.

[0069] Step 1024: Determine the saturation sampling point by identifying the sub-waveform segments that are determined to be abnormal.

[0070] In addition, the method also includes:

[0071] Step 201: During the training phase of the dual-branch convolutional network and the multi-head attention fusion network, to address the extreme imbalance problem where the proportion of saturated sampling points in the full waveform sample data of the LiDAR is less than 5%, the loss function is constructed in the following way:

[0072] Step 2011: Calculate the proportion of saturated sampling points in the full waveform sample data of the LiDAR in the current training round to obtain the category weighting factor. :

[0073] ; (4)

[0074] In the formula, Parameters are adjusted for dynamic categories. This represents the number of saturated sampling points in the full waveform sample data of the lidar. This represents the total number of sampling points in the full waveform sample data of the lidar.

[0075] As an example, It can be set to 0.03.

[0076] Step 2012: Introduce the focus factor of easy and difficult samples, and adjust the loss function together with the category weighting factor. :

[0077] ; (5)

[0078] ; (6)

[0079] In the formula, As a prediction probability adjustment factor, For focus adjustment parameters, For the true labels of the samples, Predict probabilities for the model.

[0080] S103. A reconstruction network is established based on the Transformer encoder structure and the residual convolution structure, and the amplitude value of the saturated sampling point is reconstructed to obtain the reconstructed sub-waveform segment.

[0081] In practical implementation, the reconstruction network reconstructs the amplitude values ​​of the saturated sampling points in the following manner:

[0082] The Transformer encoder is used to model the long-range dependency of the original sub-waveform segments, and residual convolution is combined to enhance the local detail restoration capability. The amplitude value of the sampling point is reconstructed using the Wave block decoder. At the same time, a gating fusion mechanism is introduced to dynamically balance the features of the reconstructed sub-waveform segments and the original sub-waveform segments. The original sub-waveform segments refer to the sub-waveform segments remaining after excluding saturated sampling points in the abnormal sub-waveform segments.

[0083] Here, the Transformer encoder is used to encode the features of each sampling point in the original sub-waveform segment, and the Wave block decoder is used to decode based on the feature encoding of the sampling points in order to reconstruct the amplitude value of the sampling points.

[0084] In addition, the positions of the excluded saturated sampling points are located in the original sub-waveform segments, and the amplitude values ​​of the sampling points reconstructed by the reconstruction network are used to cover them to obtain the reconstructed sub-waveform segments.

[0085] Furthermore, before reconstructing the amplitude values ​​of the saturated sampling points, the method further includes:

[0086] Step 301: Based on the number of detected saturated sampling points, select a suitable reconstruction mechanism using the following method:

[0087] Step 3011: When the number of saturated sampling points is greater than 1 and less than or equal to 6, the amplitude values ​​of the saturated sampling points are reconstructed using a parallel reconstruction network.

[0088] Step 3012: When the number of saturated sampling points is greater than 6, the saturated sampling points are downsampled and then reconstructed using a parallel reconstruction network.

[0089] Specifically, when the number of saturated sampling points is 7 or 9, downsampling is performed by sampling at intervals with a step size of 2. When the number of saturated sampling points is 8 or 10, non-uniform sampling is performed according to predefined rules.

[0090] Furthermore, after reconstructing the amplitude values ​​of the saturated sampling points, the method further includes:

[0091] Step 401: If the saturated sampling points are obtained through downsampling, then based on the function fitting between continuous sampling points, the sampling density of the reconstructed sub-waveform segment is restored using a cubic spline interpolation algorithm to ensure the continuity of the wave shape.

[0092] S104. Insert the reconstructed sub-waveform segments into the corresponding positions of the normalized full waveform data of the lidar, and then perform the inverse normalization operation to obtain the reconstructed full waveform data.

[0093] In this step, an inverse normalization operation is performed based on the minimum and maximum amplitude values ​​used in the normalization process to obtain the reconstructed full waveform data. Specifically, it is expressed as follows:

[0094] ; (7)

[0095] In the formula, To reconstruct the actual amplitude value of the full waveform data.

[0096] Here, please refer to... Figure 2 The image shows a comparison of the LiDAR waveform before and after reconstruction in a real-world area. Figure 2 As shown, the method of this application can accurately identify saturated sampling points and perform reconstruction with high accuracy.

[0097] The above content is only a preferred embodiment of the present invention. For those skilled in the art, many changes can be made in the specific implementation and application scope based on the ideas of the present invention. As long as these changes do not depart from the concept of the present invention, they all fall within the protection scope of the present invention.

Claims

1. A method for reconstructing saturated waveforms of lidar based on hybrid neural networks, characterized in that, The method includes: After normalizing the full waveform data of the lidar, a fixed-length sliding window is used to traverse the data to extract the sub-waveform segments with larger total amplitude within the sliding window. Local mutation features and global morphological features of each sub-waveform segment are extracted using a dual-branch convolutional network. Then, a multi-head attention fusion network is used to fuse the extracted features to detect abnormal sub-waveform segments and corresponding saturation sampling points. A reconstruction network is established based on the Transformer encoder structure and the residual convolution structure, and the amplitude values ​​of the saturated sampling points are reconstructed to obtain the reconstructed sub-waveform segments. The reconstructed sub-waveform segments are inserted into the corresponding positions of the normalized full waveform data of the lidar, and then the inverse normalization operation is performed to obtain the reconstructed full waveform data. The method of fusing extracted features using a multi-head attention fusion network to detect abnormal sub-waveform segments and corresponding saturation sampling points includes: After the extracted local mutation features and global morphological features are concatenated along the channel dimension, they are weighted and fused using learnable weight coefficients to obtain preliminary fused features. Each attention head performs a linear projection on the preliminary fused features to obtain a query vector, a key vector, and a value vector; The query vector, key vector, and value vector of multiple attention heads are concatenated and then linearly mapped to obtain fused features, which are used to determine whether each sub-waveform segment is abnormal. The saturation sampling point is determined by the sub-waveform segment that is identified as abnormal; The reconstruction network reconstructs the amplitude values ​​of the saturated sampling points in the following manner: The Transformer encoder is used to model the long-range dependency of the original sub-waveform segments, and residual convolution is combined to enhance the local detail restoration capability. The amplitude value of the sampling point is reconstructed using the Wave block decoder. At the same time, a gating fusion mechanism is introduced to dynamically balance the features of the reconstructed sub-waveform segments and the original sub-waveform segments. The original sub-waveform segments refer to the sub-waveform segments remaining after excluding saturated sampling points in the abnormal sub-waveform segments.

2. The method as described in claim 1, characterized in that, The full waveform data of the lidar is normalized using the following method: To standardize the amplitude scale of different waveforms, normalization is performed on each waveform using the following formula: In the formula, This is the current waveform number. The amplitude value at each sampling point This is the current waveform number. The normalized amplitude value of each sampling point and These are the minimum and maximum amplitude values ​​in the current waveform.

3. The method as described in claim 1, characterized in that, The sub-waveform segments with larger sums of amplitudes within the sliding window are extracted using the following method: assuming the normalized full waveform is... Where N is the number of sampling points in the full waveform data, and the length of the sliding window is L, then at each starting position... The total amplitude within the sliding window is calculated as follows: ; Select to make Largest index The sub-waveform segment most likely to have a saturation sampling point is obtained as follows: In the formula, This indicates the amplitude value at the beginning of the sub-wavelength segment. This indicates the amplitude value at the end position of the sub-waveform segment.

4. The method as described in claim 1, characterized in that, The extraction of local abrupt change features and global morphological features for each sub-waveform segment using a dual-branch convolutional network refers to: Convolutional networks using 3×1 small kernels can extract local abrupt changes in peak-distorted waveforms, while convolutional networks using 5×1 large kernels can extract global morphological features of flat-topped truncated waveforms.

5. The method as described in claim 1, characterized in that, The method further includes: During the training phase of the dual-branch convolutional network and the multi-head attention fusion network, to address the extreme imbalance problem where the proportion of saturated sampling points in the full waveform sample data of LiDAR is less than 5%, the loss function is constructed in the following way: The proportion of saturated sampling points in the full waveform sample data of the LiDAR in the current training round is used to obtain the category weighting factor. : In the formula, Parameters are adjusted for dynamic categories. This represents the number of saturated sampling points in the full waveform sample data of the lidar. The total number of sampling points in the full waveform sample data of the lidar; a focus factor for easy and difficult samples is introduced, and the loss function is adjusted together with the category weighting factor. : ; In the formula, As a prediction probability adjustment factor, For focus adjustment parameters, For the true labels of the samples, Predict probabilities for the model.

6. The method as described in claim 1, characterized in that, Before reconstructing the amplitude values ​​of the saturated sampling points, the method further includes: Based on the number of detected saturated sampling points, a suitable reconstruction mechanism is selected using the following method: When the number of saturated sampling points is greater than 1 and less than or equal to 6, the amplitude values ​​of the saturated sampling points are reconstructed using a parallel reconstruction network. When the number of saturated sampling points is greater than 6, the saturated sampling points are downsampled and then reconstructed using a parallel reconstruction network. Specifically, when the number of saturated sampling points is 7 or 9, downsampling is performed by sampling at intervals with a step size of 2. When the number of saturated sampling points is 8 or 10, non-uniform sampling is performed according to predefined rules.

7. The method as described in claim 1, characterized in that, After reconstructing the amplitude values ​​of the saturated sampling points, the method further includes: If the saturated sampling points are obtained through downsampling, the sampling density of the reconstructed sub-waveform segment is restored using a cubic spline interpolation algorithm based on the function fitting between consecutive sampling points, so as to ensure the continuity of the wave shape.

Images