Laser radar saturation waveform reconstruction method based on hybrid neural network

Abstract

The invention provides a laser radar saturation waveform reconstruction method based on a hybrid neural network, and the method comprises the steps: carrying out the normalization processing of laser radar full waveform data, carrying out the traversal through employing a sliding window with a fixed length, and extracting a sub-waveform segment with a large sum of amplitudes in the sliding window; respectively extracting local mutation features and global morphological features of each sub-waveform segment by using a double-branch convolutional network, fusing the extracted features by using a multi-head attention fusion network, and detecting abnormal sub-waveform segments and corresponding saturated sampling points; jointly establishing a reconstruction network based on an encoder structure and a residual convolution structure, and reconstructing the amplitude value of the saturated sampling point to obtain a reconstructed sub-waveform segment; and inserting the reconstructed sub-waveform segments into corresponding positions of the normalized laser radar full-waveform data, and carrying out reverse normalization operation to obtain reconstructed full-waveform data. The method is high in recognition accuracy and small in reconstruction error, and can remarkably improve the ranging precision of the laser radar and the usability of waveform data.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of laser radar data processing, and particularly relates to a laser radar saturated waveform reconstruction method based on a hybrid neural network. BACKGROUND

[0002] As an important means of current high-precision three-dimensional mapping, a full-waveform laser radar system can provide continuous distribution information of echo energy on a time axis, and is widely used in remote sensing tasks such as terrain modeling, vegetation structure inversion and city scene analysis. Compared with traditional point cloud products, full-waveform data contains richer echo features of ground objects, and has higher application value. However, in the actual measurement process, when the laser beam encounters a high reflectivity target (such as a metal roof, water surface or strongly reflective vegetation), the return signal energy may exceed the dynamic range of the receiving system, resulting in saturated distortion of the laser radar waveform. The saturated waveform often shows characteristics such as sharp peak distortion and flat top truncation, causing the loss of key physical characteristics, and thus affecting the accuracy of subsequent target identification and quantitative estimation. At present, some studies have attempted to recover saturated waveforms through signal interpolation, empirical modeling and regression reconstruction, but most of these methods are not ideal in dealing with complex saturated point distribution and large saturated region shape changes, and are difficult to adapt to the waveform distortion mode of multi-source sensors and multi-scene data. Therefore, how to design a waveform reconstruction method with structural adaptive ability and capable of processing multiple types of saturated modes has become a key problem that needs to be solved in the current full-waveform laser radar intelligent processing field. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a laser radar saturated waveform reconstruction method based on a hybrid neural network, which has high identification accuracy for saturated sampling points and small reconstruction error, and can significantly improve the laser radar ranging accuracy and waveform data availability.

[0004] The present application provides a laser radar saturated waveform reconstruction method based on a hybrid neural network, comprising: After normalizing the laser radar full-waveform data, a fixed-length sliding window is used for traversal to extract a sub-waveform segment with a larger amplitude sum in the sliding window; Local mutation features and global morphological features of each sub-waveform segment are extracted using a double-branch convolutional network, and the extracted features are fused using a multi-head attention fusion network to detect abnormal sub-waveform segments and corresponding saturated sampling points; A reconstruction network is established based on a Transformer encoder structure and a residual convolution structure, and the amplitude values of the saturated sampling points are reconstructed to obtain a reconstructed sub-waveform segment; The reconstructed wavelet segment is inserted into the corresponding position of the normalized laser radar full waveform data, and then an inverse normalization operation is performed to obtain reconstructed full waveform data.

[0005] Further, the laser radar full waveform data is normalized by the following method: In order to unify the amplitude scale of different waveforms, the normalization processing is performed on each waveform by the following formula: ; In the formula, is the amplitude value of the i-th sampling point of the current waveform, is the normalized amplitude value of the i-th sampling point of the current waveform, is the normalized amplitude value of the i-th sampling point of the current waveform, and and are the minimum and maximum amplitude values in the current waveform.

[0006] Further, the sub-waveform segment with a larger total amplitude in the sliding window is extracted by the following method: Suppose the normalized full waveform is , where N is the number of sampling points in the full waveform data, and the length of the sliding window is L. Then the total amplitude in the sliding window is calculated at each starting position : ; Select the index that makes the maximum, and the sub-waveform segment with the most likely saturated sampling points is obtained as: ; In the formula, represents the amplitude value at the starting position of the sub-waveform segment, represents the amplitude value at the terminal position of the sub-waveform segment.

[0007] Further, the use of a double-branch convolutional network to extract local mutation features and global morphological features of each sub-waveform segment refers to: The convolutional network with a 3x1 small convolution kernel is used to extract local mutation features that can reflect sharp peak distortion waveforms, and the convolutional network with a 5x1 large convolution kernel is used to extract global morphological features that can reflect flat truncated waveforms.

[0008] Further, the use of a multi-head attention fusion network to fuse the extracted features to detect abnormal sub-waveform segments and corresponding saturated sampling points includes: After the local mutation features and global morphological features are spliced in the channel dimension, a learnable weight coefficient is used for weighted fusion to obtain preliminary fusion features; Each attention head linearly projects the preliminary fusion feature to obtain a query vector, a key vector and a value vector; The query vector, the key vector and the value vector of the plurality of attention heads are spliced and then linearly mapped to obtain a fusion feature, so as to determine whether each sub-waveform segment is abnormal; The saturated sampling points are determined through the sub-waveform segments determined as abnormal.

[0009] Further, the method further comprises: In the training stage of the double-branch convolutional network and the multi-head attention fusion network, in order to solve the extreme imbalance problem that the proportion of saturated sampling points in the laser radar full waveform sample data is less than 5%, a loss function is constructed in the following manner: The proportion of saturated sampling points in the laser radar full waveform sample data in the current training round is counted to obtain a class weighting factor In the formula, is a dynamic class adjustment parameter, is the number of saturated sampling points in the laser radar full waveform sample data, is the total number of sampling points in the laser radar full waveform sample data; A difficulty sample focus factor is introduced, and the loss function is jointly adjusted with the class weighting factor In the formula, is a prediction probability adjustment factor, is a focus adjustment parameter, is a sample true label, is a model prediction probability.

[0010] Further, before reconstructing the amplitude values of the saturated sampling points, the method further comprises: According to the number of detected saturated sampling points, a suitable reconstruction mechanism is selected in the following manner: When the number of saturated sampling points is greater than 1 and less than or equal to 6, the amplitude values of the saturated sampling points are reconstructed by using a parallel reconstruction network; When the number of saturated sampling points is greater than 6, the saturated sampling points are down-sampled and then reconstructed by using a parallel reconstruction network; When the number of saturated sampling points is 7 or 9, the down-sampling operation is performed by interval sampling with a step of 2, and when the number of saturated sampling points is 8 or 10, non-uniform sampling is performed according to a predefined rule.

[0011] ​​​​​Further, the reconstruction network reconstructs the amplitude value of the saturated sampling point by the following manner: The original subwaveform segment is modeled by a Transformer encoder to model long-range dependencies, and a residual convolution is combined to enhance the local detail restoration capability. A Wave block decoder is used to reconstruct the amplitude value of the sampling point, and a gating fusion mechanism is introduced to dynamically balance the features of the reconstructed subwaveform segment and the original subwaveform segment. The original subwaveform segment refers to the subwaveform segment remaining after excluding the saturated sampling point in the abnormal subwaveform segment.

[0012] Further, after reconstructing the amplitude value of the saturated sampling point, the method further comprises: If the saturated sampling point is obtained through the downsampling operation, the sampling density of the reconstructed subwaveform segment is recovered by using a cubic spline interpolation algorithm based on the function fitting between the continuous sampling points, so as to ensure the continuity of the waveform.

[0013] The laser radar saturated waveform reconstruction method based on the hybrid neural network provided in the application has high identification accuracy for the saturated sampling point, small reconstruction error, and can significantly improve the laser radar ranging accuracy and waveform data availability. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A flowchart of the laser radar saturated waveform reconstruction method based on the hybrid neural network provided in the embodiment of the application is shown; Figure 2 A comparison chart before and after the real region laser radar waveform reconstruction provided in the embodiment of the application is shown. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical scheme and advantages of the technical scheme more clear and understandable, the technical scheme will be further described in detail below in combination with specific embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the technical scheme.

[0016] Please refer to the flowchart of the laser radar saturated waveform reconstruction method based on the hybrid neural network as shown in Figure 1 As shown in Figure 1 The method comprises: S101, after normalizing the laser radar full waveform data, a fixed length sliding window is used for traversal to extract a subwaveform segment with a larger amplitude sum in the sliding window.

[0017] In this step, since a saturated sampling point can appear continuously in multiple subwaveform segments, multiple subwaveform segments with a larger amplitude sum in the sliding window need to be selected here.

[0018] In specific implementation, the laser radar full waveform data is normalized by the following method: In order to unify the amplitude scale of different waveforms, normalization is performed on each waveform by the following formula: (1) In the formula, is the amplitude value of the i-th sampling point of the current waveform, is the normalized amplitude value of the i-th sampling point of the current waveform, and are the minimum and maximum amplitude values in the current waveform.

[0019] Further, a sub-waveform segment with a larger total amplitude in the sliding window is extracted by the following method: Suppose the normalized full waveform is where N is the number of sampling points in the full waveform data, and the length of the sliding window is L, then the total amplitude in the sliding window is calculated at each starting position (2) The index that makes maximum is selected, and the sub-waveform segment in which the saturated sampling point is most likely to appear is obtained as: (3) In the formula, represents the amplitude value at the starting position of the sub-waveform segment, represents the amplitude value at the terminal position of the sub-waveform segment.

[0020] As an example, L is taken as 60, and of course other values can also be taken according to actual conditions, which are not limited in the present application.

[0021] S102, local mutation features and global morphological features of each sub-waveform segment are extracted by using a double-branch convolutional network, and the extracted features are fused by using a multi-head attention fusion network to detect abnormal sub-waveform segments and corresponding saturated sampling points.

[0022] Among them, the local mutation features and global morphological features of each sub-waveform segment are extracted by using a double-branch convolutional network, which means that a convolutional network with a 3x1 small convolution kernel is used to extract local mutation features that can reflect sharp peak distortion waveforms, and a convolutional network with a 5x1 large convolution kernel is used to extract global morphological features that can reflect flat top truncated waveforms.

[0023] ​​​Here, one branch in the double-branch convolutional network adopts a one-dimensional convolutional kernel with a receptive field of 1x3 to strengthen local information such as edge mutations; the other branch adopts a large receptive field convolutional kernel of 1x5 to obtain global features such as expansion and platform.

[0024] In specific implementation, the abnormal sub-waveform segment and the corresponding saturated sampling point are detected in the following manner: Step 1021, after the extracted local mutation features and global morphological features are spliced in the channel dimension, the preliminary fusion features are obtained by using the learnable weight coefficients for weighted fusion.

[0025] Step 1022, each attention head linearly projects the preliminary fusion features to obtain query vectors, key vectors and value vectors.

[0026] Step 1023, after the query vectors, key vectors and value vectors of the multiple attention heads are spliced, linear mapping is performed to obtain fusion features, so as to determine whether each sub-waveform segment is abnormal.

[0027] Step 1024, the saturated sampling point is determined through the sub-waveform segment determined as abnormal.

[0028] In addition, the method further comprises: Step 201, in the training stage of the double-branch convolutional network and the multi-head attention fusion network, in order to solve the extreme imbalance problem that the proportion of saturated sampling points in the laser radar full waveform sample data is less than 5%, a loss function is constructed in the following manner: Step 2011, the proportion of saturated sampling points in the laser radar full waveform sample data in the current training round is counted to obtain a class weighting factor : ; (4) In the formula, is a dynamic class adjustment parameter, is the number of saturated sampling points in the laser radar full waveform sample data, is the total number of sampling points in the laser radar full waveform sample data.

[0029] As an example, can be set to 0.03.

[0030] Step 2012, a difficulty sample focus factor is introduced, and the loss function is jointly adjusted with the class weighting factor : ; (5) ; (6) In the formula, is a prediction probability adjustment factor, a focal point adjustment parameter, a sample true label, a model prediction probability.

[0031] S103, jointly establishing a reconstruction network based on a Transformer encoder structure and a residual convolution structure, and reconstructing the amplitude values of the saturated sampling points to obtain a reconstructed sub-waveform segment.

[0032] In a specific implementation, the reconstruction network reconstructs the amplitude values of the saturated sampling points in the following manner: The Transformer encoder is used to model long-range dependencies of the original sub-waveform segment, and the residual convolution is used to enhance the local detail restoration capability. The Wave block decoder is used to reconstruct the amplitude values of the sampling points, and a gating fusion mechanism is introduced to dynamically balance the features of the reconstructed sub-waveform segment and the original sub-waveform segment; wherein the original sub-waveform segment refers to the sub-waveform segment remaining after excluding the saturated sampling points in the abnormal sub-waveform segment.

[0033] Here, the Transformer encoder is used to encode the features of each sampling point in the original sub-waveform segment, and the Wave block decoder is used to decode the feature encoding of the sampling points to reconstruct the amplitude values of the sampling points.

[0034] In addition, the positions of the excluded saturated sampling points in the original sub-waveform segment are located, and the reconstructed amplitude values of the sampling points are used to cover them to obtain the reconstructed sub-waveform segment.

[0035] In addition, before reconstructing the amplitude values of the saturated sampling points, the method further comprises: Step 301, according to the number of detected saturated sampling points, a suitable reconstruction mechanism is selected in the following manner: Step 3011, when the number of saturated sampling points is greater than 1 and less than or equal to 6, the amplitude values of the saturated sampling points are reconstructed by using a parallel reconstruction network.

[0036] Step 3012, when the number of saturated sampling points is greater than 6, the saturated sampling points are down-sampled, and then reconstructed by using a parallel reconstruction network.

[0037] When the number of saturated sampling points is 7 or 9, the down-sampling operation is performed by interval sampling with a step of 2, and when the number of saturated sampling points is 8 or 10, non-uniform sampling is performed according to a predefined rule.

[0038] Further, after reconstructing the amplitude values of the saturated sampling points, the method further comprises: Step 401, if the saturated sampling points are obtained through the downsampling operation, the sampling density of the reconstructed sub-waveform segment is recovered by using a cubic spline interpolation algorithm based on the function fitting between the continuous sampling points, so as to ensure the continuity of the waveform.

[0039] S104, the reconstructed sub-waveform segment is inserted into the corresponding position of the normalized laser radar full waveform data, and then a reverse normalization operation is performed to obtain reconstructed full waveform data.

[0040] In this step, the reverse normalization operation is performed according to the minimum and maximum amplitude values used in the normalization processing to obtain the reconstructed full waveform data , which is specifically represented as follows: ; (7) In the formula, is the actual amplitude value of the reconstructed full waveform data.

[0041] Here, please refer to the real area laser radar waveform reconstruction before and after comparison chart as shown in Figure 2 As shown in Figure 2 , the method of the present application can accurately identify the saturated sampling points and has high accuracy in reconstruction.

[0042] The above is only the preferred embodiment of the present application, for those skilled in the art, according to the technical content of the idea, in the specific implementation and application range can make many changes, as long as these changes do not deviate from the concept of the present application, all belong to the protection scope of the present application.

Claims

1. A laser radar saturation waveform reconstruction method based on a hybrid neural network, characterized in that: The method comprises: After normalizing the full-waveform data of the lidar, a fixed-length sliding window is used to traverse it to extract the sub-waveform segments with the largest sum of amplitudes within the sliding window. A dual-branch convolutional network is used to extract the local mutation features and global morphological features of each sub-waveform segment. The extracted features are then fused using a multi-head attention fusion network to detect abnormal sub-waveform segments and corresponding saturation sampling points. A reconstruction network is jointly established based on a Transformer encoder structure and a residual convolution structure, and the amplitude value of the saturated sampling point is reconstructed to obtain a reconstructed sub-waveform segment; The reconstructed sub-waveform segment is inserted into the corresponding position of the normalized lidar full waveform data, and then the denormalization operation is performed to obtain the reconstructed full waveform data.

2. The method according to claim 1, wherein The lidar full waveform data is normalized in the following way: In order to unify the amplitude scales of different waveforms, each waveform is normalized using the following formula: ； Where, The current waveform The amplitude value of the sampling point, The current waveform The normalized amplitude value of the sampling points, and are the minimum and maximum amplitude values ​​in the current waveform.

3. The method according to claim 1, wherein The sub-waveform segments with the largest sum of amplitudes within the sliding window are extracted in the following way: Assume that the normalized full waveform is , where N is the number of sampling points in the full waveform data, and the length of the sliding window is L. At each starting position The total amplitude in the sliding window is calculated as: ； Select Largest index , the sub-waveform segment with the most likely saturation sampling point is: ； Where, Indicates the amplitude value of the starting position of the sub-waveform segment, Indicates the amplitude value of the terminal position of the sub-waveform segment.

4. The method according to claim 1, wherein The method of extracting the local mutation features and global morphological features of each sub-waveform segment by using a dual-branch convolutional network is as follows: The convolutional network extraction using a small 3×1 convolution kernel can reflect the local mutation characteristics of the spike distortion waveform, and the convolutional network extraction using a large 5×1 convolution kernel can reflect the global morphological characteristics of the flat-top truncated waveform.

5. The method according to claim 1, wherein The multi-head attention fusion network is used to fuse the extracted features to detect abnormal sub-waveform segments and corresponding saturation sampling points, including: After the extracted local mutation features and global morphological features are spliced ​​in the channel dimension, they are weighted fused using learnable weight coefficients to obtain preliminary fusion features; Each attention head performs linear projection on the preliminary fused features to obtain a query vector, a key vector, and a value vector; After concatenating the query vectors, key vectors, and value vectors of multiple attention heads, linear mapping is performed to obtain fused features to determine whether each sub-waveform segment is abnormal. The saturation sampling point is determined by the sub-waveform segment determined to be abnormal.

6. The method according to claim 1, wherein The method further comprises: During the training phase of the dual-branch convolutional network and the multi-head attention fusion network, the loss function is constructed in the following way to address the extreme imbalance problem in which the saturated sampling points account for less than 5% of the full-waveform sample data of the lidar: Count the proportion of saturated sampling points in the full waveform sample data of the current training round of lidar to obtain the category weighting factor : ； Where, Adjust parameters for dynamic categories, is the number of saturated sampling points in the full waveform sample data of the lidar, is the total number of sampling points in the lidar full waveform sample data; Introduce the difficult and easy sample focus factor and adjust the loss function together with the category weighting factor : ； ； Where, is the predicted probability adjustment factor, is the focus adjustment parameter, is the true label of the sample, Predict probabilities for the model.

7. The method according to claim 1, wherein Before reconstructing the amplitude value of the saturation sampling point, the method further includes: Based on the number of detected saturated sampling points, the appropriate reconstruction mechanism is selected in the following way: When the number of saturation sampling points is greater than 1 and less than or equal to 6, the amplitude values ​​of the saturation sampling points are reconstructed using a parallel reconstruction network; When the number of saturated sampling points is greater than 6, the saturated sampling points are downsampled and then reconstructed using a parallel reconstruction network; When the number of saturated sampling points is 7 or 9, downsampling is performed by interval sampling with a step size of 2, and when the number of saturated sampling points is 8 or 10, non-uniform sampling is performed according to predefined rules.

8. The method according to claim 1, wherein The reconstruction network reconstructs the amplitude value of the saturation sampling point in the following manner: The Transformer encoder is used to model the long-range dependencies of the original sub-waveform segments, and the residual convolution is combined to enhance the ability to restore local details. The Wave block decoder is used to reconstruct the amplitude values ​​of the sampling points. At the same time, a gated fusion mechanism is introduced to dynamically balance the features of the reconstructed sub-waveform segments and the original sub-waveform segments. The original sub-waveform segments refer to the sub-waveform segments remaining after excluding the saturated sampling points in the abnormal sub-waveform segments.

9. The method according to claim 1, wherein After reconstructing the amplitude value of the saturation sampling point, the method further includes: If the saturated sampling points are obtained through a downsampling operation, the sampling density of the reconstructed sub-waveform segment is restored using a cubic spline interpolation algorithm based on function fitting between consecutive sampling points to ensure the continuity of the wave shape.

Images