Spatial correlation score-based fractional order GM-APD lidar depth image denoising method

Abstract

The present application belongs to the technical field of laser radar imaging, and particularly relates to a fractional order integral GM-APD laser radar depth image denoising method based on spatial correlation, comprising the following steps: step one, a multi-scale superpixel fusion depth image completion algorithm is used to complete the target depth image obtained by histogram statistics; step two, a pixel neighborhood spatial correlation kernel function is introduced, a G-L fractional order integral operator is optimized, abnormal noise existing in the depth image is corrected, and GM-APD depth image denoising is realized. The present application effectively improves the quality of the processed depth image; can well play a role in depth image denoising processing, improves the target restoration degree and peak signal-to-noise ratio, and makes the processed depth image clearer; not only can realize GM-APD laser radar depth image denoising, but also still has strong applicability under the condition of less statistical frame number, effectively improves the GM-APD laser radar imaging frame frequency.

Description

Technical Field

[0001] This invention belongs to the field of lidar imaging technology, specifically relating to a spatially correlated fractional integral GM-APD lidar depth image denoising method. Background Technology

[0002] The target echo signal of the GM-APD lidar is severely affected by noise from the atmosphere and sunlight, resulting in a large amount of range anomalous noise in the acquired target depth image. In addition, since the GM-APD is a probabilistic statistical device, multiple statistical measurements are required to obtain the target's position information. When the number of statistical frames is insufficient, the detector will lose information because it does not detect any echo photons, resulting in a large number of empty pixels in the acquired target depth image and poor image quality.

[0003] Currently, denoising methods for depth images mainly fall into two categories. One is information-assisted depth image denoising, which uses color image guidance to denoise the target depth image. This effectively eliminates artifacts and weakens texture overlap in the guiding image, but its ability to recover image edges and depth values ​​needs improvement, and it requires additional image acquisition from an RGB camera. The other is direct depth image denoising, which uses median filtering and bilateral filtering to achieve good denoising results and also has a strong effect on handling singularities. However, the depth images processed by these two methods are too smooth, resulting in the loss of a large amount of edge detail information. Summary of the Invention

[0004] The purpose of this invention is to provide a spatially correlated fractional integral GM-APD lidar depth image denoising method to overcome the problems of existing technologies, such as the need for improvement in the recovery of image edges and depth values, the additional requirement for RGB cameras to acquire images, and the excessive smoothness of the filtered depth image, resulting in the loss of a large amount of edge detail information.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: a GM-APD lidar depth image denoising method based on spatial correlation fractional order, comprising the following steps:

[0006] Step 1: Use a multi-scale superpixel fusion depth image completion algorithm to complete the empty pixels in the target depth image obtained by histogram statistics;

[0007] Step 2: Introduce a pixel neighborhood spatial correlation kernel function and optimize the GL fractional integral operator to correct a large amount of anomalous noise in the depth image, thus achieving GM-APD depth image denoising. The specific steps are as follows:

[0008] 2.1 Set the maximum number of iterations of the fractional integral denoising algorithm;

[0009] 2.2 According to the derivation of the G-L fractional definition, construct a fractional integral operator based on spatial correlation, which contains two masks of sizes 3×3 and 5×5;

[0010] 2.3 According to the size of the fractional mask, construct a neighborhood spatial correlation matrix, optimize the fractional operator, and obtain a new denoising mask;

[0011] 2.4 Perform a convolution operation on the optimized fractional operator and the depth image data, and calculate the PSNR (Peak Signal-to-Noise Ratio) of the denoised depth image and the ideal depth image;

[0012] 2.5 Repeat step 2.4 until PSNR[n + 1]<PSNR[n] or the maximum number of iterations is reached, and stop the calculation;

[0013] 2.6 Output the denoised depth image.

[0014] Furthermore, in step 2.2 above, the construction process of the fractional integral operator based on spatial correlation is as follows

[0015] 1) Determine that the shortest equal-interval distance h = 1 at which the digital image changes occur, and obtain the v (v≥0) -order fractional integral expression of the unary signal through the G-L fractional integral definition formula,

[0016] 2) Generalize the fractional order from one dimension to two dimensions to obtain the fractional integral formulas for the x-axis and y-axis

[0017]

[0018]

[0019] 3) Obtain the numerical values of the fractional integral formula under the G-L definition along the X-axis and Y-axis directions from formulas (3) and (4), and the calculation expression is:

[0020]

[0021]

[0022] Extend formulas (6) and (7) to the other six directions to obtain the fractional integral operator W in eight directions .

[0023] Furthermore, in step 2) of step 2.2 above, according to formulas (3) and the multi-scale superpixel schematic diagram constructed from radar echo signals, construct the denoising process of the depth image using an 8-direction 3×3 mask and a 16-direction 5×5 mask.

[0024] Furthermore, in step 2.3 above, a neighborhood spatial correlation kernel function is introduced to optimize the fractional integral operator. The expression for the spatial correlation kernel function is as follows:

[0025]

[0026] Based on the spatial relationship between neighboring pixels, a neighborhood spatial correlation mask of a pixel is constructed using equation (13) and the spatial relationship diagram of neighboring pixels. The fractional integral operator is then optimized using the spatial correlation information to obtain the denoising operator in equation (14):

[0027] W = W v ·(14).

[0028] Furthermore, the above-mentioned GM-APD lidar depth image denoising method based on spatial correlation fractional order is characterized in that: in step 2.4, the obtained denoised depth image data

[0029]

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0031] 1. The spatial correlation-based fractional integral depth image denoising method provided by this invention constructs a neighborhood spatial correlation matrix, optimizes the fractional integral operator, and prevents the fractional integral operator from over-diffusion, which would cause the image to become over-smooth. This invention introduces fractional calculus theory into the depth image processing of GM-APD lidar, corrects anomalous noise in the depth image, obtains reasonable depth values, and effectively improves the quality of the processed depth image.

[0032] 2. The spatial correlation-based fractional integral depth image denoising method provided by this invention sets 3×3 denoising masks in 8 directions and 5×5 denoising masks in 16 directions during the fractional denoising process. It performs convolutional filtering on the GM-APD lidar depth image with a resolution of 64×64, which can effectively denoise the depth image, improve the target restoration and peak signal-to-noise ratio, and make the processed depth image clearer.

[0033] 3. The spatial correlation-based fractional integral depth image denoising algorithm designed in this invention can not only denoise the depth images of GM-APD lidar, but also has strong applicability even with a small number of statistical frames, effectively improving the imaging frame rate of GM-APD lidar. Attached Figure Description

[0034] Figure 1 Flowchart for denoising fractional integral operators;

[0035] Figure 2 For fractional integral operator Wv ;

[0036] Figure 3(a) shows a 3×3 mask and (b) shows a 5×5 mask;

[0037] Figure 4 Let α be the spatial location;

[0038] Figure 5 A spatial relationship diagram of neighboring pixels;

[0039] Figure 6 LiDAR system schematic diagram;

[0040] Figure 7 Target scene diagram (a) Real-world target (b) Target depth image;

[0041] Figure 8(a) shows a depth image obtained by peaking method and (b) shows a depth image obtained by multi-scale superpixel fusion algorithm;

[0042] Figure 9 Comparison of evaluation metrics between peak method and multi-scale superpixel fusion algorithm;

[0043] Figure 10 Denoising experiment results: (a) Denoising using bilateral filtering algorithm; (b) Denoising using median filtering algorithm; (c) Denoising using this algorithm (3×3 mask); (d) Denoising using this algorithm (5×5 mask).

[0044] Figure 11 Evaluation metrics for noise reduction experiments. Detailed Implementation

[0045] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0046] The design concept of this invention is as follows: First, a multi-scale superpixel fusion depth image completion algorithm is used to complete the empty pixels of the target depth image obtained by histogram statistics; then, a pixel neighborhood spatial correlation kernel function is introduced, and the GL fractional integral operator is optimized to correct a large amount of anomalous noise in the depth image, thereby realizing GM-APD depth image denoising.

[0047] Example:

[0048] like Figure 1 As shown, the specific steps of the spatially correlated fractional integral denoising algorithm are as follows:

[0049] Step 1: Employ a multi-scale superpixel fusion depth image completion algorithm to complete the target depth image obtained from histogram statistics with empty pixels. The specific steps are as follows:

[0050] 1. Input the raw data (range image), perform histogram statistics on all individual pixels by accumulating frame counts, obtain the superpixel histogram using a sliding window h, then extract the superpixel peak value, and iterate through all pixels to obtain the depth image; change the sliding window h and repeat the process to obtain depth images at different resolutions. Since the depth image resolution of the GM-APD lidar studied in this invention is 64×64, the maximum size of the sliding window h is set to 4, i.e., O = 4.

[0051] 2. Use the low-resolution depth image to fill in the empty pixel values ​​of the high-resolution depth image until O1 reaches the maximum resolution of 64×64;

[0052] Step 2: Introduce a pixel neighborhood spatial correlation kernel function and optimize the GL fractional integral operator to correct a large amount of anomalous noise in the depth image, thus achieving GM-APD depth image denoising. The specific steps are as follows:

[0053] 2.1. Set the maximum number of iterations for the fractional integral denoising algorithm;

[0054] 2.2 Based on the derivation of the fractional integral operator of GL, a spatially dependent fractional integral operator is constructed, which contains two masks of sizes 3×3 and 5×5. See [link to relevant documentation]. Figures 2 to 5 The construction process of the spatially dependent fractional integral operator is as follows:

[0055] like Figure 2 As shown, for a given real function f(t), t∈[a,b], and a<b, a,b∈R, then

[0056] The definition of a GL fractional integral of order v (v >= 0) is:

[0057]

[0058] in, h represents the integration step size.

[0059] Let the duration interval of a one-dimensional signal f(t) be [a, b]. If it is divided into equal units, i.e., h = 1, then we have: The equivalent expression for the fractional integral of order v (v≥0) of a univariate signal is:

[0060]

[0061] Define a two-dimensional image signal f(x,y). Utilize the separability of the Fourier transform to extend the fractional integral from one dimension to two dimensions. Since digital images are finite digital quantities, and the shortest distance for image changes occurs between adjacent pixels, digital images can only be represented by pixels in each direction. The minimum equal interval distance h = 1. Therefore, by dividing the two-dimensional image signal f(x,y) into equal parts according to the pixel interval h = 1, we can obtain the fractional integral formulas for the x-axis and y-axis.

[0062]

[0063]

[0064] N represents the number of terms in the polynomial. Let the coefficients of the v-th order fractional integral be... Then there is

[0065]

[0066] From equations (3) and (4), the numerical calculation expressions of the fractional integral formula under the definition of GL along the X and Y axes are:

[0067]

[0068]

[0069] Extending formulas (6) and (7) to the remaining six directions yields the fractional integral operator W for eight directions. v At this point, the integral convolution template of the image has rotation invariance, such as... Figure 2 .

[0070] in

[0071] wv0=1

[0072] wv1=v

[0073]

[0074] Setting N=3 means taking the first three terms of the expression. The approximate numerical solutions of the fractional integral along the X and Y axes are as follows:

[0075]

[0076]

[0077] In depth images, adjacent pixels share a certain degree of similarity. For noise reduction, feature information from the local neighborhood of the target pixel can be used to recover a reasonable distance value from the noise. Theoretically, a larger mask size results in a relatively blurrier image, but more neighborhood information is available, meaning it is less affected by noise; a smaller mask size results in a relatively clearer image, but the denoising effect is relatively worse. Therefore, this embodiment uses 3×3 and 5×5 fractional-order masks to process the depth image.

[0078] As shown in Figure 3, based on Equation (3) and the multi-scale superpixel diagram constructed with radar echo signals, a denoising process for depth images using an 8-direction 3×3 mask is constructed. To make fuller use of neighboring pixel information, the 5×5 mask uses 16 directions, i.e., α≠0.

[0079] Using a 16-directional mask optimizes the convolution process. Its advantage is that it fully utilizes the information of neighboring pixels. Compared with the original 8-directional 3×3 mask, this invention uses a 16-directional 5×5 mask to process the depth image, which utilizes more neighborhood information and means that it is less affected by noise. Experimental comparisons show that the 16-directional mask has a significantly better denoising effect.

[0080] in accordance with Figure 4 The value of α is determined by the spatial location of the pixel α and the spatial Euclidean distance in equation (11).

[0081]

[0082] Where (x1, y1) represents the position of the center pixel, (x i ,y j Let α be the spatial location of a neighboring pixel. Let the Euclidean distances between points a and b and the center pixel be d1 and d2, respectively, and the Euclidean distance between c and the center be d3. Let the inverse distances of d1, d2, and d3 be ρ1, ρ2, and ρ3, respectively. Since a, b, and c have different degrees of influence on the center, α is assigned a value according to equation (12) based on the inverse distance weights, i.e.:

[0083]

[0084] like Figure 5 As shown, the fractional integral operator diffusion model improves the denoising effect of noisy images to a certain extent and can better preserve the boundary (the high-frequency part that is correlated with the surrounding pixels). However, it will still over-difflate in flat areas (low-frequency part), producing a staircase effect.

[0085] 2.3 Based on the size of the fractional-order mask, construct the neighborhood space correlation matrix, optimize the fractional-order operator, and obtain a new denoising mask;

[0086] To solve the problem in 2.2, this step introduces a neighborhood space-related kernel function to optimize the fractional integral operator. According to the principle of similarity, the neighborhood space information of pixels (such as Figure 5 ) can, to a certain extent, suppress the influence of surrounding noise in image processing and better process the internal depth information of the image. The expression of the space-related kernel function is as follows:

[0087] <00,00223>

[0088] Based on the spatial relationship between neighboring pixels, a neighborhood space correlation mask of pixels is constructed from Equation (13) and Figure 5 to optimize the fractional integral operator using the spatial correlation information, and thus the denoising operator in Equation (14) can be obtained:

[0089] W = W v ·Y (14)

[0090] 2.4 Perform a convolution operation on the optimized fractional operator and the depth image data, and calculate the PSNR (Peak Signal-to-Noise Ratio) between the denoised depth image and the ideal depth image: <000023!>In this step, a convolution operation is performed on the noisy depth image by moving the mask pixel by pixel to obtain the denoised depth image data

[0092] <U

[0093] 2.5 Repeat Step 2. A until PSNR[n + 1]<PSNR[n] or the maximum number of iterations is reached, and then stop the calculation: During the iterative process of the denoising algorithm, I(x, y) itself contains a large amount of abnormal noise, and the iteration termination condition is that the current PSNR is less than the PSNR in the previous iteration or the maximum number of iterations is reached.

[0094] 2.6 Output the denoised depth image.

[0095] In this invention, a 64×64 array GM-APD lidar is built to verify the performance of the proposed algorithm. The lidar system is as Figure 6 shown.

[0096] Experimental equipment: a 1064nm fiber laser, a 64×64 array GM-APD camera, transmitting and receiving optical paths with a transmittance of 0.9, where the transmitting and receiving fields of view are 0.8°×0.8° respectively, the maximum laser output energy of the laser is 100 uJ, the pulse width is 5 ns, and the repetition frequency is 10 kHz. <U [[ID=4!]]

[0097] The lidar system uses a GM-APD as the detector. First, the emitting optical system illuminates the target area with a laser beam, generating a light feedback signal to activate the GM-APD's timing function. Once the laser beam undergoes diffuse reflection on the target surface, the receiving optical system focuses the light signal onto the focal plane of the GM-APD. At this point, the GM-APD stops timing. The readout circuit then converts the time-of-flight of the laser photons into a digital signal and transmits the data to a host computer for signal processing. After algorithmic processing, a depth image is displayed.

[0098] Imaging Experiment Verification: To verify the denoising performance of the proposed algorithm, an imaging experiment was conducted on a 1.531km residential building under strong sunlight. The target scene is shown in Figure 7.

[0099] When the number of frames counted is 25, the target depth image is reconstructed using the peak method and the depth image completion algorithm based on multi-scale superpixel fusion, respectively. The results are shown in Figure 8.

[0100] As shown in Figure 8, the algorithm proposed in this invention, compared to the peak method, can clearly reveal the outline of the target, and the number of noise points is relatively reduced. The evaluation metrics for both are as follows: Figure 9 As shown.

[0101] from Figure 9 As can be seen, the multi-scale superpixel fusion algorithm improves the target fidelity by 43.3% and the peak signal-to-noise ratio by 14.9%.

[0102] Since the extracted depth image still contains a large amount of anomalous noise, denoising processing is performed on the depth image. Median filtering, bilateral filtering, and the fractional integral denoising algorithm proposed in this invention (mask 3×3 and mask 5×5) are used respectively. The denoised depth image is shown below. Figure 10 As shown, Figure 11 These are the evaluation indicators.

[0103] from Figure 10 It can be seen that although bilateral filtering and median filtering can reveal a relatively complete outline of the target, a large amount of noise still remains. The algorithm proposed in this invention, whether using a 3×3 mask or a 5×5 mask fractional integral algorithm, essentially removes most of the noise, with only a small portion concentrated in the background region. Figure 11 The evaluation metrics show that the algorithm proposed in this invention outperforms bilateral filtering and median filtering in both target restoration accuracy and peak signal-to-noise ratio (PSNR). Compared to median filtering, the 3×3 mask denoising algorithm proposed in this invention improves the K-value by 33.3% and PSNR by 6.2%; for the 5×5 mask, the K-value improves by 22.8% and PSNR by 22.8%. Based on the above experimental data, the effectiveness of the algorithm proposed in this invention for denoising GM-APD LiDAR depth images is verified.

[0104] The above embodiments are merely preferred technical solutions of the present invention and should not be regarded as limitations on the present invention. The scope of protection of the present invention should be the technical solution described in the claims, including equivalent substitutions of the technical features described in the claims. That is, equivalent substitutions and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. A GM-APD lidar depth image denoising method based on spatial correlation fractional order, characterized by: Include the following steps Step 1: Use a depth image completion algorithm based on multi-scale superpixel fusion to complete the empty pixels of the target depth image obtained by histogram statistics; Step 2: Introduce a pixel neighborhood space correlation kernel function to optimize the G-L fractional integral operator, and correct a large amount of abnormal noise in the depth image to achieve GM-APD depth image denoising. The specific steps are as follows: 2.

1. Set the maximum number of iterations of the fractional integral denoising algorithm; 2.

2. According to the derivation of the G-L fractional order definition, construct a fractional integral operator based on spatial correlation, which contains two masks of sizes 3×3 and 5×5; 2.

3. According to the size of the fractional order mask, construct a neighborhood space correlation matrix, optimize the fractional order operator, and obtain a new denoising mask; 2.

4. Perform a convolution operation on the optimized fractional order operator and the depth image data, and calculate the peak signal-to-noise ratio PSNR between the denoised depth image and the ideal depth image; 2.

5. Repeat step 2.4 until PSNR[n + 1]<PSNR[n] or the maximum number of iterations is reached, and stop the calculation; 2.

6. Output the denoised depth image; In step 2.3, a neighborhood space correlation kernel function is introduced to optimize the fractional integral operator. The expression of the space correlation kernel function is as follows: According to the spatial relationship between neighboring pixels, construct a neighborhood space correlation mask for pixels from Equation (13) and the neighborhood pixel spatial relationship diagram, and use the spatial correlation information to optimize the fractional integral operator to obtain the denoising operator in Equation (14): W=W v ·Y (14); where W v is a fractional integral operator of order eight in eight directions.

2. The spatial correlation score-based GM-APD lidar depth image denoising method according to claim 1, characterized in that: In step 2.2, the construction process of the fractional integral operator based on spatial correlation is as follows 1) Determine the shortest equal-interval distance h = 1 at which the digital image changes, and obtain the v-th fractional integral expression of the unary signal through the G-L fractional integral definition formula, where v≥0; 2) Generalize the fractional order in one dimension to two dimensions to obtain the fractional integral formulas for the X-axis and Y-axis; 3) Obtain the numerical values of the fractional integral formula under the G-L definition along the X-axis and Y-axis directions from the fractional integral formulas for the X-axis and Y-axis. The calculation expression is: wherein, is the vth fractional integral coefficient; Extending equations (6) and (7) to the remaining six directions, we obtain the fractional integral operator W in eight directions v .

3. The spatial correlation score-based GM-APD lidar depth image denoising method according to claim 2, characterized in that: In step 2.2(2), according to the fractional integral formula for the X-axis and the multi-scale superpixel schematic diagram constructed with radar echo signals, construct denoising processing of the depth image using an 8-direction 3×3 mask and a 16-direction 5×5 mask.

4. The GM-APD lidar depth image denoising method based on spatial correlation fractional order according to claim 3, characterized in that: Step 2.4, the obtained denoised depth image data

Images