A three-dimensional imaging method based on adaptive scanning

Abstract

The present invention belongs to the field of laser radar technology, and specifically discloses a three-dimensional imaging method of adaptive scanning. The present invention adopts an adaptive scanning method to solve the problem of long laser radar acquisition time under large background conditions, and adopts an innovative regularized denoising model to implement three-dimensional reconstruction, thereby obtaining a better depth map. The main steps include: Step 1, using an adjustable spot size laser radar to realize pre-scanning and spot prior; Step 2, generating a spatial blur kernel and a scanning area matrix; Step 3, determining a specific scanning area for single-photon data acquisition according to the scanning matrix; Step 4, constructing a regularized denoising optimization function model; Step 5, subjecting the experimental data to regularized denoising to realize three-dimensional reconstruction and restore the depth map.

Description

Technical Field

[0001] The present invention belongs to the technical field of laser radar, and in particular relates to a three-dimensional imaging method based on adaptive scanning. Background Art

[0002] Traditional 3D imaging LiDAR is a common active optical 3D imaging system, which has been widely used in industry, scientific research and even defense. However, with the increasing maturity of high-sensitivity photon detection technology and high-precision electronic timing technology, LiDAR technology represented by time-correlated single photon counting 3D imaging technology has gradually become a development trend in the field of long-distance imaging and target recognition.

[0003] Although single photon counting radar can respond to echo photon signals at the single photon level, after adopting the single photon counting system, while improving the detection sensitivity, the scattered noise from the light particles, as well as the influence of environmental background noise and detector dark counts become non-negligible, so it is necessary to increase the irradiation time to improve the signal-to-noise ratio of the echo photons, which greatly reduces the imaging performance and seriously affects the application of this technology in practice. In the field of large background target recognition, people often only care about the photon detection in the target area. Due to the low reflectivity and long distance of the background area, in the same acquisition time, the background area can often only collect a small amount of photons, which does not play a big role in target imaging. At the same time, since most laser radars use scanning mode, the relationship between the actual size of the spot and the actual size of the pixel interval will also affect the imaging. When the actual size of the spot is larger than the interval between pixels, the current detection pixel will introduce the information of the adjacent pixels, resulting in signal superposition, thereby reducing the resolution of the imaging.

[0004] Therefore, it is currently necessary to explore a method to overcome the influence of background scanning time and spot size, so as to achieve efficient and high-quality three-dimensional imaging in a large background environment, and enhance the practical application capability of time-correlated single photon counting three-dimensional imaging technology. Summary of the invention

[0005] Existing single-photon three-dimensional imaging methods have certain limitations, such as: ignoring the background detection efficiency problem, taking the same detection time for each pixel; ignoring the problem that the spot size is larger than the actual interval between pixels, assuming the spot is perfect; and poor imaging effect under low signal-to-noise ratio and low light flux conditions. The present invention provides a regularized denoising three-dimensional deconvolution imaging method with prior spot detection and adaptive scanning. By adopting a priori spot detection and adaptive scanning strategy in the imaging system, and regularized denoising in the algorithm, the quality of single-photon three-dimensional reconstructed images is greatly improved.

[0006] The present invention provides a three-dimensional imaging method of adaptive scanning, comprising the following steps:

[0007] Step 1: Use the prior light spot detection laser radar to perform a short-term rapid pre-scan of each pixel, record the photon count and photon arrival time of each pixel, and generate a three-dimensional matrix of pre-scan photon counts; at the same time, perform prior light spot detection and use integration to obtain the light spot energy distribution during the pre-scan period. The light spot prior process is carried out simultaneously with the pre-scan acquisition, and there is no need to measure the light spot information again after the scan is completed.

[0008] The laser radar uses an electromagnetic two-dimensional orthogonal scanning galvanometer for raster scanning. The scanning galvanometer is divided into a horizontal lens and a vertical lens, which control the horizontal and vertical scanning directions respectively. When the signal acquisition card outputs a fixed voltage, the horizontal and vertical lenses are maintained in a fixed position. The voltage sequence V is written using the signal acquisition card. x and V y , set the scan time T for each pixel i,j , realizing raster scanning of laser radar. i,j Indicates the scanning time of pixel (i, j);

[0009] Step 2, generating a spatial blur kernel h according to the pre-scan photon counting three-dimensional matrix and the spot energy distribution during the pre-scan period;

[0010] Step 3, obtain the pre-scan photon count histogram through the pre-scan photon count three-dimensional matrix, obtain the photon count of each pixel, select pixels above the photon count threshold as the scanning area, and record the scanning area in the scanning matrix M.

[0011] Step 4: Locate the target pixel for fine scanning according to the scanning matrix M, set the scanning voltage sequence in the horizontal and vertical directions, set the fine scanning time for each pixel, use the time-correlated single photon counting method, set a fixed gate width, collect single photon data, and generate a three-dimensional single photon counting histogram.

[0012] The scanning matrix M is used to locate the fine scanning target pixel, and the scanning time matrix is ​​set to T, which is expressed as

[0013] T i,j =M i,j ×t

[0014] t is the single pixel fine scanning time, T i,j is the scanning time of pixel (i, j). The voltage sequence V is adopted by scanning time matrix T x and V y Perform fine scanning to obtain a three-dimensional single photon counting histogram

[0015] Step 5: construct a regularized denoising optimization function model for the three-dimensional single photon counting histogram in step 4. Denoising is performed to achieve three-dimensional reconstruction and restore the depth map; the regularized denoising optimization function is related to the spatial blur kernel h.

[0016] Furthermore, in step 2, a spatial blur kernel h is formed according to the pre-scanning photon counting three-dimensional matrix and the spot energy distribution during the pre-scanning period, specifically:

[0017] The spatial blur kernel is based on the tensor product of the planar convolution kernel and the temporal convolution kernel, where the planar convolution kernel uses a planar kernel function matrix h p Indicates that the time convolution kernel uses the time kernel function matrix h t express;

[0018] The spatial blur kernel is expressed as:

[0019]

[0020] Planar kernel function matrix h p Using a two-dimensional matrix form; normalizing the spot energy distribution to obtain the plane distribution function h xy , by bringing the actual distance between pixels into the function h xy Estimate the plane kernel function matrix h p ;

[0021] Normalized function h of spot energy plane distribution xy It is expressed as:

[0022]

[0023] (σ x ,σ y ) is the diffusion half-height width in the horizontal direction x and the vertical direction y. (x, y) is the coordinate of a given pixel, and (x', y') is the coordinate of the illumination center pixel. The obtained spot energy distribution data is differentiated in the horizontal and vertical directions to obtain (σ x ,σ y ), by adjusting the lens group, we can obtain σ with different spot sizes. x and σ y .

[0024] For the pixel with the largest number of photons in the pre-scan photon counting three-dimensional histogram, the photon counting histogram at the pixel position is obtained, and the histogram is fitted with a piecewise exponential function to obtain the corresponding fitting function h t’ , time kernel function matrix h t h t’ The normalized result of .

[0025] Furthermore, in step 3, the scanning matrix M is a logic matrix composed of 0 and 1. When the number of pre-scanned photons of pixel (i, j) is greater than the photon number threshold, M i,j =1, indicating that the pixel needs to be scanned finely, otherwise M i,j =0, indicating that the pixel does not need to be scanned in detail.

[0026] The scan matrix M is expressed as:

[0027] M=M i,j ×E i,j

[0028] =(count i,j >β×avg)×E i,j

[0029] Among them, count i,j is the pre-scan photon count of pixel (i, j), avg is the average number of photons obtained from the pre-scan, β is the photon coefficient, E i,j is a logic matrix of all 1s, and the photon number threshold is expressed as β×avg.

[0030] Furthermore, in step 5, a regularized denoising optimization function model is constructed, and a penalty term of the convex function of the denoiser is adopted to prevent overfitting of the reconstruction of edge depth information.

[0031] The regularized denoising optimization function model is:

[0032]

[0033]

[0034] in, It is a regularized convex penalty function used to penalize the non-smoothness in 3D image reconstruction, and τ is the penalty coefficient;

[0035] is the loss function, expressed as:

[0036]

[0037] in, To initialize the scene single photon 3D matrix, To finely scan the single-photon three-dimensional matrix, is the convolution operation, b is the background noise; since the single photon detection principle obeys the Poisson distribution, satisfies the negative Poisson log-likelihood.

[0038] Furthermore, the above equation is solved by using the sequential quadratic approximation of the Poisson log-likelihood. At the kth iteration, The second-order Taylor series is used for approximate calculation. The iterative gradient descent method is adopted to quickly converge the loss function.

[0039] The next iteration is

[0040]

[0041] where α k is the scale similarity identity matrix.

[0042] This iteration can be approximated as a series of subproblems using the following separation, written as

[0043]

[0044]

[0045] where s k is the total step length of gradient descent, expressed as

[0046]

[0047] In this model, a denoising regularization approach is adopted to flexibly select an iterative optimization algorithm to minimize the above function and ensure that the result converges to the global optimal value defined as . The regularization term is

[0048]

[0049] Where f() is a denoiser, and a convex function penalty term is used to prevent overfitting of edge depth information reconstruction. Its penalty effect is proportional to the inner product of the image and its denoising residual. Therefore, the optimization problem can be transformed into

[0050]

[0051]

[0052] definition The gradient of can be expressed as

[0053]

[0054] Based on the assumption of availability of directional derivatives of the denoiser, Available Instead, the gradient expression can be further simplified to

[0055]

[0056] Beneficial effects: The present invention adopts the pre-scanning and spot prior process to distinguish the scene background and target and generate the spatial convolution kernel, determine the pixel position that needs to be scanned, and greatly improve the data acquisition efficiency. A regularized denoising optimization function model is constructed, which can effectively reconstruct the final depth map with obvious effect. The method is highly feasible and universal. The experimental results show that our method has a shorter photon collection time and better depth recovery effect under large background conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of the method of the present invention.

[0058] Figure 2 It is a diagram of the experimental device of the present invention.

[0059] Figure 3 This is a graph showing the relationship between the scanning time and the photon coefficient β.

[0060] Figure 4 It is the relationship between the root mean square error of the depth image and the photon coefficient β.

[0061] Figure 5 This is a comparison chart of the root mean square error of different algorithms. DETAILED DESCRIPTION

[0062] In order to deepen the understanding of the present invention, the present invention will be further described in detail below in conjunction with examples. The examples are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention.

[0063] Adaptive scanning 3D imaging methods, such as Figure 1 As shown, the following steps are included:

[0064] Step 1, using a priori spot detection laser radar to achieve pre-scanning and spot prior;

[0065] The present invention adopts a single-photon laser radar system device structure diagram as shown in Figure 2 As shown. A pulsed laser (Laser) with a wavelength of 530nm and a frequency of 500kHz is used. A spot size adjustment device (spot size adjustment device) is constructed using a lens group. The device consists of a fixed and movable lens A (Lens A) and a fixed lens B (Lens B) to adjust the spot size. The pulse light is split by a polarization beam splitter (PBS), one part of which is used for spot prior through an electronically coupled sensor (CCD), and one part is used for pre-scanning. An electromagnetic two-dimensional scanning galvo is used for scanning, and the number of scanning pixels is set to 100×100, and the horizontal and vertical scanning voltage range is set to: V xThe range is -12V to +12V, V y The range is -12V to +12V. x and V y Divide into 100 equal parts and sequence into voltage sequence V x and V y The receiving end uses a single photon avalanche diode (SPAD) to detect photon events, and a fixed gate width time-correlated single photon counting (TCSPC) to record the photon count and arrival time of each pixel.

[0066] The mathematical modeling of single-photon lidar is as follows:

[0067] The number of photons recorded at position (i, j) in time interval n can be expressed as:

[0068]

[0069] Where c is the speed of light, s is the periodic pulse, d is the depth value at a certain pixel, Δt is the time interval of the time-to-digital converter, g is the instrument response function, η∈[0,1) is the quantum efficiency, b λ Represents the influence of ambient light with wavelength λ;

[0070] Pre-scan process:

[0071] The detection time of each pixel is set to 50 microseconds, and the pre-scan time is 500 seconds. The data acquisition card is used to generate the voltage sequence V x and V y The raster scanning mode is adopted to minimize the impact of the galvanometer position switching. For each pixel, the photon count and photon arrival time of each pixel are recorded to generate a three-dimensional matrix of pre-scan photon counts.

[0072] The process of spot prior is as follows:

[0073] (1) Set the electronic coupling sensor to the integration mode, set the integration time equal to the pre-scan time, the pre-scan time is 500 s, and perform two-dimensional distribution accumulation of the spot energy.

[0074] (2) The normalized function of the spot energy plane distribution is:

[0075]

[0076] (σ x ,σ y ) is the diffusion half-height width in the horizontal direction x and the vertical direction y. (x, y) is the coordinate of a given pixel, and (x', y') is the coordinate of the illumination center pixel. The obtained spot energy distribution data is differentiated in the horizontal and vertical directions to obtain (σ x ,σy ).

[0077] (3) The spot size can be changed by adjusting the lens group. The experiment used three different spot sizes, as shown in Table 1.

[0078] Spot number 1 2 3 <![CDATA[σ x (mm)]]> 0.15 0.75 3.5 <![CDATA[σ y (mm)]]> 0.4 2 10

[0079] Step 2, generating a spatial blur kernel;

[0080] Spatial blur kernel generation:

[0081] (1) Find the pixel with the most single-photon counts in the pre-scan 3D, obtain the photon count histogram of the pixel, and use the following piecewise exponential function to fit the photon count histogram curve.

[0082]

[0083] Among them, [a1, a2, a3, a4, a5] are fitting parameters, a1 represents the amplitude of the photon response function, a2 represents the highest point of the photon response function, and a3 and a4 represent the left and right half slopes of the piecewise exponential function. By fitting the photon counting histogram data, [a1, a2, a3, a4, a5] with the minimum error can be obtained. Substituting the parameters into the obtained M(t) for normalization, the laser radar time blur kernel h is obtained. t .

[0084] (2) The actual distance Δs between measured pixels is 3 mm, and the plane blur kernel size is 3×3. Substituting the normalized function of the spot energy plane distribution into the 3×3 plane kernel function h p . Then the spatial kernel function is the tensor product of the plane kernel function and the time kernel function, that is

[0085]

[0086] Step 3, obtain the scanning area matrix M:

[0087] Calculate the average photon number avg of the pre-scanned pixels and obtain the fine scanning area matrix M, where M i,j =(count i.j >β×avg), β is the photon coefficient. i.j is the pre-scan photon count of pixel (i, j). When the pixel photon is greater than the photon coefficient multiplied by the average number of pre-scanned pixel photons, M i,j =1, indicating that the pixel (i, j) needs to be scanned finely, otherwise M i,j = 0, indicating that the point does not need fine scanning. Set the fine scanning time of each pixel point to 0.5s, then the fine scanning time matrix is ​​T = 0.5 × M.

[0088] Step 4: Determine the specific scanning area for single photon data acquisition according to the scanning matrix;

[0089] The scan time of each pixel is set by the fine scan time matrix T obtained in step 3, and the voltage sequence V x and V y By raster scanning, a fine scanning single photon three-dimensional matrix can be obtained. Figure 3 The relationship between the total scanning time and the photon coefficient β under the current detected background. When β is greater than 0.1 and less than 0.9, the adaptive scanning takes only half the time of the original scanning method, which can increase the scanning speed by more than double, and the actual number of scanned pixels does not change much. When β is greater than 1.3, although the time required for adaptive scanning is further reduced, many valid pixels will be lost.

[0090] The evaluation index of the depth map detected by LiDAR is the root mean square error (RMSE). The RMSE formula is:

[0091]

[0092] Where x and y are the number of rows and columns of the horizontal and vertical scan images of the laser radar, is the depth map after regularization and denoising, and d is the real clean depth map. i,j for The depth of pixel (i,j) in is the depth of d pixels (i, j).

[0093] Figure 4 The relationship between the root mean square error of the depth image and the photon coefficient β is shown in Figure 2. When β is greater than 0.1 and less than 0.9, the root mean square error of the depth image remains unchanged, indicating that the acquired area is the same. When β is greater than 1.3, many pixels on the target do not meet the acquisition requirements and are not acquired, resulting in the depth value of the pixel being set to the default background value, increasing the depth error.

[0094] Step 5: Construct a regularized denoising optimization function model;

[0095] For the regularized denoising process, the loss function

[0096] in, To initialize the scene single photon 3D matrix, To finely scan the single-photon three-dimensional matrix, is the convolution operation, and b is the background noise.

[0097] The regularized denoising optimization function model is:

[0098]

[0099]

[0100] in is a regularized convex penalty function used to penalize non-smoothness in 3D image reconstruction, and τ is the penalty coefficient. The above problem equation is solved by using the sequential quadratic approximation of the Poisson log-likelihood.

[0101] At the kth iteration, The second-order Taylor series can be used for approximate calculation, that is, the next iteration is

[0102]

[0103] where α k is the scale similarity identity matrix, is the first-order derivative identifier, is the first-order derivative function of F(). This iteration can be approximated by the following separation to obtain a series of sub-problems, written as

[0104]

[0105]

[0106] where s k is the total step length of gradient descent, expressed as

[0107]

[0108] In our algorithm, we adopt the denoising regularization approach to flexibly select the iterative optimization algorithm to minimize the above function and ensure that the result converges to the global optimal value defined as . The regularization term is

[0109]

[0110] Where f() is a denoiser, and its penalty effect is proportional to the inner product of the image and its denoised residual, so the optimization problem can be transformed into

[0111]

[0112]

[0113] In the present invention, s=1, H=I, When

[0114]

[0115]

[0116] definition The gradient of can be expressed as

[0117]

[0118] Based on the assumption of availability of directional derivatives of the denoiser, Available Instead, the gradient expression can be further simplified to

[0119]

[0120] Considering the gradient of the function, the rapid descent method is the simplest method, which is equivalent to the above equation, that is,

[0121]

[0122] Step 6: Use the regularized denoising algorithm to reconstruct the experimental data in three dimensions and restore the depth map;

[0123] The objective function of step 5 is solved by iterative solution. The initial input is the regularization parameter Denoising algorithm f() and its denoising level σ f , and the number of iterations N. Initialization When k is equal to 1 to N, the result of the denoiser is referenced Update next step results For the single-photon three-dimensional matrix obtained by regularization denoising, the depth result is obtained for each pixel to form a depth map.

[0124] The final test results are as follows Figure 5 shown. Figure 5 The data is obtained from experiments under three spot sizes, with a signal-to-noise ratio of about 22.7dB and an average number of photons per pixel of about 500. The method of the present invention is compared with the maximum likelihood estimation algorithm (ML) proposed by Shin and the reversible jump Markov Monte Carlo (RJMCMC) method proposed by Tachella. Shin's paper is Photon-efficient super-resolution laser radar, published in MIT in 2017. Tachella's paper is Bayesian 3D Reconsyruction of Complex Scenes from Single-Photon Lidar Data, published in the journal Siam Journal on Imaging in 2018. Compared with several other methods, the method of the present invention has a stable imaging effect and can achieve better depth recovery.

[0125] The above verification is only an implementation example of the present invention. It should be pointed out that the present invention is not limited to the use of gated time-correlated single photon counting, and is not limited to the use in single-photon large background environment imaging, nor is it limited to the single-photon imaging field such as single-photon high-resolution imaging. For ordinary technicians in this technical field, without departing from the principle of the present invention, several appropriate designs can be made for the parameter vector, and such changes or modifications should also be regarded as the protection scope of the present invention.

Claims

1. A three-dimensional imaging method of adaptive scanning, characterized in that: The steps include: Step 1: Use laser radar to perform pre-scanning, record the photon count of each pixel and the photon arrival time, and generate a three-dimensional matrix of pre-scan photon counts; at the same time, perform a priori light spot detection to obtain the light spot energy distribution during the pre-scanning period; Step 2, forming a spatial blur kernel h according to the pre-scan photon counting three-dimensional matrix and the spot energy distribution during the pre-scan period; Step 3, obtain a pre-scan photon count histogram by pre-scanning the photon count three-dimensional matrix, obtain the photon count of each pixel, select pixels above the photon count threshold as the scanning area, and record the scanning area in the scanning matrix M; Step 4: Locate and scan the target pixels according to the scanning matrix M, collect single photon data, and generate a three-dimensional single photon counting histogram. Step 5: construct a regularized denoising optimization function model for the three-dimensional single photon counting histogram in step 4. Denoising is performed to achieve three-dimensional reconstruction and restore the depth map; the regularized denoising optimization function is related to the spatial blur kernel h; The regularized denoising optimization function model is constructed and expressed as: in It is a regularized convex penalty function used to penalize the non-smoothness in 3D image reconstruction, and τ is the penalty coefficient; is the loss function, expressed as: in, To initialize the scene single photon 3D matrix, To finely scan the single-photon three-dimensional matrix, is the convolution operation, b is the background noise; since the single photon detection principle obeys the Poisson distribution, satisfies the negative Poisson log-likelihood.

2. The three-dimensional imaging method of adaptive scanning according to claim 1, characterized in that: In step 2, a spatial blur kernel h is formed according to the pre-scan photon count three-dimensional matrix and the spot energy distribution during the pre-scan period, specifically: The spatial blur kernel is based on the tensor product of the planar convolution kernel and the temporal convolution kernel, where the planar convolution kernel uses a planar kernel function matrix h p Indicates that the time convolution kernel uses the time kernel function matrix h t express; The spatial blur kernel is expressed as: Normalize the spot energy distribution to obtain the plane distribution function h xy , by bringing the actual distance between pixels into the function h xy Estimate the plane kernel function matrix h p ; For the pixel with the largest number of photons in the pre-scan photon counting three-dimensional histogram, the photon counting histogram at the pixel position is obtained, and the histogram is fitted with a piecewise exponential function to obtain the corresponding fitting function h t’ , and then the fitting function h t’ Normalized to get the time kernel function matrix h t .

3. The three-dimensional imaging method of adaptive scanning according to claim 1, characterized in that: In step 3, the scanning matrix M is a logical matrix composed of 0 and 1. When the number of pre-scanned photons of pixel (i, j) is greater than the photon number threshold, M i,j =1, indicating that the pixel needs to be scanned finely, otherwise M i,j =0, indicating that the pixel does not need to be scanned in detail; The scan matrix M is expressed as: M=M i,j ×E i,j =(count i,j >β×avg)×E i,j Among them, count i,j is the pre-scan photon count of pixel (i, j), avg is the average number of photons obtained from the pre-scan, β is the photon coefficient, E i,j is a logic matrix of all 1s, and the photon number threshold is expressed as β×avg.

4. The adaptive scanning three-dimensional imaging method according to claim 1, characterized in that: The regularized denoising optimization function model is solved by using a sequential quadratic approximation of the Poisson log-likelihood.

5. The three-dimensional imaging method of adaptive scanning according to claim 1, characterized in that: The denoising regularization method is used to flexibly select the iterative optimization algorithm to minimize the function model and ensure that the result converges to the global optimal value defined as; the regularization term is Where f() is a denoiser, and its penalty effect is proportional to the inner product of the image and its denoised residual.

6. The adaptive scanning three-dimensional imaging method according to claim 5, characterized in that: The iterative process uses the second-order Taylor series for approximate calculation and the denoiser directional derivative availability assumption to further improve the optimization problem; at the kth iteration Using the second-order Taylor series approximation, the next iteration is where α k is the scale-similar identity matrix; this iteration uses the following separation approximation to obtain a series of subproblems, written as where s k is the total step length of gradient descent, expressed as definition The gradient of in, Based on the assumption of availability of directional derivatives of the denoiser, use Instead, the gradient expression is further simplified to

Images