Depth imaging method and system of single-photon laser radar
Through regularized denoising model and noise screening technology, combined with isotropic and anisotropic total variation, the depth imaging problem under the small number of photons and noise interference in single-photon lidar is solved, achieving high-precision depth reconstruction and edge Retained, improving imaging effect.
Patent Information
- Application Number
- CN202411607068.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-12
- Publication Date
- 2024-12-27
AI Technical Summary
In a single-photon lidar system, when the number of photons is small and there is noise interference, it is difficult to achieve high-precision depth imaging, resulting in increased imaging errors and poorer effects.
Using a regularized denoising model, combining isotropic and anisotropic total variation, a loss function of the photon probability distribution is established, and the reflectance and depth estimates are calculated by the alternating multiplier method, and the lack of photons is used to fly. Time pixels are subject to zero-complement and noise screening, retain signal photons, and restore depth maps.
In the case of fewer photons and noise interference, efficient and high-quality depth reconstruction is achieved, reducing the impact of pixel data loss and noise on depth estimation, retaining object edge information, and improving imaging accuracy.
Smart Images
Figure CN119199893A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of single-photon laser radar imaging, and in particular to a depth imaging method and system of a single-photon laser radar. Background Art
[0002] As an active detection technology, LiDAR can quickly and accurately obtain the three-dimensional spatial information of the target and is widely used in satellite ranging, intelligent driving, three-dimensional imaging, atmospheric detection, medical detection and other fields. Compared with passive optical imaging technology, active optical imaging uses the illumination provided by the system itself to detect scene target objects and is not affected by ambient light. Therefore, it has significant advantages in detecting dim targets. Single-photon detectors have highly sensitive photon detection capabilities. Combined with high-time-resolution time-to-digital converters and time-correlated single-photon counting technology, they can obtain the distance information of the target under conditions where the laser echo is very weak.
[0003] In a single-photon LiDAR system, for each pixel in the target object of the imaging scene, the system constructs a histogram of photon detection time by recording a certain number of echo signals received during the illumination time. The amplitude of this histogram is related to the reflectivity of the target, and the time corresponding to the peak of the histogram is related to the depth of the target object. As the signal acquisition time increases, the number of recorded photons increases, and the flight time and photon count histogram of each pixel becomes more refined, so that more accurate depth and reflectivity estimates can be obtained. A large amount of LiDAR data places higher demands on the storage and computing power of the device. However, in many scenarios, the signal returned by the target object is very weak, the number of photons is small, and it is impossible to support the construction of the flight time and photon count histogram. In addition, under the interference of noise, the error of depth imaging increases and the imaging effect deteriorates. Summary of the invention
[0004] Purpose of the invention: The purpose of the present invention is to provide a depth imaging method and system for a single-photon lidar, so as to solve the imaging problem when the number of photons is small and there is noise interference, and to achieve high-precision depth imaging in weak light conditions.
[0005] Technical solution: The single-photon laser radar depth imaging method of the present invention comprises the following steps:
[0006] Obtain the number of photons and photon flight time at each pixel of the target object based on the single-photon laser radar;
[0007] A regularized denoising model is established according to the photon count distribution and noise at each pixel of the target object; the regularized denoising model adds isotropic total variation and anisotropic total variation to the loss function of the photon probability distribution;
[0008] Calculating a reflectivity estimate of a target object using the regularized denoising model;
[0009] Filling zeros for pixels lacking photon flight time, determining whether the detected photons are signal photons or noise photons based on the reflectivity estimation value, and retaining the signal photons;
[0010] For signal photons, the regularized denoising model is used to calculate the depth estimation value of the target object and restore the depth map.
[0011] Further, the regularized denoising model Where L(u) is the loss function of the photon probability distribution, J(u) = J ani (u)-τJ iso (u), J iso (u) and J ani (u) represents isotropic total variation and anisotropic total variation respectively, τ is a parameter, u = {α, d}, α is the reflectivity, and d is the depth.
[0012] Further, zero padding is performed on pixels lacking photon flight time, and the detected photons are judged to be signal photons or noise photons according to the reflectivity estimation value, and retaining the signal photons includes:
[0013] For each pixel of the target object, the median of the photon flight time of the pixel and several pixels around it is calculated. For pixels whose medians cannot be calculated, the median is set to 0 to generate a median image of the photon flight time.
[0014] For each pixel (i, j) in the median image, calculate the absolute difference between its arrival time and the arrival time of the surrounding p pixels, and take the sum of the first q smallest absolute differences R i，j , if the following conditions are met, it is a signal photon, otherwise it is a noise photon;
[0015]
[0016] Among them, T r is a repeating cycle, is the total number of signal photons per repetition period, h(t) is the laser pulse, T p is the pulse signal width, B is the total noise count per repetition period, is an estimate of reflectivity.
[0017] Furthermore, the photon s detected in the t-th time bin at pixel (i, j) is i，j，t It is expressed as:
[0018]
[0019] Let u i，j={α i，j , d i，j}, the probability distribution of observing a photon is:
[0020]
[0021] Taking the negative log-likelihood of the probability distribution of the photon, we get the loss function L(u):
[0022]
[0023] Among them, b i，j are the background and dark counts at pixel (i, j), A is the linear projection of the target object; α i，j is the reflectivity at pixel (i, j), d i，j is the depth at pixel (i, j).
[0024] Furthermore, the regularized denoising model is solved by an alternating multiplier method.
[0025] The single-photon laser radar depth imaging system of the present invention comprises:
[0026] A laser detection unit, used to obtain the number of photons and the photon flight time at each pixel of the target object according to the single-photon laser radar;
[0027] A regularized denoising model building unit is used to build a regularized denoising model according to the photon count distribution and noise at each pixel of the target object; the regularized denoising model adds isotropic total variation and anisotropic total variation to the loss function of the photon probability distribution;
[0028] A reflectivity estimation unit, which calculates a reflectivity estimation value of a target object using the regularized denoising model;
[0029] A noise screening unit, used to fill zeros for pixels lacking photon flight time, determine whether the detected photons are signal photons or noise photons based on the reflectivity estimation value, and retain the signal photons;
[0030] The depth estimation unit calculates the depth estimation value of the target object using the regularized denoising model for the signal photons and restores the depth map.
[0031] The single-photon laser radar system of the present invention comprises a laser, a beam splitter, a scanning galvanometer and a single-photon detector;
[0032] The laser emits a light pulse, which passes through a beam splitter and then irradiates and scans a target object through a scanning galvanometer; the light pulse is reflected by the target object, and the reflected light passes through the scanning galvanometer and is received by a single-photon detector for photon detection to obtain the number of photons and the photon flight time at each pixel of the target object; the depth map of the target object is obtained using the depth imaging method of the single-photon laser radar.
[0033] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the depth imaging method of the single-photon laser radar is implemented.
[0034] The computer-readable storage medium described in the present invention stores a computer program, and when the computer program is executed by a processor, the depth imaging method of the single-photon laser radar is implemented.
[0035] Beneficial effects: Compared with the prior art, the advantages of the present invention are: (1) The present invention constructs an optimized function model of weighted anisotropy and isotropic regularization to smooth the image while retaining the edge of the object; (2) The present invention combines the reflectivity of the object with the spatial correlation to denoise the single-photon lidar data, thereby reducing the impact of pixel data loss and noise on depth estimation. In summary, the present invention denoises the single-photon lidar data and reconstructs it using weighted anisotropy and isotropic regularization, which can efficiently and high-quality reconstruct depth when the number of photons is small and there is noise interference. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 is a flow chart of the depth imaging method of the present invention;
[0037] Figure 2 A schematic diagram of a single-photon laser radar system according to an embodiment of the present invention;
[0038] Figure 3 A depth reconstruction comparison diagram of whether or not the photon data has been screened according to an embodiment of the present invention;
[0039] Figure 4 is a relationship diagram of the root mean square error of the depth image, the average number of photons and the signal-to-noise ratio of an embodiment of the present invention;
[0040] Figure 5 The depth imaging effect diagrams of different methods of the embodiments of the present invention are shown;
[0041] Figure 6 Graph showing the relationship between the root mean square error and the average number of photons of different methods according to an embodiment of the present invention. DETAILED DESCRIPTION
[0042] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.
[0043] like Figure 1 As shown, the depth imaging method of the single-photon laser radar comprises the following steps:
[0044] Step 1: Collect the scanning data of the single-photon lidar system and record the number of photons and flight time at each pixel in the target object.
[0045] The single-photon laser radar system of this embodiment is as follows Figure 2 As shown. The laser emits light pulses in a repetitive cycle. After the light pulses pass through the beam splitter (Beam Splitter, BS), the light beam passes through the scanning galvo (Scanning Galvo) to illuminate and scan the target object. The light beam is reflected by the target object, and the reflected light is guided to the single-photon detector (SPD) again through the BS. The photon detection is completed in the SPD. When the SPD detects a photon, it triggers a timestamp with picosecond resolution, records the flight time of the photon relative to the previous laser pulse, and obtains the photon flight data of each pixel of the target object. The photon flight data includes the photon flight time and the number of photons. The laser emits light pulses in a repetitive cycle. After the laser pulses pass through the beam splitter (Beam Splitter, BS), the light beam passes through the scanning galvo (Scanning Galvo) to illuminate and scan the target object. The light beam is reflected by the target object, and the reflected light is guided to the single-photon detector (SPD) again through the BS. The photon detection is completed in the SPD. When the SPD detects a photon, it triggers a timestamp with picosecond resolution, records the flight time of the photon relative to the previous laser pulse, and obtains the photon flight data of each pixel of the target object. The photon flight data includes the photon flight time and the number of photons. r The light pulse h(t) is emitted, and (i, j)∈{1, 2,,...,N r}×{1, 2, ..., N c} to index N r ×N c The position of the image pixel, s i，j，t represents the single photon lidar data detected in the tth time bin, and the total length of the time bin is T. Since the depth of each pixel is d i，j =c*t i，j / 2,t i，j represents the light pulse response time corresponding to the position (i, j). In order to avoid distance aliasing, this embodiment assumes that T r >2d max / c, where c is the speed of light in a vacuum, d max Represents the maximum depth of the target.
[0046] Step 2: Construct weighted isotropic and anisotropic regularized denoising optimization function models.
[0047] Since the observed photon counts follow a Poisson distribution, the number of photons s detected in the t-th time bin at pixel (i, j) is i,j，t It is expressed as:
[0048]
[0049] Among them, αi，j >0 represents the reflectivity of the laser pulse irradiating the object, b i，j represents the background and dark counts at that position. and B are the total signal photon number and noise count per repetitive pulse period.
[0050] According to the observation model, the probability of observing a photon can be expressed as follows: i，j ={α i，j , d i，j}, A is the linear projection of the target object:
[0051]
[0052] Taking the negative log-likelihood for the above formula, the loss function L(u) is expressed as:
[0053]
[0054] Weighted Isotropic and Anisotropic Total Variation Regularization The expression is as follows, the parameter τ∈[0,1] adjusts the ratio of anisotropic to isotropic total variation and controls the gradient sparsity at each pixel:
[0055] J(u)=j ani (u)-τJ iso (u)
[0056] J iso (u) and J ani (u) represents the isotropic total variation and anisotropic total variation, respectively, D x , D y denote the horizontal and vertical partial derivative operators respectively, and S = [D x ;D y ] is the gradient operator
[0057]
[0058] J ani (u):=||Du||1=||D x u||1+||D y u||1
[0059] Isotropic total variation regularization assumes that the signal or image changes uniformly in all directions and will smooth the image or signal equally in all directions. Anisotropic total variation regularization assumes that the signal or image changes differently in different directions and can retain more edge details. Weighted isotropic and anisotropic total variation regularization are added to the loss function as denoising regularization terms. The smoothness of the image is controlled by the weight τ. While removing noise, the image structure and edges are kept to the maximum extent. Therefore, the weighted isotropic and anisotropic regularization denoising optimization function model c(u) can be written as Referred to as regularized denoising model.
[0060] Step 3: The reflectivity of the target object is estimated through the regularized denoising model.
[0061] This embodiment uses the alternating multiplier method (ADMM) as an example to solve the regularized denoising model, which has both the variable decomposability of the dual method and the fast convergence characteristics of the multiplier method. The regularized denoising model is transformed into the following form of optimization problem to solve the reflectivity and obtain the reflectivity estimate
[0062]
[0063] In order to facilitate calculation, two auxiliary variables v and w are introduced. x ,w y ), becomes the following optimization problem:
[0064]
[0065] stα=v,α=w
[0066] Introducing the Lagrange multiplier, we get the augmented Lagrangian form of the equation:
[0067]
[0068] Where y,z are Lagrange multipliers and β>0 is the penalty parameter.
[0069] The ADMM solver equations are iteratively performed as follows:
[0070]
[0071] y k+1 =y k +β k (α k+1 -v k+1 )
[0072]
[0073] β k+1=σβ k
[0074] Step 4: Screen the photon flight data based on the reflectivity data combined with the spatial correlation of the target object to remove noise data.
[0075] In the case of low average photon number per pixel, i.e. low light, some pixels may lack photon flight time data. To solve this data missing problem, the present invention combines the spatial correlation between object reflectivity and depth and takes the median M of all photon flight times detected for pixel (i, j) and its surrounding eight pixels. i,j , for the pixel (i, j) at the edge of the image or the surrounding 8 pixels with no data and unable to obtain the median from the adjacent pixels, let M i,j = 0, thereby generating a median image M of the photon flight time. By filling the data gaps at the pixels, the misjudgment of noise screening caused by missing data is avoided, and a more accurate result is obtained.
[0076] Since the detected photon data s i,j,t It may come from noise. Image M is disturbed by noise, which will affect depth estimation. In order to reduce the impact of noise on depth estimation, noise screening is performed based on the proposed probability model of noise and signal photons. Specifically, for each pixel (i, j) in image M, the absolute difference between its arrival time and the arrival time of the surrounding eight pixels is calculated, and the sum of the first four smallest absolute differences R is taken. i,j If the detected photon meets the following conditions:
[0077]
[0078] It is considered that the detected photon belongs to the signal photon rather than the noise photon, where T p is the pulse signal width.
[0079] Step 5: The denoised photon flight data is passed through a regularized denoising model to estimate the depth of the target object and restore the depth map.
[0080] In this embodiment, the alternating multiplier method (ADMM) is used as an example to solve the regularized denoising model for the photon flight data after noise screening, and the regularized denoising model is converted into the following form of optimization problem to solve the depth and obtain the depth estimation value
[0081]
[0082] In order to facilitate calculation, two auxiliary variables v and w are introduced. x ,w y ), becomes the following optimization problem:
[0083]
[0084] std=v,d=w
[0085] Introducing the Lagrange multiplier, we get the augmented Lagrangian form of the equation:
[0086]
[0087] Where y,z are Lagrange multipliers and β>0 is the penalty parameter.
[0088] The ADMM solver equations are iteratively performed as follows:
[0089]
[0090] y k+1 =y k +β k (d k+1 -v k+1 )
[0091]
[0092] β k+1 =σβ k
[0093] Finally, the depth estimate Recover the depth map.
[0094] The Middlebury dataset is used to verify the method of the present invention and evaluate the impact of changes in the average number of signal and noise photons per pixel on the depth imaging of the method of the present invention.
[0095] The evaluation index of the depth map detected by LiDAR is the root mean square error (RMSE), which is expressed as:
[0096]
[0097] in, and d i,j are the estimated value and reference value of the depth respectively.
[0098] Figure 3 The comparison of the depth reconstruction results of the photon flight data screening is shown. Figure 3 (a), (b), and (c) are the reference image, the depth reconstruction result without noise screening, and the depth reconstruction result with noise screening, respectively. The depth map of the present invention after noise screening is visually significantly improved compared to the depth map without noise screening. The image is smoother, the presentation of the object is clearer and closer to the reference image, and the root mean square error is also significantly reduced.
[0099] Figure 4 The depth estimation error of the method of the present invention varies with the signal-to-noise ratio under different photon numbers (average signal photon number PPP is 3, 6, 12, 20 and 30). The experimental results show that the depth reconstruction effect is significantly improved with the increase of the signal-to-noise ratio. Even when the number of photons is small, the present invention can still achieve excellent depth reconstruction effect.
[0100] The method of the present invention is compared with the maximum likelihood estimation method (ML), the efficient 3D reconstruction algorithm, and the reversible jump Markov Monte Carlo method. The efficient 3D reconstruction algorithm is recorded in "Photon-Efficient Computational 3-D and Reflectivity Imaging With Single-Photon Detectors" published by Shin in 2015, and the reversible jump Markov Monte Carlo method is recorded in "Bayesian 3D Reconstruction of Complex Scenes from Single-Photon Lidar Data" published by Tachella in 2018.
[0101] Figure 5 The comparison of the reconstructed depth map between the method of the present invention and other methods is shown when the signal-to-noise ratio is 1. Figure 5 (a) in the figure is the reference image. Figure 5 (b) in the figure is the maximum likelihood estimation method. Figure 5 (c) in the above equation represents Shin's method. Figure 5 (d) in the equation represents Tachella's method. Figure 5 (e) represents the method of the present invention; Figure 5 In (b) to (e), the average photon numbers of the three rows of images from top to bottom are 3, 6, and 10. It can be clearly seen from the comparison chart that the method of the present invention has a good denoising effect, effectively fills the depth data at the vacant pixels, and retains clear object edge information, thereby obtaining a good depth reconstruction effect.
[0102] Figure 6 The root mean square error comparison between the method of the present invention and other methods in depth reconstruction is shown under the condition of a signal-to-noise ratio of 1. The results show that compared with other methods, the present invention can still maintain the stability of the imaging effect and achieve better depth recovery when the average number of photons is small.
[0103] The single-photon laser radar depth imaging system of the present invention comprises:
[0104] A laser detection unit is used to obtain the number of photons and the photon flight time at each pixel of the target object according to the single-photon laser radar, and calculate the depth of the target object;
[0105] A regularized denoising model building unit is used to build a regularized denoising model according to the reflectivity and depth of the target object; the regularized denoising model adds isotropic total variation and anisotropic total variation to the loss function of the photon probability distribution;
[0106] A reflectivity estimation unit, which calculates a reflectivity estimation value of a target object using the regularized denoising model;
[0107] A noise screening unit, used to fill zeros for pixels lacking photon flight time, determine whether the detected photons are signal photons or noise photons based on the reflectivity estimation value, and retain the signal photons;
[0108] The depth estimation unit calculates the depth estimation value of the target object using the regularized denoising model for the signal photons and restores the depth map.
[0109] like Figure 2 As shown, the single-photon laser radar system described in the present invention includes a laser, a beam splitter, a scanning galvanometer and a single-photon detector; the laser emits a light pulse, and the light pulse passes through the beam splitter and then irradiates and scans the target object through the scanning galvanometer; the light pulse is reflected by the target object, and the reflected light passes through the scanning galvanometer and is received by the single-photon detector for photon detection to obtain the number of photons and the photon flight time at each pixel of the target object; the depth map of the target object is obtained by using the depth imaging method of the single-photon laser radar.
[0110] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the depth imaging method of the single-photon laser radar is implemented.
[0111] The computer-readable storage medium described in the present invention stores a computer program, and when the computer program is executed by a processor, the depth imaging method of the single-photon laser radar is implemented.
[0112] The computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store program code in the form of instructions or data structures and that can be accessed by a computer.
[0113] The processor is used to execute the computer program stored in the memory to implement each step of the method involved in the above embodiment.
Claims
1. A single-photon laser radar depth imaging method, characterized in that: The steps include: Obtain the number of photons and photon flight time at each pixel of the target object based on the single-photon laser radar; A regularized denoising model is established according to the photon count distribution and noise at each pixel of the target object; the regularized denoising model adds isotropic total variation and anisotropic total variation to the loss function of the photon probability distribution; Calculating a reflectivity estimate of a target object using the regularized denoising model; Filling zeros for pixels lacking photon flight time, determining whether the detected photons are signal photons or noise photons based on the reflectivity estimation value, and retaining the signal photons; For signal photons, the regularized denoising model is used to calculate the depth estimation value of the target object and restore the depth map.
2. The depth imaging method of single-photon laser radar according to claim 1, characterized in that: The regularized denoising model Where L(u) is the loss function of the photon probability distribution, J(u) = J ani (u)-τJ iso (u), J iso (u) and J ani (u) represents isotropic total variation and anisotropic total variation respectively, τ is a parameter, u = {α, d}, α is the reflectivity, and d is the depth.
3. The depth imaging method of single-photon laser radar according to claim 1, characterized in that: Zero padding is performed for pixels lacking photon flight time, and the detected photons are judged to be signal photons or noise photons according to the reflectivity estimation value, and the signal photons are retained including: For each pixel of the target object, the median of the photon flight time of the pixel and several pixels around it is calculated. For pixels whose medians cannot be calculated, the median is set to 0 to generate a median image of the photon flight time.
4. The depth imaging method of single-photon laser radar according to claim 3, characterized in that: Zero padding is performed for pixels lacking photon flight time, and the detected photons are judged to be signal photons or noise photons according to the reflectivity estimation value, and the signal photons are retained including: For each pixel (i, j) in the median image, calculate the absolute difference between its arrival time and the arrival time of the surrounding p pixels, and take the sum of the first q smallest absolute differences R i,j , if the following conditions are met, it is a signal photon, otherwise it is a noise photon; Among them, T r is a repeating cycle, is the total number of signal photons per repetition period, h(t) is the laser pulse, T p is the pulse signal width, B is the total noise count per repetition period, is an estimate of reflectivity.
5. The depth imaging method of single-photon laser radar according to claim 1, characterized in that: The number of photons s detected in the tth time bin at pixel (i, j) i,j,t It is expressed as: Let u i,j ={α i,j ,b i,j }, the probability distribution of observing a photon is: Taking the negative log-likelihood of the probability distribution of the photon, we get the loss function L(u): Among them, b i,j are the background and dark counts at pixel (i, j), A is the linear projection of the target object; α i,j is the reflectivity at pixel (i, j), d i,j is the depth at pixel (i,j).
6. The depth imaging method of single-photon laser radar according to claim 1, characterized in that: The regularized denoising model is solved by the alternating multiplier method.
7. A single-photon laser radar depth imaging system, characterized in that: include: A laser detection unit, used to obtain the number of photons and the photon flight time at each pixel of the target object according to the single-photon laser radar; A regularized denoising model building unit is used to build a regularized denoising model according to the photon count distribution and noise at each pixel of the target object; the regularized denoising model adds isotropic total variation and anisotropic total variation to the loss function of the photon probability distribution; A reflectivity estimation unit, which calculates a reflectivity estimation value of a target object using the regularized denoising model; A noise screening unit, used to fill zeros for pixels lacking photon flight time, determine whether the detected photons are signal photons or noise photons based on the reflectivity estimation value, and retain the signal photons; The depth estimation unit calculates the depth estimation value of the target object using the regularized denoising model for the signal photons and restores the depth map.
8. A single-photon laser radar system, characterized in that: Includes laser, beam splitter, scanning galvanometer and single photon detector; The laser emits a light pulse, which passes through a beam splitter and then irradiates and scans a target object through a scanning galvanometer; the light pulse is reflected by the target object, and the reflected light passes through the scanning galvanometer and is received by a single-photon detector for photon detection to obtain the number of photons and the photon flight time at each pixel of the target object; a depth map of the target object is obtained using the depth imaging method of the single-photon laser radar described in any one of claims 1 to 6.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into the processor, the depth imaging method of the single-photon laser radar according to any one of claims 1-6 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the depth imaging method of the single-photon laser radar according to any one of claims 1-6 is implemented.