Undersampled non-line-of-sight imaging method based on deconvolution optimization
By adopting a deconvolution optimization method based on non-sight imaging technology, combining transient images and prior information of targets, the problem of low reconstruction quality under undersampling conditions is solved, and efficient and low-cost non-sight is achieved Domain imaging.
Patent Information
- Application Number
- CN202411000554.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-07-25
AI Technical Summary
Existing non-field-of-sight imaging technology is difficult to achieve high-quality reconstruction under undersampling conditions, and it has high computational cost and low efficiency.
The method based on deconvolution optimization is adopted, combined with the transient image and the prior information of the target, and the generalized alternating projection method is used to solve the Fourier domain, optimize the model and solve it, and high-quality reconstruction of non-sight targets is achieved.
Provide high-quality reconstruction results under undersampling conditions, improve imaging efficiency, reduce computational costs, and enable clear target reconstruction at lower sampling rates.
Smart Images

Figure CN118518591B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of non-line-of-sight imaging, and in particular relates to an under-sampling non-line-of-sight imaging method based on deconvolution optimization. Background Art
[0002] Non-Line-Of-Sight (NLOS) technology can reconstruct targets outside the direct detection range of the detector and has a wide range of applications in medical imaging, autonomous driving, public safety, etc. In recent years, researchers have proposed many reconstruction methods for NLOS targets, such as speckle correlation, Fermat path, acoustic echo, occlusion-based imaging, and transient imaging.
[0003] Transient imaging technology has attracted much attention because of its ability to reconstruct the three-dimensional shape of hidden targets. In transient NLOS imaging technology, the measurement value is the encoded information of the hidden scene. Therefore, NLOS imaging can be mathematically expressed as an inverse problem. At present, many imaging models have been proposed to reconstruct hidden targets. A. Velten et al. conducted the first experimental demonstration of NLOS imaging and reconstructed the target using the back-projection (BP) method (Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature communications 3(1), 745 (2012)). Subsequently, O'Toole et al. simplified the formation of transient images into a three-dimensional convolution process and reconstructed the hidden target using the light cone transformation (LCT) method (Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555(7696), 338–341 (2018)). In 2019, the frequency-wavenumber migration (FK) method was proposed. This method uses the wave equation to describe the propagation of light and uses frequency domain interpolation technology to reconstruct the target (Wave-based non-line-of-sight imaging using fast fkmigration. ACM Transactions on Graphics (ToG) 38(4), 1–13 (2019)). The above imaging model realizes the reconstruction of NLOS targets. However, in order to ensure the accuracy of the inverse problem solution, these methods rely on high-precision transient imaging models and use uniform raster scanning to obtain transient data, which requires a lot of scanning time.
[0004] In order to improve imaging efficiency, many NLOS imaging methods under undersampling conditions have been proposed. Some researchers have designed specific scanning modes to improve sampling efficiency, such as circular scanning (Efficient non-line-of-sight imaging from transient sinograms. In: European Conference on Computer Vision, pp. 193–208 (2020). Springer), Archimedean spiral scanning (Non-line-of-sight imagingbased on archimedean spiral scanning. Optics Communications 537, 129450(2023)), etc. These scanning methods speed up NLOS imaging, but inevitably face the problems of limited observation range and reconstruction distortion. The idea of compressed sensing has also been introduced into NLOS imaging to strike a balance between acquisition time and imaging quality. In 2021, by exploiting the redundancy of traditional light field acquisition, ultrafast light field tomography (LIFT) can capture complete four-dimensional spatiotemporal information in a single snapshot (Ultrafast light fieldtomography for snapshot transient and non-line-of-sight imaging. Nature communications 12(1), 2179 (2021)). On this basis, single-pixel cameras in passive mode are used in NLOS imaging, effectively realizing spatial multiplexing detection (SMD), but the insufficient light flux caused by the coding mode leads to poor reconstruction results (None-line-of-sight imaging enhancedwith spatial multiplexing. Optics Express 30(4), 5855–5867 (2022)). Due to the development of deep learning, convolutional neural network models have been used to solve the NLOS reconstruction problem under undersampling conditions. Although deep learning methods can provide high-quality reconstruction results, their performance is heavily dependent on the training dataset.In recent years, reconstruction of NLOS targets under underscanning conditions is usually combined with regularization methods (Z.: Few-shot non-line-of-sight imagingwith signal-surface collaborative regularization. In: Proceedings of theIEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 13303–13312 (2023), Curvature regularization for non-line-of-sight imaging from under-sampled data. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)), which can obtain high-quality reconstruction. However, it should be noted that due to the high dimensionality of the measurements and the voxelization of the NLOS scene, solving the linear inverse problem is a difficult computational task, regardless of whether regularization is performed in advance. Summary of the invention
[0005] The object of the present invention is to provide an under-sampling non-viewing area imaging method based on deconvolution optimization to solve the problems raised in the above background technology.
[0006] To achieve the above object, the present invention provides the following technical solution: an under-sampling non-viewing-area imaging method based on deconvolution optimization, comprising:
[0007] Step 1: The pulse laser scans the relay wall, and the single photon avalanche diode SPAD collects the signal at each point to obtain an under-sampled photon histogram;
[0008] Step 2: Based on the prior information of the transient graph and the target, a transient graph-target joint prior optimization model is constructed. The model considers the Poisson signal distribution characteristics, the smoothness of the transient graph, the non-negativity of the non-viewing area target, and the sparsity prior constraints of the non-viewing area target.
[0009] Step 3, use the generalized alternating projection method to solve in the Fourier domain, optimize the model and solve it to obtain the optimized result of the non-viewing area target representation.
[0010] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.
[0011] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.
[0012] A computer program product comprises a computer program, which implements the steps of the above method when executed by a processor.
[0013] Compared with the prior art, the present invention has the following beneficial effects:
[0014] The present invention introduces a computationally efficient optimization-based deconvolution algorithm to solve the ill-posed problem, which can provide high-quality reconstruction under underscanning conditions; the present invention adopts a computationally simple framework to achieve NLOS imaging, instead of using a computationally expensive linear least squares method to calculate the volume albedo. In order to complete the deconvolution, the present invention combines the prior information of the transient image and the target, continuously updates and regularizes the transient image and the object, and deconvolves the iterated transient image with the original three-dimensional convolution kernel to obtain a better reconstruction effect. The present invention also develops an effective numerical algorithm based on the alternating direction multiplier method, and verifies its effectiveness on simulation data sets and experimental data sets. Using the prior information of the target and the transient image, the framework of the present invention can accurately reconstruct the target under undersampling conditions, providing a reference for fast and high-precision NLOS imaging. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0016] Figure 1 Schematic diagram of the under-scanned NLOS imaging system and algorithm design based on deconvolution optimization, where (a) is the optical system structure, (b) is the actual optical system, and (c) is the schematic diagram of the collaborative optimization algorithm and the normalization term ablation study.
[0017] Figure 2 Comparison of simulation data reconstruction methods at different scanning points.
[0018] Figure 3 Comparison of reconstruction methods for common data comparison under different underscanning modes.
[0019] Figure 4 Comparison of reconstruction methods based on public data at different exposure times, where a) is the comparison of reconstruction results of the target “sculpture” between this method and other methods at different exposure times, and b) is the comparison of reconstruction results of the target “dragon” between this method and other methods at different exposure times.
[0020] Figure 5 To compare the reconstruction results of LCT and the proposed method on the experimental data of different scanning points, a) is the comparison of the reconstruction results of the target "L" by the light cone transform LCT and the proposed method at different sampling rates, and b) is the comparison of the reconstruction results of the target "T" by the light cone transform LCT and the proposed method at different sampling rates.
[0021] Figure 6 Comparison of experimental data between LCT and the proposed method at different exposure times. DETAILED DESCRIPTION
[0022] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0023] Non-Line-Of-Sight (NLOS) imaging aims to reconstruct targets outside the field of view and has potential applications in various fields. However, a fundamental problem is that the process of obtaining transient information is very time-consuming, and it is difficult to balance the reconstruction quality and detection efficiency.
[0024] The present invention proposes a computationally efficient optimization-based deconvolution algorithm to solve the ill-posed problem, which can provide high-quality reconstruction under underscanning conditions. The present invention adopts a computationally simple framework to achieve NLOS imaging instead of using a computationally expensive linear least squares method to calculate the volume albedo. Specifically, due to the lack of data, NLOS imaging under undersampling conditions can be simplified to the problem of inaccurate deconvolution. In order to complete the deconvolution, the present invention combines the prior information of the transient image and the target, continuously updates and regularizes the transient image and the object, and deconvolves the iterated transient image with the original three-dimensional convolution kernel to obtain a better reconstruction effect. The present invention also develops an effective numerical algorithm based on the alternating direction method of multipliers (ADMM), and verifies its effectiveness on simulation data sets and experimental data sets. The results show that when the sampling rate of the original data is 0.15%, that is, when the acquisition speed is increased by about 642 times, the structural similarity index (SSIM) of the reconstructed target can reach 0.55. This is 0.3 higher than the index of LCT reconstruction of the target under the same conditions. Using the prior information of the target and transient images, the framework of the present invention can accurately reconstruct the target under undersampling conditions, providing a reference for fast and high-precision NLOS imaging.
[0025] The key to NLOS imaging is to measure the time-of-flight (ToF) and radiance of photons after multiple reflections interacting with the NLOS scene. Figure 1 As shown in (a): The laser emitted by the pulsed laser passes through the scanning galvanometer and illuminates the illumination point on the visible wall. Then, the photon moves toward the target Finally, the reflected photons are detected at the detection point on the visible wall. The light is reflected for the third time at the wall and captured by a detector with time resolution. Repeating this process at multiple illumination points and detection points can obtain 5-dimensional transient information of light transmission. The imaging model of the present invention makes the following assumptions: first, the reflection of light is isotropic; second, the light reflected from the wall is reflected only once on the surface of the hidden target, that is, there is no mutual reflection in the hidden scene; finally, there is no occlusion between the hidden scene parts. is the 3D scene coordinate, , They are the illumination point and detection point on the visible wall. The single photon avalanche diode SPAD collects a total of Points, The transient graph collected by the SPAD at time is modeled as:
[0026]
[0027] In the formula, is the full-sample transient measurement value, For the 3D reconstruction domain The volume albedo of the hidden scene in is the light transfer function, is the speed of light. When the integration domain is the focus and When the illumination point and the detection point are the same, that is, When , the NLOS confocal model can be obtained as follows:
[0028]
[0029] O'Toole et al. used LCT to simplify the forward model as follows:
[0030]
[0031] In the formula, is the vector representation of the light transient information, where is the number of lighting points, is the number of detection points, is the number of time bins. The convolution process for confocal NLOS imaging is encoded. is the vectorized representation of the NLOS volume albedo, where , , are the number of voxels respectively.
[0032] The measurement matrix Simplified as a convolution, the volume albedo can be recovered by inverting equation (3) . In this way, a computationally efficient deconvolution algorithm can be used to solve the NLOS imaging problem. Without sacrificing efficiency, this inverse problem can even be regularized using priors such as non-negativity, sparsity, and smoothness. To achieve fast NLOS imaging, the most direct way is to reduce the number of scanning points and shorten the acquisition time. However, due to the lack of data caused by underscanning, the collected transient image will lack effective information. In addition, due to the incomplete transient image, the convolution model of formula (3) becomes inaccurate. When the number of scanning points or the exposure time is reduced, the convolution model of formula (3) degenerates into:
[0033]
[0034] in Represents the histogram of under-sampling measurement, matrix The convolution process of undersampled NLOS imaging is encoded. Therefore, optimizing this deconvolution model is the key to achieving efficient computation of undersampled NLOS imaging.
[0035] The present invention proposes a new method to optimize the deconvolution process by using the transient image and the prior information of the target. Figure 1 As shown in (c) in the figure. Directly deconvolving the transient state graph under undersampling conditions will result in poor reconstruction, large noise, and almost indistinguishable shape of the object. Instead, we choose to constrain the transient state graph and target under undersampling first, and then perform deconvolution. When optimizing the transient state graph, not only the Poisson and smooth characteristics of the transient state graph itself are considered, but also the characteristics of the target are constrained, and the transient state graph is constrained through the forward model. This idea of collaborative prior optimization of deconvolution not only utilizes the efficient calculation of deconvolution, but also facilitates the use of regularization terms in the deconvolution process, and can obtain better reconstruction effects under undersampling conditions.
[0036] Specifically, considering the characteristics of SPAD and the undersampling situation, the imaging model can be expressed as:
[0037]
[0038] in represents the histogram collected under undersampling conditions, represents the volume albedo of the hidden object, represents the photon detection due to ambient light and dark counts. For ease of explanation, Remove. Matrix The convolution process for undersampled NLOS imaging is encoded.
[0039] The problem of recovering hidden targets from noisy and undersampled transient images is also a deconvolution problem with Poisson noise. In order to solve the optimization problem with a priori, we formulate the reconstruction problem as a constrained maximum likelihood estimation:
[0040]
[0041] in To control the smoothing parameters, About The total variation norm is often used to enhance edge features. For the goal The non-negativity constraint of To control the sparsity parameters, For the goal The sparsity constraint.
[0042] In the above formula It is the measured value Target observed in The latter three terms are additional priors related to the recovered transient graph and the target, which will be discussed in detail below. Without loss of generality, we adopt a separation technique to decompose the objective function into independent penalty terms while enforcing global consistency through constraints:
[0043]
[0044] Represents the actual acquired transient graph, It is the measured value Target observed in The possibility of K represents an observation matrix, z is a slack variable, which has different meanings in different update variables. , represents the transient graph obtained according to the forward model, , Represents the target, represents the objective function with Poisson constraints, represents the objective function with smoothness constraints, represents the objective function with non-negative constraints, represents the objective function with sparse term constraints, is a unit vector;
[0045] In this formula, , is a slack variable. Using the augmented Lagrangian of equation (5), ADMM can derive an iterative update scheme, the augmented Lagrangian function of the objective It is expressed as:
[0046]
[0047] in is the dual variable or Lagrange multiplier, is a positive parameter. To simplify the process, we use a scaled version of ADMM, which uses a scaled dual variable , rather than the Lagrange multiplier .
[0048] Next, we will discuss how to use prior information to solve each subminimal problem in the optimized deconvolution algorithm.
[0049] B1. About The sub-minimization problem is:
[0050] In order to make full use of the priors of transient graphs and targets, The sub-minimization problem can be defined as 4 types. First, for the Poisson noise caused by the SPAD acquisition mechanism and undersampling, the proximal operator of the Poisson term can be defined as:
[0051]
[0052] In the ADMM algorithm, z and Update in an alternating or sequential manner. Since we have 4 constraints, each time u is updated, , , , All need to be updated. is the scaled dual variable of the Poisson constraint term.
[0053] According to the Poisson noise probability distribution model, the objective function can be written as:
[0054] (10)
[0055] We set the gradient of the objective function to 0:
[0056] (11)
[0057] Thus, the operator eliminates the need to include the system matrix A, making the noisy observations independent of each other. This can be seen by observing that J is related to We can see the gradient of each element in the graph:
[0058] (12)
[0059] This expression is a classic quadratic equation root-finding problem. Each Can be solved independently. is one of the roots of the quadratic equation and is the solution to this minimization problem, express The transient graph corresponding to the jth element in , express The dark noise corresponding to the jth element in is, represents the observation matrix of the system, express The target corresponding to the jth element in express The scaled dual variable corresponding to the jth element in . Due to the non-negativity of the objective function, we only focus on the positive roots. Therefore, equation (9) can be written as:
[0060] (13)
[0061] This operator is a closed-form solution that is applied to each pixel independently and can be implemented analytically without any iterations. In each update of the ADMM, we need to set the parameters . Especially when the Poisson noise caused by undersampling scanning is large, Should be higher.
[0062] Second, in order to suppress Poisson noise, the parameter The large value of will inevitably cause edge blur. Therefore, a 3D-TV penalty term is used to improve the edge characteristics of the transient image. 3D-TV can be defined as:
[0063] (14)
[0064] Indicates about The total variation norm is used to enhance edge features. Indicates that it contains The three-dimensional cube of ;
[0065] Therefore, we use the proximal operator To find the global minimum of the objective function:
[0066]
[0067] is the scaled dual variable of the smoothness constraint, is a denotation that represents a minimization problem regarding the smoothness prior of the transient graph.
[0068] Third, in addition to directly constraining the transient image, we also hope to make full use of the prior information of the target to ensure the accuracy of deconvolution. Figure 1 As shown in (c), the optimized target can further optimize the transient image through the physical model of formula (4), so we impose certain constraints on the target index function. The proximal operator can be defined as:
[0069]
[0070] is the scaled dual variable of the non-negative constraint term, is a denotation that represents a minimization problem with respect to a non-negative prior on the target.
[0071] Finally, note that we can only reconstruct the surface of the hidden object that the photons can reach. Therefore, the directional albedo is sparse. The nearest neighbor operator with respect to the sparsity prior can be defined as:
[0072]
[0073] is the scaled dual variable of the sparse constraint term, is a denotation that represents a minimization problem with respect to the sparsity prior of the target. The Split Bregman method can solve the minimization problem with respect to the sparsity constraint of z4.
[0074] In summary, we fully utilize the priors of transient images and targets by updating the proximal operator to make up for the transient image loss caused by undersampling as much as possible. Next, we will use the optimized to optimize the deconvolution process.
[0075] B2.About The sub-minimization problem is:
[0076] The convolutional form not only provides significant computational advantages, but also makes it possible to solve the inverse problem with regularization priors. Therefore, instead of computing explicit matrix-vector products, we can efficiently compute the underscanned target using a few Fourier transforms and element-wise multiplications and divisions. Equation (4) can be expressed as a multiplication in the frequency domain:
[0077]
[0078] Where c is the three-dimensional convolution kernel, is a one-dimensional convolution along the time axis, is the Fourier transform along the time dimension and operates Vectorize a 3D data cube into a single column vector.
[0079] Updated using B1 , we can express the proximal operation of the quadratic term in equation (8) through the normal equations as:
[0080]
[0081] The difference between the second-to-last line and the first-to-last line is that the former uses vectorized discrete variables, while the latter uses the same number of non-vectorized versions. , the matrix-vector multiplication in the time domain can be expressed as multiplication in the frequency domain through convolution; is the optical transfer function in the frequency domain;
[0082] in Generally speaking, and the denominator remain unchanged in each iteration of ADMM. We update The complete transient graph is continuously updated to achieve the purpose of optimizing deconvolution.
[0083] B3.About The sub-minimization problem is:
[0084] Finally, the Lagrange multiplier is updated using the gradient ascent method as follows:
[0085]
[0086] In order to evaluate the indispensability of different components in the proposed method, the present invention conducts ablation experiments. The reconstruction results are shown in Figure 1 The results of direct deconvolution of the under-scanned transient image are shown in the first row. Sparse measurements will lead to degradation of the reconstructed image. Therefore, it is necessary to complete the sparse transient image to make the deconvolution more accurate.
[0087] Considering the priors of the transient graph and the target, we designed different constraints when completing the transient graph. Figure 1 The second to third rows in Figure (c) are the results without Poisson constraint, edge constraint, non-negativity constraint and sparsity constraint respectively. The lack of Poisson constraint leads to poor signal-to-noise ratio of the reconstructed target, because the Poisson constraint term is crucial for undersampled imaging. The transient image is a three-dimensional cube that changes with time. Undersampling will cause the diffusion shape of the transient image to be incomplete. Therefore, we use the TV term to enhance the edge of the transient image. The result without edge constraint is blurred. The constraint on the target can be further improved by Equation 4. Without non-negativity constraint, the result will be significantly deteriorated. This is because non-negativity constraint is effective for background separation and can greatly reduce the number of unknown elements in each iteration. The result without sparsity constraint is also poor, especially in the top view, where simple horizontal lines are difficult to recover. This also emphasizes the importance of target sparsity constraint.
[0088] These regularization parts are essential, and the proposed method can recover high-quality hidden objects in under-sampling situations.
[0089] The present invention is described in detail below through simulation experiments.
[0090] (1) Simulation data experiments at different scanning points:
[0091] We used simulation data to verify the reconstruction performance of this method at different sampling points. In the simulation process, a two-dimensional target depth image is first generated, in which different grayscale values of the target represent the depth information of different positions of the target. Then, the three-dimensional distribution matrix of the target is obtained using the depth image; next, the point spread function is generated using the LCT model, and its size is 256×256×1024. In the simulation, the bin width is set to 20ps and the wall size is set to 2×2 m 2 .
[0092] The results of the full-sampling simulation transient diagram reconstructed using the LCT method are shown in Figure 2As shown in the first row. In the experiment, 521 / 205 / 102 points are randomly selected as non-zero points to generate the encoding matrix, and then the encoding matrix is multiplied by the spatial distribution point of the transient image at each bin. Therefore, the processed transient image only contains valid information of non-zero points in the spatial position, which is equivalent to scanning only 521 / 205 / 102 points during actual data acquisition. Under this undersampling condition, the quality of the reconstructed hidden target will be greatly reduced due to the loss of information.
[0093] like Figure 2 As shown in the figure, when the sampling rate is 0.78%, the reconstruction results of the back projection (FBP) method, LCT and phasor field can no longer distinguish the original target structure. The reconstruction results of the Fk method based on frequency domain offset can reconstruct the original target structure, but the reconstruction quality and signal-to-noise ratio are significantly reduced. However, this method can effectively reconstruct the three-dimensional structure of the original hidden target, and the SSIM can reach 0.78, which is significantly higher than the Fk method. When the sampling rate continues to decrease to 0.31%, neither the FBP, LCT, phasor field method nor the Fk method can reconstruct a distinguishable hidden target structure. However, this method can still reconstruct the three-dimensional structure of the hidden target more clearly, and the structural similarity reaches 0.63. When the sampling rate is reduced to 0.15%, although there is slight noise in the reconstruction results of the method of the present invention, the shape of the four letters can still be clearly seen from the front. At this sampling rate, the sampling speed can be increased by 642 times, which can effectively improve the possibility of dynamic target reconstruction in NLOS scenarios.
[0094] (2) Public data experiments with different underscanning modes:
[0095] The proposed method is robust to various undersampling patterns. Whether it is random undersampling or uniform undersampling, the proposed method can achieve good reconstruction results. Here, we compare the reconstruction results of the proposed method with other methods under two different undersampling patterns.
[0096] We verified this on the publicly available Stanford data. 2 The wall is scanned at 64×64 locations with an exposure time of 15s. The reconstruction results of our method and other methods on the fully sampled data are shown in Figure 3 As shown in the first row. Since the exposure time is only 15 seconds, the existence of Poisson noise has deteriorated the reconstruction results of some algorithms. On this basis, we compared the reconstruction results under two different scanning modes: random and uniform. The FBP reconstruction results are shown in Figure 3As shown in the figure. Regardless of whether it is a random or uniform scanning mode, FBP cannot reconstruct the target at an exposure time of 15s. The reconstruction quality of the Fk algorithm for short exposure data is also very poor, and the shape of the target cannot be distinguished at all. The phasor field achieves high-resolution near-field imaging by constructing a virtual wave transmission model. However, the selection of the virtual wavelength λ is related to the sampling interval d, and must satisfy the condition of λ<2d. When the undersampling mode is random, the phasor field cannot determine the wavelength due to the uneven scanning interval, so the reconstruction effect in the random undersampling mode is poor. The reconstruction effect in the uniform undersampling mode is better, but it is greatly disturbed by Poisson noise. The image reconstructed by LCT can be roughly identified, but it contains a lot of noise. In particular, when the downsampling is about 6.25%, no matter what the sampling method is, the dragon reconstructed by the LCT algorithm is completely submerged in the noise. It can be observed that our method can produce satisfactory reconstruction results regardless of random or uniform scanning mode, even if the number of scanning points is reduced to 16×16. Experiments verify that this method has strong robustness to different downsampling modes and can better adapt to actual scenes.
[0097] (3) Public data experiments with different exposure times:
[0098] To further demonstrate the effectiveness of the proposed method, we evaluate it on real-world data from scenes (statues and dragons). These two scenes are 2 × 2 m in size. 2 64×64 positions were sampled on the wall of the image, and two measurements were captured at different exposure times (10 minutes and 30 minutes, respectively). The number of photons is determined by the exposure time. The longer the exposure time, the more photons are collected at each point, and the higher the signal-to-noise ratio. In order to improve the efficiency of NLOS imaging, shortening the exposure time of each scanning point is a common method. However, the Poisson noise caused by short exposure time reduces the reconstruction quality of many algorithms. From Figure 4It can be seen that, except for our optimized deconvolution method, other comparison methods cannot reconstruct satisfactory images on the data set collected with short exposure time. In these two sets of experimental data, we use the reconstruction results of the collected data with an exposure time of 180 minutes as GT, and SSIM is used here to measure the quality of the reconstruction results. When the exposure time is 30 minutes, our method is able to reconstruct NLOS objects with clear structures. The SSIM value reaches above 0.83. The image reconstructed by the FBP method is completely submerged by noise. For the scene statue, the top view and side view can slightly identify the position of the object, but the details of the object cannot be distinguished at all. Compared with the reconstruction result with an exposure time of 180 minutes, the SSIM is only 0.0985. For the scene dragon, whether it is the front view, side view or top view, the position and structural information of the object cannot be distinguished. The image reconstructed by LCT can recognize the shape of the target, but there is a lot of noise. In particular, for the scene dragon, from the top view and side view, the noise and the object are mixed, and the SSIM is only 0.49. Although the virtual phasor field can reconstruct the fine structure of the object, it is still affected by noise. The SSIM value of the reconstruction result is more than 0.3 lower than our SSIM value.
[0099] When the exposure time is 10 minutes, the reconstruction results of the above methods are seriously affected by noise. It can be seen from the image reconstructed by FBP that the target is completely submerged in the noise, and the SSIM value is only slightly greater than 0.1. The reconstruction effect of LCT is slightly better, and the shape of the target can be roughly identified, but the existence of Poisson noise cannot be ignored. For the scene dragon, the Poisson noise generated by the short exposure time makes the side view and top view of the dragon indistinguishable. The SSIM value is only about 0.34. The SSIM of the phasor field reconstruction result is only slightly greater than 0.2. Our method has the best effect. In this set of data with an exposure time of 10 minutes, the SSIM of the reconstruction result of our method remains above 0.67, which is much higher than the other three methods. The reconstruction result not only has a clear structure, but also has extremely low noise, which provides a new idea for high-quality and fast near-field imaging.
[0100] (4) Experimental verification of real system:
[0101] A. Experimental setup:
[0102] In order to verify the proposed method, this paper also proposes a compact and efficient NLOS imaging system. The confocal NLOS system is mainly composed of a laser and a detector. The experimental setup is as follows Figure 1As shown in (b), in the experimental system, the pulsed laser emits 1064nm linearly polarized light at a repetition frequency of 30MHz. The average power of the emitted laser beam is 300mW. The laser beam first passes through a beam splitter (Thorlabs BSW29) and then enters the scene controlled by a 2-axis scanning galvanometer (Thorlabs GVS012). The galvanometer is controlled by a data acquisition device (NI-DAQ USB-6343) to modify the scanning path on the visual wall in the LabVIEW environment. The visible wall is about 0.5 meters away from the system. The returned light returns to the beam splitter along the same path and is focused onto the detector through a reflective collimator (Thorlabs RC12FC-P01). The detector is a single-pixel InGaAs SPAD (AUREA SPD_OEM_NIR) with a free-running mode and a gated mode. The detector used has a detection efficiency of ~ 20%, a dark count rate of ~1900cps, and a dead time of 10us. In order to prevent light from being directly reflected from the visible wall to the detector, the detector of the present invention operates in a gated mode. At the same time, the laser trigger signal is transmitted to the SPAD through a delay unit (PicoQuant MPD picosecond delay layer), and the gating time can be adjusted. In addition, a time-correlated single photon counter (TCSPC, Swabian Instruments Time Tagger Ultra) is used to record the time when the laser pulse is emitted and the time when the SPAD detects the photon. It is worth noting that the frequency of the TCSPC recording histogram needs to be consistent with the movement frequency of the galvanometer to ensure that each histogram corresponds to each scanning position. The total time jitter of our system, including the laser, SPAD, and TCSPC, is about 250 ps.
[0103] B. Verification of experimental data at different scanning points:
[0104] To verify the reliability of this method, we used LCT and this method to compare and reconstruct the data collected by the actual system. The object is made of diamond-grade reflective film. The system scans 32×32 locations on the visual wall, and the acquisition time for each point is 100ms. TCSPC records the histogram distribution of the returned photons with a time resolution of 20ps / bin. After obtaining the raw data from TCSPC, the resulting histogram is aligned in time so that the pulse directly returned from the wall occurs at time t = 0. The histogram is then trimmed to 512 bins for our experiments. The distance between the system and the visible wall is 0.5m, and the distance between the hidden object and the wall is 0.25m. In this set of experiments, we compared the reconstruction results of traditional LCT and the optimized deconvolution method at different scanning points, such as Figure 5 shown.
[0105] Randomly select 512 / 205 / 102 points to generate the encoding matrix. The transient image has signals only at non-zero positions, which is equivalent to underscan acquisition. We reconstruct different targets "L" and "T" respectively. The width of the letter "L" is 15cm and the length is 30cm. The width of the letter "T" is 5cm and the length is 30cm. In the case of full sampling, the results of reconstructing objects "L" and "T" using the LCT algorithm are shown in the following figure. Figure 5 As shown in the first row. When the number of random sampling points is only 50%, the reconstruction result of LCT degrades slightly. Especially from the top view and side view, there are artifacts in the reconstruction result. However, our method can still clearly reconstruct the structure of the target. When the number of sampling points is 205, the reconstruction quality of LCT degrades significantly and the target is submerged in noise. It can be seen from the top view and side view of object "T" that the Poisson noise caused by spatial underscanning leads to a poor signal-to-noise ratio of the reconstruction result. However, this method can reconstruct the 3D structure of the target with high quality. When the number of sampling points drops to 10% of the original data, the LCT algorithm is basically unable to distinguish the structure of the target. However, our method is still effective, especially the edges of the target can be clearly reconstructed.
[0106] C. Verification of experimental data under different exposure times:
[0107] This section will demonstrate an experiment of collecting different exposure times at each scanning point. In the experiment, the system scans 64×64 positions on the visual wall, and the wall size is set to 0.8×0.8m 2 . The time resolution of the recorded histograms is 10ps / bin, and then these histograms are pruned to 1024 bins for our experiments. The distance between the system and the visible wall is 1m, and the distance between the hidden object U and the visible wall is 0.8m. The width of U is 6cm and the length is 40cm. In the experiment, we set the exposure time, that is, once the waiting time is greater than the exposure time, the system will scan to the next point. The exposure time of each scanning point is set to 5ms (a total of 20.48 s), 10ms (a total of 40.96 s), 50ms (a total of 3.41 min), and 100 ms (a total of 6.82 min).
[0108] The results are as follows Figure 6 As shown. It can be seen that as the exposure time decreases, the reconstruction quality of the target becomes worse. Especially when the exposure time is 5ms, the LCT reconstructed image can be roughly identified but contains a lot of noise. In the scene, our method maintains a clear advantage and can robustly reconstruct structural details. In general, the proposed optimized deconvolution method has more advantages than direct deconvolution under underscanning conditions, which makes it possible for fast imaging NLOS.
[0109] In summary, this paper focuses on fast and high-quality NLOS imaging by optimizing deconvolution. Combining transient images and prior features of the target, a computationally efficient optimized deconvolution method is used to reconstruct hidden targets from underscanned measurements. Compared with traditional methods, our pipeline provides high-quality reconstruction and only requires 0.15% of the original measurement time. Simulation and experimental results show that the proposed method outperforms existing state-of-the-art methods under underscan conditions. In addition, the proposed method is robust to different underscan patterns and can lay the foundation for flexible underscanned NLOS imaging.
[0110] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
[0111] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. A method for undersampling non-viewing-area imaging based on deconvolution optimization, characterized in that: include: Step 1: The pulse laser scans the relay wall, and the single photon avalanche diode SPAD collects the signal at each point to obtain an under-sampled photon histogram; The single photon avalanche diode SPAD collects a total of The transient graph collected by the SPAD is modeled as: ; In the formula, is the full-sample transient measurement value, For the 3D reconstruction domain The volume albedo of the hidden scene in is the light transfer function, is the speed of light, is the 3D scene coordinate, , They are the lighting point and detection point on the visible wall respectively; when When the integration domain is the focus and When the illumination point and the detection point are the same, that is, When , the NLOS confocal model is obtained as follows: (2); This formula simplifies to: ; In the formula, is the vector representation of the light transient information, where is the number of lighting points, is the number of detection points, is the number of time bins; The convolution process of confocal NLOS imaging is encoded; is the vectorized representation of the NLOS volume albedo, where , , are the number of voxels, respectively; When the number of scanning points or the exposure time decreases, the convolution model of formula (3) degenerates into: ; in Represents the histogram of under-sampling measurement, matrix The convolution process for undersampled NLOS imaging is encoded; Step 2: Based on the prior information of the transient graph and the target, a transient graph-target joint prior optimization model is constructed. The model takes into account the Poisson signal distribution characteristics, the smoothness of the transient graph, the non-negativity of the non-viewing area target, and the sparsity prior constraints of the non-viewing area target. Taking into account the characteristics of SPAD and the undersampling situation, the imaging model is expressed as: ; in represents the histogram collected under undersampling conditions, represents the volume albedo of the hidden object, represents the photon detection due to ambient light and dark counts; Remove; Matrix The convolution process for undersampled NLOS imaging is encoded; The problem of recovering hidden targets from noisy and undersampled transient images is a deconvolution problem with Poisson noise; in order to solve the optimization problem with a priori, the reconstruction problem is formulated as a constrained maximum likelihood estimation: ; in It is the measured value Target observed in The possibility of To control the smoothing parameters, About The total variation norm of For the goal The non-negativity constraint of To control the sparsity parameters, For the goal Sparsity constraints; Step 3, use the generalized alternating projection method to solve in the Fourier domain, optimize the model and solve it to obtain the optimization result of the non-viewing area target representation, specifically: Using separation techniques, the objective function is decomposed into independent penalty terms, while global consistency is enforced through constraints: ; In the formula It is the measured value Target observed in The possibility of, K represents an observation matrix, z is the slack variable, , represents the transient graph obtained according to the forward model, , Represents the target, represents the objective function with Poisson constraints, represents the objective function with smoothness constraints, represents the objective function with non-negative constraints, represents the objective function with sparse term constraints, is a unit vector; Augmented Lagrangian function of the objective It is expressed as: ; in is the dual variable or Lagrange multiplier, is a positive parameter; Using scaled dual variables , the specific update process is as follows: (a) About The sub-minimization problem about The sub-minimization problem is defined as 4 categories; For Poisson noise caused by SPAD acquisition mechanism and undersampling, the proximal operator of the Poisson term can be defined as: ; is the scaled dual variable of the Poisson constraint term; According to the Poisson noise probability distribution model, the objective function Can be written as: ; Let the gradient of the objective function be equal to 0: (11) This operator eliminates the need to include the system matrix A, making the noisy observations independent of each other; observations J are related to The gradient of each element in: ; is one of the roots of the quadratic equation and is the solution to this minimization problem, express The transient graph corresponding to the jth element in , express The dark noise corresponding to the jth element in is, represents the observation matrix of the system, express The target corresponding to the jth element in express The scaled dual variable corresponding to the j-th element in ; Equation (9) can be written as: ; The edge features of the transient graph are improved by the 3D-TV penalty term, which is defined as: ; about The total variation norm is used to enhance edge features. Indicates that it contains The three-dimensional cube of ; Therefore, using the proximal operator To find the global minimum of the objective function: ; is the scaled dual variable of the smoothness constraint, is a reference that represents a minimization problem regarding the smoothness prior of the transient graph; By constraining the target indicator function, the proximal operator can be defined as: ; is the scaled dual variable of the non-negative constraint term, is a reference that represents a minimization problem with respect to the non-negative prior of the target; The nearest neighbor operator with respect to sparsity prior can be defined as: ; is the scaled dual variable of the sparse constraint term, is a reference, indicating a minimization problem with respect to the target sparsity prior; By updating the proximal operator, the transient image loss caused by undersampling is completed, and the optimized To optimize the deconvolution process; (b) With regard to The sub-minimization problem Formula (4) can be expressed as multiplication in the frequency domain: ; Where c is the three-dimensional convolution kernel, is a one-dimensional convolution along the time axis, is the Fourier transform along the time dimension and operates Vectorize a three-dimensional data cube into a single column vector; Use the updated , equation (8) can be expressed as: ; in ; is the optical transfer function in the frequency domain; (c) About The sub-minimization problem The Lagrange multiplier is updated using the gradient ascent method as follows: (20)。 2. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to claim 1 are implemented.
3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to claim 1 are implemented.
4. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claim 1 are implemented.
Citation Information
Patent Citations
Single-pixel imaging method based on plug-and-play prior
CN113724146A
Terahertz single-pixel imaging method and system
CN114894740A
Alternating iteration decoupling deconvolution image processing method based on Poisson process
CN118096568A
Cross-domain network based magnetic resonance imaging undersampling pattern optimization and reconstruction method
US20230324486A1
Terahertz single-pixel imaging method and system
WO2023193461A1
