Few-photon non-vision field imaging method based on adaptive windowing

Abstract

The invention discloses a few-photon non-vision field imaging method based on adaptive windowing, and the method comprises the steps: collecting photon signals reflected from a hidden target through a non-coaxial two-dimensional scanning system, building a probability model of signal photons and noise photons, and analyzing the distribution characteristics of the signal photons and the noise photons in a time domain; the method comprises the following steps: combining adjacent pixels into pixel blocks by using space-time correlation, determining the adaptive window width of each pixel block through a matched filtering method, and separating signal photons from noise photons by applying windowing operation on a time domain; and filling the windowed transient data by using full-variable regularization constraint, and solving a reconstruction problem by using an alternating direction multiplier method to obtain a target reconstruction result. According to the method, the detection efficiency of sparse photons is remarkably improved through the space-time related pixel blocks and the matched filtering technology, and target reconstruction under the extremely low photon level is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of non-line-of-sight imaging, and in particular to a few-photon non-line-of-sight imaging method based on adaptive windowing. BACKGROUND

[0002] Non-line-of-sight (NLOS) imaging technology can reconstruct the target outside the line of sight by analyzing the information of multiple scattered photons, and has important application value in the fields of automatic driving, medical detection, military reconnaissance, disaster relief and remote sensing. Recently, many methods have been proposed to reconstruct the NLOS target, such as: speckle correlation, thermal imaging, acoustic imaging, Fermat path imaging, occlusion-based imaging technology and transient imaging.

[0003] Among these methods, transient-based NLOS imaging is widely utilized due to its beneficial three-dimensional reconstruction capability. By utilizing the Time-of-Light (ToF) information of multiple-scattered photons, transient NLOS imaging can reconstruct the hidden target. However, unlike traditional in-los imaging, NLOS scenes suffer from severe photon attenuation due to multiple scattering, which significantly increases the complexity of signal extraction. In 2012, Velten et al. first realized NLOS imaging using a streak camera (A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging,” Nat. communications 3, 745 (2012).). Although this work achieved groundbreaking success, the low detection efficiency of the streak camera limited the imaging quality. Subsequently, Single-Photon Avalanche Diodes (SPAD) were widely used due to their high internal gain and single-photon sensitivity. For example, O’Toole et al. proposed the Light Cone Transform (LCT) algorithm, which uses a confocal configuration and utilizes SPAD for detection (M. O’Toole, D. B. Lindell, and G. Wetzstein, “Confocal non-line-of-sight imaging based on the light-cone transform,” Nature 555, 338-341 (2018).), while Lindell adopted the Frequency-wavenumber (FK) migration algorithm under the same system configuration (D. B. Lindell, G. Wetzstein, and M. O’Toole, “Wave-based non-line-of-sight imaging using fast fk migration,” ACM Transactions on Graph. (ToG) 38, 1-13 (2019).). However, due to the need for point-by-point scanning during the imaging process, single-point SPAD detection leads to a significant prolongation of the acquisition time, severely affecting the imaging efficiency.Recently, a superconducting nanowire single-photon detector (SNSPD) has been used in NLOS scenarios, which is superior to traditional detectors in detection efficiency and can perform NLOS imaging in the near-infrared and mid-infrared wavebands. However, its high cost and large cooling system limit its integration into a compact platform. There are also methods that attempt to achieve parallel scanning by encoding a single SPAD with a SPAD array or a digital micromirror device (DMD). However, in these non-confocal systems, the geometric mismatch between the illumination point and the detection point leads to a sharp decrease in photon counting at the edge of the relay wall. To solve this problem, higher laser power is often required. The above methods all rely on extending the exposure time of each scanning point, using efficient detectors, or excessively increasing the laser power to compensate for the lack of photons in the NLOS scenario, resulting in excessive photon detection and inefficient target reconstruction. Therefore, achieving high-precision NLOS reconstruction under conditions of a lack of photons is still a key and urgent problem.

[0004] Many computational imaging methods have been proposed to improve the photon utilization efficiency in NLOS imaging. For example, Feng et al. proposed a light field tomography (LIFT) technique that achieved NLOS imaging using a pulsed laser with an average power of 65 mW (X. Feng and L. Gao, “Ultrafast lightfield tomography for snapshot transient and non-line-of-sight imaging,” Nat. communications 12, 2179 (2021).). Similarly, Xu et al. demonstrated an innovative approach in their work (F. Xu, G. Shulkind, C. Thrampoulidis, J. H. Shapiro, A. Torralba, F. N. Wong, and G. W. Wornell, “Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging,” Opt. express 26, 9945-9962 (2018).), which utilized occluded scenes to achieve imaging with a photons per pixel (PPP) of only about 69, comparable to imaging with a PPP of about 1100. Despite these methods, the scarcity of photons remains a significant challenge, which directly leads to poor imaging quality. First-photon imaging techniques have also been explored and used in NLOS scenes. Tsai et al. systematically analyzed the geometric constraints of the first-returning photons and proposed a path back-projection method for reconstructing hidden targets (C.-Y. Tsai, K. N. Kutulakos, S. G. Narasimhan, and A. C. Sankaranarayanan, “The geometry of first-returning photons for non-line-of-sight imaging,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2017), pp. 7216-7224.).Building on this, Li et al. performed NLOS reconstruction by marking the first photon data within each time bin (Z. Li, X. Liu, J. Wang, Z. Shi, L. Qiu, and X.Fu, “Fast non-line-of-sight imaging based on first photon event stamping,” Opt. letters 47, 1928-1931 (2022).). Liu et al. further improved this approach by introducing a distance decay constraint, reducing the photon count to just one photon per pixel (J. Liu, Y. Zhou, X. Huang, Z.-P. Li, and F.Xu, “Photon-efficient non-line-of-sight imaging,” IEEE Transactions on Comput. Imaging 8, 639-650 (2022).). However, in NLOS scenarios involving multiple scattering, it is extremely challenging for the detector to distinguish whether the received first photon originates from the target or background noise. This difficulty limits these first-photon methods to reconstructing single-depth planar objects. Summary of the Invention

[0005] In order to overcome the deficiencies of the prior art, the present invention provides a few-photon non-line-of-sight imaging method based on adaptive windowing.

[0006] The technical solution for achieving the purpose of the present invention is: a few-photon non-line-of-sight imaging method based on adaptive windowing, comprising:

[0007] Step 1: Use a non-coaxial two-dimensional scanning system to collect photon signals reflected from the hidden target, establish a probability model of signal photons and noise photons, and analyze their distribution characteristics in the time domain;

[0008] Step 2: Using spatiotemporal correlation, adjacent pixels are grouped into pixel blocks. The adaptive window width of each pixel block is determined by the matched filtering method, and a windowing operation is applied in the time domain to separate the signal photons from the noise photons.

[0009] In step 3, the full variable regularization constraint is used to fill the transient data after windowing, and the alternating direction multiplier method is used to solve the reconstruction problem to obtain the target reconstruction result.

[0010] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0011] A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the above method.

[0012] A computer program product comprising a computer program which, when executed by a processor, implements the steps of the above method.

[0013] Compared with the prior art, the present application has the following advantages:

[0014] (1) By using the spatio-temporal correlation pixel block and the matched filter technique, the detection efficiency of the sparse photons is significantly improved, and the target reconstruction at a very low photon level is realized; reliable target reconstruction can be realized under the condition of 0.02 signal photons per pixel (PPP).

[0015] (2) The method can complete target reconstruction in a very short exposure time; in the actual system, complete target reconstruction is realized in a very short exposure time of 0.82 seconds, which greatly improves the imaging efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 Fig. 1 is an AW-NLOS schematic diagram, wherein (a) is an experimental system schematic diagram of AW-NLOS imaging, and the time jitter is 300 ps; (b) is an optical module of an actual system; and (c) is a synchronization circuit module.

[0017] Figure 2 Fig. 2 is a photon counting process of the twice echo signals and noise.

[0018] Figure 3 Fig. 3 is a comparison of the transient histograms of a single pixel and a pixel block.

[0019] Figure 4 Fig. 4 is an algorithm flow of AW-NLOS, wherein (a) is to obtain three-dimensional photon transient data, (b) is to form a pixel block by using spatial correlation, (c) is to determine an adaptive window width by using a matched filter, (d) is to perform windowing in the time domain, (e) is to perform transient completion by using TV regularization constraint, and (f) is to reconstruct a target.

[0020] Figure 5 Fig. 5 is a comparison of the transient information and the reconstruction results of different windowing methods, wherein (a) is no windowing, (b) is fixed window width windowing (the window width is twice the system time jitter), (c) is global windowing (the window width is the signal pulse width obtained by superimposing all pixel photon information), and (d) is adaptive windowing.

[0021] Figure 6 Fig. 6 is a comparison of the transient information and the reconstruction results of the detectors with different detection efficiencies by using the AW-NLOS and FK methods, wherein (a) is a comparison of the transient information, and (b) is a comparison of the reconstruction results.

[0022] Figure 7 Comparison of AW-NLOS and FK for transient histogram and reconstruction results under different laser power.

[0023] Figure 8 Influence of time resolution on signal detection, where (a) is the accumulated histogram of two pulses with time resolution of 15 ps, and (b) is the accumulated histogram of two pulses with time resolution of 40 ps.

[0024] Figure 9 Single-point photon histogram and reconstruction results of AW-NLOS and FK, LCT, PF and FBP under different time bin width, where the time bin width is 50 ps, 20 ps, 10 ps and 5 ps, respectively.

[0025] Figure 10 Comparison of transient information and reconstruction results of AW-NLOS and FK, LCT, PF and FBP under different exposure time conditions, where (a) is the actual scene and GT; (b) is the transient information under different exposure time conditions; SBR: signal-to-background ratio in the dashed box; (c) is the reconstruction results and corresponding SSIM under different exposure time conditions.

[0026] Figure 11 Comparison of reconstruction results of AW-NLOS and FK, LCT, PF and FBP under different exposure time conditions, where (a) is the actual scene and GT; the exposure time of each pixel is (b) 6.87 ms and (c) 2.29 ms, respectively.

[0027] Figure 12 Comparison of single-point histogram and reconstruction results of AW-NLOS and FK, LCT, PF and FBP under different distances, where (a) is the actual scene; (b) is the decrease of signal photon number with increasing distance; (c) is the reconstruction results and corresponding SSIM at 2 m, 3 m and 4 m. DETAILED DESCRIPTION

[0028] Non-line-of-sight imaging aims to reconstruct the shape of hidden targets. However, most existing NLOS systems require hundreds or even thousands of photon detections per pixel to accurately reconstruct hidden targets, which requires very long exposure times and complex hardware configurations. This paper proposes adaptive windowing-based non-line-of-sight imaging (AW-NLOS) to achieve reconstruction with high photon utilization in the sub-photon state. Specifically, by utilizing spatiotemporally correlated pixel blocks, the detection efficiency of sparse photons is significantly improved; then, the enhanced signal is used to achieve accurate window width estimation through matched filtering; for each pixel of the transient image, an adaptive short-time interval window can be applied to eliminate detection caused by noise; and the transient data is also subjected to total variation (TV) regularization to reconstruct measurement results with a high signal-to-background ratio. Experiments show that this method can achieve reliable target reconstruction at the sub-photon level of 0.02 signal photons per pixel (PPP).

[0029] The present invention designs a non-coaxial two-dimensional scanning system to achieve photon-efficient NLOS imaging. Figure 1 (a) shows a schematic diagram of the experimental system, where the distance between the non-coaxial 2D scanning system and the relay wall is 7 meters, and the hidden target is located 3 meters from the relay wall. Unlike traditional coaxial transceiver systems, this invention uses only two galvanometer mirrors (transmitter / receiver) to perform raster scanning of the relay wall. In this setup, a slight spatial separation is designed between the illumination point and the receiving field of view to minimize the impact of primary reflected light. The spatial separation is controlled to approximately 2 cm, which is much smaller than the distance between the hidden target and the relay wall.

[0030] The optical modules of the actual system are as follows Figure 1 As shown in (b) of the figure. On the transmitting end, a 1064nm collimated laser with adjustable power and a repetition rate of 5MHz illuminates the transmitting galvanometer, which performs raster scanning on the relay wall. On the receiving end, a SPAD detector (AUREA, SPD_OEM_NIR) with ultrafast gating is used. This detector is coupled to a multimode fiber and a collimating lens to collect the echo signal from the secondary reflection. A narrowband filter (Thorlabs FLH 1064-3) is also integrated into the transmitting galvanometer to reduce background noise interference on the SPAD.

[0031] In order to achieve accurate ToF measurement, a Figure 1The synchronization circuit module shown in (c) can accurately timestamp the laser emission and photon detection time. At the same time of each photon emission, the laser emits two identical synchronization signals: one is sent directly to the Time-Correlated Single Photon Counting (TCSPC) system (Time Tagger Ultra, Swabian Instruments) as a start signal, and the other is sent to the gating port of the SPAD through a delay-compensated Field Programmable Gate Array (FPGA). When the photon reflected by the target is detected by the SPAD, the TCSPC system generates and records the corresponding stop signal, thereby realizing the calculation of the ToF value. In addition, to solve the synchronization and storage problem of the pixel-by-pixel scanning signal, a National Instruments data acquisition device (NI-DAQ, 134 USB-6343) is used to control the positions of the two galvanometer scanners respectively. After completing the scanning of each point, it sends a restart signal to the TCSPC to ensure that the information and storage of each point are completely matched. The time jitter of this system includes laser pulse width, SPAD jitter, TCSPC jitter, and circuit delay, etc., and the measurement result is 300ps, as shown in (a) of FIG. Figure 1 .

[0032] In the NLOS imaging system, the received signal includes signal photons and noise photons affected by various factors. Due to the propagation characteristics of light, the signal photons under short exposure time show non-uniform Gaussian distribution in the time domain. In contrast, the noise photons mainly composed of background noise and dark noise are uniformly distributed in the time domain and belong to a uniform Poisson process. The basic difference in the time distribution patterns of signal photons and noise photons constitutes the basis of the adaptive windowing technology of the present application, which enables effective separation of the two to enhance the imaging quality.

[0033] In the NLOS imaging system proposed by the present application, the signal received by the detector is affected by various factors. Under the condition that the working wavelength is , the number of signal photons detected by the detector per second is expressed as:

[0034] (1)

[0035] wherein, is the average optical power of the pulsed laser, is the Blakemore constant, c is the speed of light, , are the reflectivities of the target and the relay wall respectively, is the angle between the normal of the relay wall and the laser beam, is the area of the hidden target, is the range of the field of view (FOV) on the relay wall, is the area of the receiving field of view of the detector, is the transmittance of the receiving system, is the transmittance of the air, is the distance between the optical system and the relay wall, is the distance between the target and the relay wall.

[0036] In addition to the signal photons, the NLOS imaging system also receives noise photons composed of background noise and dark noise, the number of noise photons received per second can be expressed as:

[0037] (2)

[0038] where, is the bandwidth of the ideal band-pass filter, is the solar spectral irradiance, is the angle of incidence of the sun, is the dark count rate (DCR) of the detector.

[0039] When the system detection efficiency (SDE) of the detector is , the photon flux reaching the detector can be expressed as:

[0040] (3)

[0041] Considering the acquisition mechanism of TCSPC, the arrival time of the photons collected by the SPAD can only be recorded discretely in different time bins. The average number of photons detected in the nth time bin with a width of can be expressed as:

[0042] (4)

[0043] where, is the time bin index, is the maximum index of the time bin.

[0044] Affected by the detection mechanism, the photon signal detected by the SPAD obeys the Poisson distribution. The probability of observing k photons in the nth time bin can be expressed as:

[0045] (5)

[0046] In addition, considering the dead time of SPAD , the probability of detecting a signal photon within the th time bin can be expressed as:

[0047] (6)

[0048] The detection efficiency of a noise photon can be expressed as:

[0049] (7)

[0050] For a single scan point, the detection of signal and noise is not significantly different, so it is difficult to distinguish them. To separate the signal and noise, prior knowledge of the different probability models of the signal and noise during detection can be used. Specifically, as shown in equation (1), the signal process is related to the illumination pulse laser. When multiple pulses are irradiated on the same scan point, the detection will be clustered together near the true depth. The noise signal mainly includes background noise and dark noise, which belong to the homogeneous Poisson process and do not exhibit clustering effect in the time domain. The corresponding process is shown in Figure 2 , where the non-uniform Poisson process is represented by a yellow line, which is the sum of a non-uniform signal process and a uniform background noise process. Among them, the signal photons are represented by blue arrows, and the noise photons are represented by red arrows.

[0051] Since the noise photons are uniformly distributed in the time domain, the spatiotemporal correlation can be used to separate the noise photons and the signal photons. This enables the system to achieve photon-efficient NLOS imaging.

[0052] In the method of the present application, an NLOS imaging system is used to obtain a complete three-dimensional (3D) transient map, including pixels, each pixel having a plurality of time bins, as shown in (a) of Figure 4 . Since the signal detection is more likely to cluster compared to the noise detection, a direct method is to identify the clustered peaks. One way to define the detection cluster is to select the window duration , and a well-placed window should be large enough to capture most or all signal photons without accepting a large number of noise photons. Unlike single-photon imaging systems, each scan point in the NLOS imaging system detects the superposition of all point echoes on the self-concealing target, as shown in (a) of Figure 1 . The window width of the signal cluster detected by different angles ​​and time are different. Therefore, a fixed window does not apply to all the scanning points, and it is of great significance to explore an adaptive window width for the acquisition of signal photon events in NLOS imaging.

[0053] In the NLOS imaging system, the photon flux acquired from each scanning point is very weak and difficult to identify the position of the cluster, because the photon experiences three reflections of "relay wall-target-relay wall". As shown in Figure 3 , the single-pixel transient exhibits sparsity, and it is difficult to determine the start and end positions of the window. In order to improve the signal detection probability, it is necessary to form a pixel block in cooperation with the surrounding pixels. In the method of the present application, by utilizing the spatio-temporal correlation of the surrounding scanning points, the nearest 4x4 pixels are grouped into a pixel block . The photon count in the pixel block can be represented as:

[0054] (8)

[0055] wherein, is the pixel block index. The detection from similar neighboring pixels can help amplify the low signal level by making the signal detection cluster more obvious, as shown in Figure 3 . Therefore, by adopting the spatial correlation, the window of each pixel block becomes more reliable and useful.

[0056] In order to further determine the size of the window of each pixel block, the present application uses a matched filtering method to preprocess the photon count data. The matched filtering method is usually used to analyze the similarity between two time-domain signals, so as to obtain the fine statistical distribution of the photon arrival time. In the acquired transient data, the background noise detection is uniformly distributed in time, while the signal intensity follows a Gaussian distribution and exhibits different clustering characteristics in the time domain. Therefore, the purpose of matched filtering is to eliminate the background noise and effectively acquire the duration of the window of each pixel block from a small number of echo photons. The instrument response function (IRF) of the system is used as the template for matched filtering. The size of the IRF is affected by multiple devices of the system, such as the laser pulse width, the jitter of the detector, and the jitter of the TCSPC. This IRF is obtained by pre-measurement, as shown in Figure 1 (a). The matched filtering result of each pixel block is shown in Figure 4 (c), which can be represented as:

[0057] (9)

[0058] The larger the pixel block The probability of a photon counting event in the pixel coming from a hidden target is higher. Subsequently, the window width size of the pixel block can be calculated using the extremum point of the IRF and the half-bandwidth of the system :

[0059] (10)

[0060] When the echo photon number of the target is sparse, it is difficult to determine the start and end time of the window only by using the photon counting histogram. However, the spatio-temporal correlation method based on IRF matching can use the clustering effect formed by a small number of photons to obtain a more fine statistical curve of the photon arrival time, thereby further ensuring the selection of the adaptive window width.

[0061] Based on the previously calculated adaptive window width, the windowing can be applied to the original transient information in the time domain. As described in Section 2.2, the signal will be more easily clustered together than the noise. Therefore, in the case of low photon counting, the windowing can better separate the signal photons from the noise, thereby leading to the improvement of the reconstruction result. As shown in Fig. 2.2 (d), the windowing process can be represented as: Figure 4

[0062] (11)

[0063] wherein t0is the start time of the window and .

[0064] The window width selected based on the signal characteristics can separate the signal from the noise while preserving the integrity of the target signal as much as possible. This not only reduces the impact of noise on the subsequent target reconstruction, but also reduces the memory requirement and computational complexity. However, since the adaptive windowing operation forcibly sets the photons outside the window to zero, it destroys the spatio-temporal correlation of the transient data. To solve this problem, a TV regularization constraint is introduced to fill in the transient data after windowing, as shown in Fig. 2.3 (e). The optimized transient information Figure 4 can be represented as:

[0065] (12)

[0066] wherein is a regularity parameter associated with the measured Signal-to-Noise Ratio (SNR) level.

[0067] After obtaining the optimized transient information, the hidden target is reconstructed. In the confocal NLOS scenario, the axial position of the relay wall is set to 0. The transient information captured at the scanning point can be represented as:

[0068] ​​​​ (13)

[0069] Assume a diffuse reflection scenario, where is a point on the hidden object is the distance to the laser or the detector. represents the attenuation of the photons scattered by the relay wall after two reflections, is the time of flight of the photons. The Dirac function represents the surface of a four-dimensional hypercone in spacetime, which can be expressed as This function models the light propagation process from the relay wall to the hidden object and back to the relay wall. Perform a variable substitution on equation (13) as follows:

[0070] (14)

[0071] This equation is essentially a 3D convolution process: where represents the 3D convolution kernel, and represent the resampling operations along the z-axis and t-axis, respectively. Solving the problem of from the transient information can be formulated as a regularized least squares problem. Let be the 3D convolution operation associated with the kernel This problem can be represented in matrix form as:

[0072] (15)

[0073] where is a regularization parameter that serves as a weighting constant to balance the trade-off between the data fidelity term and the regularization term. Then, the Alternating Direction Method of Multipliers (ADMM) is employed to solve this minimization problem. The final reconstruction result is shown in Figure 4 (f).

[0074] NLOS imaging can reconstruct the hidden object using the coefficient photons that have undergone multiple scattering. However, during the detection process, the detector cannot distinguish between the received photons originating from the target or the background noise during the acquisition process. In this experiment, a pulsed laser with an average power of 141 mW was used to perform a 64x64 grid scan of the relay wall surface with an exposure time of 1 ms per point. To facilitate visualization, Figure 5 (a) in Fig. 1 shows the captured transient information along a one-dimensional expansion. As Figure 5 ​As shown in (a) of FIG. 1, although the target signal is distinguishable in the raw transient data, each scan point accumulates background noise composed of ambient light and dark noise. Therefore, the reconstructed 3D voxel space is full of noise. Windowing methods have been widely applied to LOS single-photon imaging to separate signal and noise. Most methods usually adopt a fixed window width twice the system time jitter. However, when applying the fixed windowing method to the NLOS scenario with 300 ps system time jitter, it can be observed that, although the noise is effectively suppressed, it brings significant signal loss, resulting in an incomplete reconstructed target structure, as shown in (b) of FIG. 1. This limitation stems from the fundamental difference between the LOS and NLOS imaging configurations. In LOS imaging, the laser scans the object point by point. In non-line-of-sight imaging, however, the photons undergo three diffuse reflections, meaning that each scan point collects information from multiple voxels in the hidden scene. Therefore, the window width that effectively captures the full signal in LOS imaging cannot be directly applied to the non-line-of-sight scenario. To preserve more signal information, researchers have proposed an alternative method that aggregates the ToF information over all pixels. Due to the clustering effect, the accumulated signal forms a clear peak, from which a 4050 ps optimal window width can be determined. Applying this full-pixel integration method to the NLOS data, the reconstructed result is shown in (c) of FIG. 1. Although this method successfully reconstructs the complete target structure, the boundaries of the target signal in the transient image are not sufficiently refined, and noise still exists around it. This can be clearly seen in the reconstructed 3D voxels. Figure 5 Figure 5

[0075] To maintain the structural integrity of the target in NLOS reconstruction while maximizing the suppression of noise, an adaptive windowing strategy is proposed, which dynamically adjusts the window width for each scan point. Compared with fixed windowing, the method of the present application preserves the complete target geometry in the front view. Compared with global windowing, it effectively eliminates transient artifacts and noise in 3D voxels. This data-end signal-noise separation prevents reconstruction artifacts caused by noise, thereby achieving high-SNR target reconstruction under sparse signal conditions.

[0076] Not all photons incident on the SPAD surface can trigger a detectable avalanche event. The probability of a photon triggering a detectable avalanche is called the photon detection efficiency (PDE), which is a key parameter characterizing the performance of SPADs. Under the same conditions, SPADs with lower PDE capture fewer photons. The resulting few-photon data significantly increases the complexity of subsequent reconstruction. In addition to this problem, SPADs cannot distinguish between detected photons originating from the target or background noise. When using low-PDE detectors, these limitations severely degrade the performance of existing NLOS reconstruction methods, exacerbating the reconstruction challenge. As shown in (a) of FIG. 2, the raw transient data of the target is composed of a few photons.​​Figure 6 The left plot shows the transient information of SPADs in the AUREA series with 20% PDE, while the right plot shows the transient information of SPADs in the MPD series with less than 1% PDE. Under the experimental conditions of 1 ms exposure time per scan point and 287 mW power of the laser, the AUREA detector obtains 0.7246 of the photon information PPP, while the MPD can only reach 0.0217. For a transient data with X x Y scan points, the PPP calculation formula is as follows:

[0077] (16)

[0078] In this experiment, the ability of traditional methods and the method proposed in the present application to process data collected by low-PDE detectors was evaluated. Under high-power (2.59 W) illumination, the reconstruction results of the AUREA detector with high PDE were used as the ground truth (GT). As shown in (b) of FIG. 13, when the laser power is reduced to 141 mW, the reconstruction quality of both types of detectors decreases due to the decrease in photon counts. In particular, the reconstruction results of the MPD detector are almost completely submerged in noise and are difficult to distinguish. Here, the SSIM of the front view between the reconstruction results and the GT was compared. The calculation method of the SSIM is as follows: Figure 6

[0079] (17)

[0080] wherein, , , correspond to the mean, standard deviation, and covariance between x and y. and are constants used to maintain stability. And , . And is the dynamic range of pixel values.

[0081] ​When using MPD detectors with a low photon counting PPP of 0.02, the conventional method cannot distinguish the target structure with an SSIM value of about 0.5. However, the proposed method effectively separates the target structure from the background noise by amplifying the weak signal clusters using the inherent spatiotemporal correlation within the pixel block and applying an adaptive window width for each block temporal characteristic. Under the same laser power, the SSIM of the reconstruction result reaches 0.7, even exceeding that obtained using a higher PDE detector. This significant improvement indicates that the combination strategy of spatially correlated signal amplification and temporally adaptive windowing for precise noise suppression can perform robust reconstruction in the case of extremely sparse photons. This experiment proves that the proposed method can adapt to low-photon conditions caused by extremely low detector efficiency, further reducing the dependence of NLOS imaging technology on high-cost devices.

[0082] In the field of NLOS imaging, the selection of laser power has always been a key issue. On the one hand, higher laser power can significantly increase the number of signal photons, thereby improving imaging quality. However, this approach usually comes at the cost of increased power consumption and reduced safety. On the other hand, while lower laser power can reduce power consumption and increase safety, it may result in insufficient signal photons, thereby affecting the imaging result. Therefore, how to achieve high-quality NLOS imaging under low laser power has become the current research focus. Here, the reconstruction ability of the proposed method and the conventional method for objects of different complexities is compared. In the experiment, different laser powers are selected to scan a 2m×2m range on the relay wall, with an exposure time of 1ms per point. The hidden target is 3m away from the relay wall, as shown in Figure 7 The reconstruction results of the proposed method and the conventional method are shown in FIG. 6. As shown in FIG. 6, the proposed method can successfully reconstruct the target structure under the condition of 63mW laser power, while the conventional method fails to do so. This experiment proves that the proposed method can adapt to low-photon conditions caused by extremely low detector efficiency, further reducing the dependence of NLOS imaging technology on high-cost devices.

[0083] Under the condition of 63mW laser power, Figure 7The single-point histogram in the figure shows only 2-3 signal photons per detection, with a signal PPP as low as 0.8. At such low photon counts, conventional FK methods are barely able to reconstruct the target's outline. As the number of targets increases, the photon distribution per object becomes more sparse, posing a greater challenge to the reconstruction algorithm. For three objects, conventional methods fail to reconstruct the "S" structure, while the proposed method can clearly identify it. In more complex scenes, such as four objects, conventional reconstruction produces indistinguishable results with an SSIM of approximately 0.21, while the proposed method effectively separates the target from the noise. When the laser power is reduced to 42 mW, the conventional method struggles to distinguish letter shapes in scenes with multiple targets, with SSIM values ​​below 0.19. In contrast, the proposed method can clearly distinguish even closely spaced structures such as "U" and "T", achieving reconstructions with SSIMs exceeding 0.75. The proposed method demonstrates significant advantages under low laser power conditions, clearly distinguishing target structure from noise even at extremely low photon counts and maintaining high-precision reconstruction in complex multi-target scenes.

[0084] For each scan point, the quality of the photon histogram is affected not only by the laser power but also by the bin width. Decision. Figure 8 As shown in (a), when the time resolution is 15 ps, the target echo signals from the two cycles are scattered across different time bins. This dispersion affects signal accumulation, causing them to be masked by noise and making the target indistinguishable. In contrast, at a time resolution of 40 ps, ​​the target echo signals from the two cycles converge within the same time bin. This convergence allows the signal accumulation counts to be above the noise threshold, thereby enhancing target discernibility. However, while improving signal discernibility, it introduces larger ranging errors and reduces resolution.

[0085] To verify the adaptability of AW-NLOS under low-bin conditions where photon clustering is less likely to occur, the letter “U” was imaged using different temporal bin widths at 500 mW laser power and a scan time of 1 ms per point. Figure 9 The histogram and reconstruction results of a representative point under different time bin conditions are shown. As the bin width decreases, the number of signal photons decreases, making the accumulation of signal photons increasingly difficult. When the bin width is 50ps, FK, LCT, and PF can all completely reconstruct the target structure, but with obvious noise. However, when the bin width is reduced to 10ps, the signal photons are difficult to gather, and the target structure reconstructed by FK is almost indistinguishable from the noise. In contrast, the method of the present invention effectively separates the signal and noise at 50ps and 10ps resolutions, achieving high-quality reconstruction. It is worth noting that, as Figure 9In (a) of FIG. 6, under the condition of 5 ps bin width, the histogram also does not show clear signal clustering, with the maximum value of only 2 photons, making it difficult to distinguish from the background noise of one photon. Under such conditions, other traditional methods cannot reconstruct the structure of the letter. However, AW-NLOS can maintain the integrity of the target structure and avoid noise interference, resulting in significantly better reconstruction quality than traditional methods. Experimental results show that this method can adapt to hardware devices with extremely small time bin width. This ability provides great potential for high-precision imaging.

[0086] In the NLOS imaging scenario, point-by-point scanning of the relay wall is required. In order to achieve fast imaging, reducing the single-point exposure time is a common strategy. However, shorter exposure time results in fewer signal photons collected per point, making the signal very easy to be overwhelmed by background noise and dark noise. This significantly increases the pressure on the subsequent reconstruction algorithm. Here, through implementation, the AW-NLOS method proposed by the present application can successfully reconstruct the hidden target even at extremely fast acquisition speed. In the experiment, a 64x64 grid on the relay wall was scanned using a pulsed laser with a power of 2.59 W. The exposure time per point was 0.5 ms, 0.4 ms and 0.2 ms, respectively, corresponding to a total acquisition time of 2.05 s, 1.64 s and 0.82 s.

[0087] In order to quantitatively verify the improvement of the proposed adaptive windowing method relative to traditional methods under low exposure time conditions, two key indicators are compared: the signal-to-background ratio (SBR) of transient information and the SSIM of the reconstructed result front view. SBR is defined as the ratio between the maximum number of signal photons and the number of background photons:

[0088] (18)

[0089] wherein, is the number of signal peak photons, is the number of noise photons. In order to determine GT, the PF algorithm that can provide the best reconstruction quality is used to reconstruct the transient information obtained when the exposure time per point is 5 ms. As Figure 10(b) as shown in FIG. 5. For visualization, the acquired transient is unwrapped along a one-dimensional line. With a single-point exposure time of 0.5 ms, the measured SBR is 5.1663; however, after applying the windowing method, the signal and noise are effectively separated, increasing the SBR to 10.1895. Regarding the reconstruction performance, with single-point exposure times of 0.5 ms and 0.4 ms, the conventional method is able to reconstruct the target structure completely, but the resulting image quality is poor and noisy, with SSIMs always below 0.5. In contrast, the reconstruction results of the present method show little noise, with SSIMs of 0.6638 and 0.6174, respectively. When the scan time is reduced to only 0.2 ms per point, the imaging accuracy is significantly reduced under high background noise conditions. The horizontal part of the "L" structure in the reconstruction result of the conventional method is significantly blurred, accompanied by an increased noise level, resulting in SSIMs all below 0.3. In contrast, the AW-NLOS maintains the complete and clear "L" structure, with an SSIM of 0.4302, which is superior.

[0090] In addition, the present application is also verified on the public dataset Stanford dataset. In order to reduce the computational demand, the undersampled version of the dataset is adopted, selecting a 64x64 pixel grid in a 2m x 2m scan area. Two groups of data with shorter exposure times (6.87 ms and 2.29 ms) per point are selected for analysis. The GT comes from the PF reconstruction using a single-point exposure of 41.4 ms data, which provides the best reconstruction fidelity. Under the condition of a 6.87 ms single-point exposure time, the target structure reconstructed by the conventional method is overwhelmed by noise, blurring the key morphological details. In contrast, the AW-NLOS achieves excellent feature resolution, clearly delineating the skull and chest regions of the dragon, with high structural integrity, with an SSIM of 0.7892. Under the more extreme condition of a 2.29 ms single-point exposure time, the conventional method is essentially unable to restore the geometry of the target. The present method always maintains high-quality reconstruction, which ultimately proves the superior performance of the AW-NLOS under photon-poor conditions.

[0091] The method of the present application fundamentally advances low-photon NLOS imaging, achieving accurate "signal-noise" separation under photon-poor conditions. This capability helps to reconstruct hidden targets with high fidelity and has excellent noise suppression effect, which can obtain significantly higher SSIM under sub-millisecond exposure time conditions.

[0092] According to equation (18), the SBR of the signal received by the NLOS imaging system is independent of the front-end distance but rapidly decays with the increase of the back-end distance of the "relay wall-target". The real target and size are as shown in Figure 12(a) in FIG. 6. At 2m, 64x64 scanning was performed in a 2m x 2m field of view with 1ms exposure time per point at 42mW laser power. For quantitative comparison, the FK reconstruction result of high power data at 2m was chosen as the GT for SSIM calculation. The reconstruction results are shown in Figure 12 (c) in FIG. 6.

[0093] At 2m, both the conventional method and the proposed method successfully reconstructed the structure information of the target. The proposed method showed significantly reduced noise and achieved a 0.1 improvement in SSIM. At 3m, it was clear that the two slant edges and the vertical structure of "K" became blurred in the conventional FK reconstruction. In contrast, the result of the proposed method still clearly reconstructed the "K" structure and eliminated most of the noise. At 4m, the conventional method could only locate the position of the target but could not reconstruct it. In contrast, the proposed method successfully separated the signal from the noise and reconstructed the structure of the target. This experiment showed that even when the back-end distance was extended to 4m, the proposed method could reconstruct the shape from extremely sparse photons, thus providing technical support for long-distance NLOS imaging.

[0094] Adaptive windowing enables robust NLOS reconstruction in the condition of photon scarcity. Based on the distribution model of different probabilities of signal and noise photons, the proposed windowing processing method can effectively separate each component detection. The essence of this method has two key points, one is to group the signal photons returned from the spatiotemporal correlated pixel block to enhance the SNR, and the other is to remove as much noise as possible through adaptive windowing based on matched filtering. Experimental verification shows that under the condition of signal as low as 0.02 photons per pixel (which is one of the lowest values reported in current NLOS imaging), the proposed method successfully reconstructs the hidden target that the conventional method cannot identify, achieving a structural similarity index that is 0.5 higher than other methods. In the case of total exposure time less than 0.82s, the completely hidden scene is reconstructed, and it can be achieved at the farthest distance of 4m, successfully overcoming the key bottleneck in practical applications.

Claims

1. A method for few-photon non-line-of-sight imaging based on adaptive windowing, characterized in that, The method comprises the following steps: Step 1, using a non-coaxial two-dimensional scanning system to collect photon signals reflected from a hidden target, establishing a probability model of signal photons and noise photons, and analyzing the distribution characteristics of the photons in the time domain; Step 2, using the space-time correlation, grouping adjacent pixels into a pixel block, determining the adaptive window width of each pixel block through a matched filter method, and applying a windowing operation on the time domain to separate signal photons from noise photons; Step 3, using a total variation regularization constraint to fill the windowed transient data, solving the reconstruction problem by an alternating direction multiplier method, and obtaining a target reconstruction result.

2. The adaptive windowing based few-photon non-line-of-sight imaging method of claim 1, wherein, Step 1 specifically comprises: Under the condition that the working wavelength is The number of signal photons detected per second by the detector is expressed as: ； （1） wherein, is the average optical power of the pulsed laser, is the Brinkman constant, c is the speed of light, , are the reflectivities of the target and relay wall, respectively, is the angle between the normal of the relay wall and the laser beam, is the area of the hidden target, is the field of view range on the relay wall, is the area of the field of view received by the detector, is the transmittance of the receiving system, is the transmittance of the air, is the distance of the optical system from the relay wall, is the distance of the target from the relay wall; In addition to signal photons, the NLOS imaging system also receives noise photons consisting of background noise and dark noise, the number of noise photons received per second is represented as: ； （2） wherein, is the bandwidth of the ideal bandpass filter, is the solar spectral irradiance, is the solar incidence angle, is the dark count rate of the detector; When the system detection efficiency of the probe is the photon flux reaching the probe is expressed as: ； （3） The arrival times of the photons collected by the SPADs can only be recorded discretely in different time bins, within a width of The average number of photons detected within the nth time bin is expressed as: ； （4） wherein is the time bin index, is the maximum index of the time bin; The photon signal detected by the SPAD obeys a Poisson distribution, the probability of observing k photons in the nth time bin is is represented as: ； （5） The probability of detecting a signal photon within a first time bin is represented as: ； （6） Dead time for SPADs; Detection efficiency of noise photons is represented as: （7）。 3. The adaptive windowing based few-photon non-line-of-sight imaging method of claim 2, wherein, Step 2 specifically comprises: By utilizing the spatio-temporal correlation of the surrounding scanning points, the nearest 4x4 pixels are grouped into a pixel block The photon counts within the pixel block are represented as: ； （8） wherein, is a pixel block index, the photon flux acquired for each scan point; The photon counting data is pre-processed using a matched filtering method; a system response function IRF is used as a template for the matched filtering, the system response function The matched filtering result of each pixel block is expressed as: ； （9） The larger the pixel block The higher the probability that the photon counting event in the hidden target comes from The window width of the pixel block is calculated by the extreme point and the half-height bandwidth of the system : ； （10） The windowing is applied to the original transient information in the time domain, and the windowing process is represented as: ； （11） wherein is the start time of the window, .

4. The adaptive windowing based few-photon non-line-of-sight imaging method of claim 3, wherein, Step 3 specifically comprises: The TV regularization constraint is introduced to fill in the transient information after windowing, and the optimized transient information is represented as: ； （12） wherein, is a regularity parameter associated with the measured signal-to-noise level; In the confocal NLOS scenario, the axial position of the relay wall is set to 0; the instantaneous information captured at the scanning point is represented as: ； （13） where is the point on the hidden object the distance to the laser or detector; represents the attenuation of the photons scattered by the relay wall after two reflections, is the time of flight of the photons; the Dirac function represents the surface of the four-dimensional hyper-cone of spacetime, which models the light propagation process from the relay wall to the hidden target and back to the relay wall; the variable substitution on equation (13) is: ​ ； （14） The above equation is a 3D convolution process: ; where, represents a 3D convolution kernel, and represent the resampling operations along the z-axis and t-axis, respectively; from the transient information solving the problem formulation is to solve a regularized least square problem; let be the 3D convolution operation associated with the kernel The problem is represented in matrix form as: ； （15） wherein, is a regularization parameter that serves as a weighting constant balancing the trade-off between the data fidelity term and the regularization term; the minimization problem is solved using an alternating direction method of multipliers.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the method according to any one of claims 1-4.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method according to any one of claims 1-4.

7. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1-4.

Images