On-site Non-line-of-sight Imaging Method, Device, Equipment and Medium Based on Online Calibration

Abstract

A method, device, equipment, and medium for on-site non-line-of-sight imaging based on online calibration provided by the present application decouple the transient data collected on-site by the acquisition system into a visible-field transient component and a hidden transient component based on the confocal imaging principle; calculate the Gamma map based on the visible-field transient component, and re-perform online calibration and calibration on the acquisition system when at least one of the scene, relay surface setting, scanning area, and scanning mode changes; process the hidden transient component using an optimization algorithm based on the confocal imaging model, and use the gradient descent method to perform on-site reconstruction of the non-visible-field scene of the hidden object. The present application can decouple the transient obtained during on-site scanning into a visible field and a hidden component, avoiding the use of auxiliary calibration equipment such as mirrors or checkerboards; after the user performs online calibration on the galvanometer and the relay surface, they can adjust the scanning area mode, and through the Gamma map preview, they can define any scanning mode to measure different hidden scenes, and support the input transient of any scanning mode for NLOS reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of non-line-of-sight imaging, and particularly to a method, device, equipment and medium for on-site non-line-of-sight imaging based on online calibration. Background Art

[0002] People are increasingly concerned about deploying non-line-of-sight (NLOS) imaging systems to reconstruct objects behind obstacles. Non-line-of-sight (NLOS) imaging aims to reconstruct objects outside the direct detection range (line of sight) of sensors. Most active non-line-of-sight imaging uses ultrafast pulsed lasers and controls the laser beam to point to a relay surface (such as a wall). Photons will undergo three reflections: leaving the relay surface, leaving the hidden object, and then returning from the relay surface. Then, a time-resolved detector is used to collect the photons of the first and third reflections in the three reflections, and the arrival time and photon number information are recorded.

[0003] As Figure 1 shows a top view of the conventional setup of the NLOS imaging acquisition system. The first reflection corresponds to the direct reflection in the visible field, from which the geometry and reflectivity of the relay surface can be reconstructed. After removing or shielding the photons of the first reflection, the photons of the third reflection can be used to reconstruct and locate the hidden object. Non-line-of-sight imaging technology has many potential applications, including autonomous driving, remote sensing, and biomedical imaging, etc.

[0004] To improve the quality of NLOS reconstruction, researchers have improved the hardware and acquisition systems and made great contributions. For the hardware part, the focus of the development direction is to use more accurate and reasonably priced detectors and lasers. For example, a streak camera provides an accurate time resolution as low as 2 ps (corresponding to a spatial resolution of 0.6 mm), pioneering NLOS imaging. However, such a camera is extremely expensive and difficult to calibrate under non-linear spatio-temporal transformation. Another example is that a photon mixer device (PMD) has the advantages of miniaturization and low cost, but can only provide a time resolution of the order of nanoseconds.

[0005] In recent years, single-photon avalanche diodes (SPADs) have become an economical and convenient alternative. A histogram of photon counts and time intervals (4 ps) is generated at the detection point using a single-pixel SPAD with a time-correlated single-photon counting (TCSPC) module. At the laser front end, a picosecond or femtosecond pulsed laser is used to illuminate a point on the relay surface, and a pulse signal is sent and synchronized with the detector to record the arrival time of the returned photons. The characteristics of the laser, such as the average power and repetition frequency, will affect the acquisition time and the measurement effect.

[0006] Despite these improvements, building a system still requires careful calibration to obtain accurate information such as the three-dimensional coordinates of transient and scan points. In a typical NLOS system, most reconstruction algorithms use a method of uniform grid scanning consisting of N×N equally spaced points. However, the use of non-uniform sampling in NLOS imaging has gradually become a new trend, and in this method, the measurement system needs to be recalibrated. In addition, in actual measurements, changes in the position and orientation of occluders and hidden objects are also common, which also requires recalibration.

[0007] Existing solutions usually pre-calibrate the system before scanning hidden objects. On-site adjustment of the relative position of the detector-relay plane, the position and orientation of the hidden object, and the scanning mode requires re-calibration of the system. So far, almost all existing NLOS calibration schemes require auxiliary equipment. For example, a checkerboard can be used to obtain the internal and external parameters of the camera, or a regular fence grid point coupled with the camera can be used to obtain the three-dimensional coordinates of the grid points. Alternative solutions include using a mirror to establish the correspondence between the laser point and the detection point to simultaneously estimate the mirror plane and the position of the relay wall. When the scan points are reselected, or the position of the relay wall surface relative to the system changes, the above methods need to be re-calibrated, and these methods are both laborious and time-consuming, with great drawbacks. Summary of the Invention

[0008] In view of the above-mentioned disadvantages of the prior art, the purpose of this application is to provide a method, device, equipment and medium for on-site non-line-of-sight imaging based on online calibration to solve the problems in the prior art.

[0009] To achieve the above object and other related objects, this application provides a method for on-site non-line-of-sight imaging based on online calibration, the method includes: decoupling the transient data collected by the acquisition system on-site into a visible transient component and a hidden transient component based on the confocal imaging principle; calculating the Gamma map based on the visible transient component, and re-performing online calibration and calibration on the acquisition system when at least one of the scene, relay plane setting, scanning area, and scanning mode changes; processing the hidden transient component by an optimization algorithm based on the confocal imaging model, and using the gradient descent method to perform on-site reconstruction of the non-visible scene of the hidden object.

[0010] In one embodiment of the present application, the acquisition system includes: a hardware module, a LOS module, and an NLOS module; the hardware module includes: a pulsed laser, an SPAD as a detector, a beam splitter, a galvanometer, and a controller; the LOS module is used to associate the hardware module with the relay plane and corresponds to the transient LOS component; wherein, the relay plane, the scanning mode, and the measurable bounding box are defined by using the LOS module to detect hidden objects located within the bounding box; the NLOS module is used to collect the spherical light waves propagating between the relay plane and the hidden object and contains information about the hidden object for reconstruction.

[0011] In one embodiment of the present application, the transient data includes:

[0012] ≤(t; o, l, s, d) = Γ(s)∫ P δ(t - (t o→l + t l→p + t p→s + t s→d ))f(p; ω l→p , ω p→s )g(p; l, s)dA p ;

[0013] where τ represents the transient data; t represents time; o represents the illumination point of the light source; l represents the spot position of the light source on the relay plane; s represents the reflection position where the light source is scattered from the relay plane and reflected back to the relay plane by the reflection point p on the hidden object P; d represents the detection point of the detector for detecting the number of returned photons; Γ(s) represents the intensity change coefficient of the light source after reflection on the relay plane and can describe the detector's field of view; the integral ∫ P δ represents the total number of photons reflected from the micro - surface element A p centered at the reflection point p; → represents the light source path; the unit vector represents the direction vector from l to p; the unit vector represents the direction vector from p to s; f represents the bidirectional reflection function corresponding to the incident direction ω l→p and the outgoing direction ω p→s at the reflection point p; the function g represents the attenuation term including distance, light and shadow, and occlusion effects; N o represents the total number of photons emitted by the light source in one pulse from the illumination point o; ρ represents the reflectivity of the relay plane; A d represents the effective area of the detector; A s represents the detection area of the detector projected onto the relay plane; n s represents the surface normal vector at the reflection position s.

[0014] In one embodiment of the present application, the method includes: modeling the data recorded by the SPAD as: τSPAD = Pois(τ*j + b); j(t; μ, σ, κ0, κ1, γ) = Gaus(t, μ, σ) + γExp(t; μ, κ0, κ1); Wherein, SPAD represents a single-photon avalanche diode as a detector; τ represents transient data; Pois represents a Poisson likelihood function; j represents the time jitter of the entire system, and the time jitter includes two parts: a Gaussian peak represented by Gaus and an exponential tail represented by Exp(t; μ, κ0, κ1); μ, σ, κ0, κ1 represent relevant parameters of the time jitter, γ represents the weight of the exponential tail; b represents the SPAD background noise; the time jitter of multiple detection points is calculated, and the average value is taken as the time jitter of the acquisition system.

[0015] In an embodiment of the present application, the method includes: realigning the transient data of each detection point so that the arrival time of the directly reflected photons on the relay plane is at zero time; using a Gamma map to normalize the transient data τ(t; s) to enhance the acquisition quality; applying Wiener filtering to the transient data τ SPAD for noise reduction: Wherein, and respectively represent the Fourier transforms of τ(t; s) and τ SPAD (t; s); v is the frequency; the Wiener filtering kernel in the frequency domain is calculated by the Fourier transform of the time jitter and the signal-to-noise ratio η:

[0016] In an embodiment of the present application, the decoupling of the transient data collected on-site by the acquisition system into a visible-field transient component and a hidden transient component based on the confocal imaging principle includes: when the irradiation point o and the detection point d coincide to form a confocal, the transient data is simplified to: τ(t; o, s) = Γ(s)∫ P δ(t - 2(t o→s + t s→p ))f(p; ω p→s )g(p; s)dA p ; τ hidden (t; s) = ∫ P δ(t - 2t s→p )f(p; ω p→s )g(p; s)dA p ; τ LOS (t; o, s) = Γ(s)δ(t - 2t o→s ); wherein, τ(t; o, s) includes: τ LOS (t; o, s) is the visible-field transient component along the direct optical path between the relay plane and the detector, τhidden (t;s) is the hidden transient component along the indirect optical path between the relay surface and the hidden object.

[0017] In an embodiment of the present application, the optimization algorithm based on the confocal imaging model processes the hidden transient component, and uses the gradient descent method to perform on-site reconstruction of the non-line-of-sight (NLOS) scene of the hidden object, including: based on the assumption that the hidden object P is a perfect diffuse reflector, there is no occlusion between the hidden object P and the relay surface, and mutual reflection between the surfaces of the hidden object P is not considered, the attenuation term g(p;s) can be regarded as The bidirectional reflection function f(p;ω p→s ) is independent of the viewing angle and can be written as f(p), τ hidden (t;s) can be simplified to: τ = Ψf; where τ represents the transient data, Ψ represents the linear measurement matrix containing g(p;s); f represents the reflectivity; where NLOS represents the non-line-of-sight; represents the loss function; e i represents the i-th standard unit vector, τ i represents the i-th element of τ, and λ is the weight of the total variation regularization term; TV represents the total variation regularization term; the gradient descent method is used to achieve on-site reconstruction of the hidden object in the non-line-of-sight scene.

[0018] In an embodiment of the present application, calculating the Gamma map based on the line-of-sight transient component includes: when the detection point d is fixed, defining the intensity change coefficient Γ(s) of the light source after reflection on the relay surface by determining the coordinates of the reflection position s on the relay surface; based on confocal imaging, the integral of the δ function is 1, and Γ(s) is expressed as: Γ(s) = ∫τ LOS (t;o,s)dt.

[0019] In an embodiment of the present application, when at least one of the scene, relay surface settings, scanning area, and scanning mode changes, re-performing on-line calibration and calibration of the acquisition system includes: adjusting the optical path by observing the light spot on the relay surface; adjusting the beam splitter and galvanometer so that the light spot on the relay surface is clearly visible; finely adjusting the position of the SPAD so that the light reflected from the relay surface received by it reaches the maximum.

[0020] In an embodiment of the present application, when at least one of the scene, relay surface settings, scanning area, and scanning mode changes, re-performing on-line calibration and calibration of the acquisition system includes: assuming that the scanning system is linear, on-line calibrating the galvanometer; modeling the relationship between the optical deflection angle θ X , θ Y and the input voltage V X , V Y : where the initial deflection angle ∈ X and ∈Y Determined by the offset of two mirrors without input voltage; the β matrix represents the coefficients of a pair of angles relative to a pair of input voltages; the coefficients of β are optimized using a multiple linear regression algorithm with a loss function corresponding to the galvanometer ; Take the average error between the measured and calculated optical scanning angles as ∈ X and ∈ Y ; Calculate the input voltage of each group of scanning angles to control the scanning points on any surface and define the scannable area of the galvanometer.

[0021] In an embodiment of the present application, when at least one of the scene, relay surface setting, scanning area, and scanning mode changes, re - calibrating and calibrating the acquisition system online includes: based on the transient component τ LOS of the visible field, reconstruct the reflectivity and surface normal vector of the relay surface; extract τ LOS (t; s) of each detection point and its corresponding t, and calculate the depth of the relay surface Use the optical deflection angle θ X , θ Y and the depth to estimate the 3 - D coordinates of the detection point in XYZ: Among them, the relay surface is regarded as a plane W (W X , W Y , W Z ), and the equation is: W: W X X + W Y Y + W Z Z + 1 = 0; Solve for the plane W (W , W X , W Y , W Z ) of the relay surface by minimizing the root - mean - square error of the distance from a point to a plane:

[0022] In an embodiment of the present application, when at least one of the scene, relay surface setting, scanning area, and scanning mode changes, re - calibrating and calibrating the acquisition system online includes: realizing any scanning mode by calculating the input voltage corresponding to the coordinates of each detection point on the relay surface; Define a scanning area, with the origin o s located at the center of the relay surface, and construct a new set of orthogonal bases through the orthogonal basis ; Among them, is the unit normal vector of the relay surface; for the plane relay surface z = 0, therefore, the detection point can be represented in two coordinate systems as: The galvanometer input voltage is obtained by calculating the scanning angle of each detection point on the relay surface.

[0023] In one embodiment of the present application, the online recalibration and calibration of the acquisition system according to at least one change of the scene, the relay surface setting, the scanning area and the scanning mode includes: creating a free space by using the relay surface with adjustable direction and position to allow larger hidden objects; the orthogonal projection of the scanning range is used to limit the measurable bounding box of the NLOS scene with the maximum width and height of the scanning area, and the minimum depth of the bounding box is approximately: z min =ct delay ; where z represents the depth of the bounding box; t delay represents the delay in the arrival time of photons returning from the relay surface and the hidden object, and c represents the speed of light; the photon attenuation between the hidden object and the relay surface determines the maximum depth of the bounding box; Among them, the minimum value of the intensity variation coefficient Γ min The volume used to limit the scanning range and bounding box; b represents the SPAD background noise.

[0024] To achieve the above-mentioned purpose and other related purposes, the present application provides an electronic device, which includes: a decoupling module, which is used to decouple the transient data collected by the acquisition system on-site into a visible transient component and a hidden transient component based on the confocal imaging principle; a calibration module, which is used to calculate the Gamma map based on the visible transient component, and re-calibrate and calibrate the acquisition system online when at least one of the scene, relay surface setting, scanning area and scanning mode changes; a reconstruction module, which is used to process the hidden transient component based on the optimization algorithm of the confocal imaging model, and use the gradient descent method to reconstruct the hidden object on-site in the non-visual scene.

[0025] To achieve the above-mentioned purpose and other related purposes, the present application provides a computer device, which includes: a memory and a processor; the memory is used to store computer instructions; the processor executes the computer instructions to implement the above-mentioned method.

[0026] To achieve the above-mentioned purpose and other related purposes, the present application provides a computer-readable storage medium storing computer instructions, which execute the above-mentioned method when executed.

[0027] In summary, a method, device, equipment, and medium for on-site non-line-of-sight imaging based on online calibration in this application decouple the transient data collected on-site by the acquisition system into a visible-field transient component and a hidden transient component based on the confocal imaging principle; calculate the Gamma map based on the visible-field transient component, and re-perform online calibration and calibration on the acquisition system when at least one of the scene, relay surface setting, scanning area, and scanning mode changes; process the hidden transient component using an optimization algorithm based on the confocal imaging model, and use the gradient descent method to perform on-site reconstruction of the non-visible-field scene of the hidden object.

[0028] It has the following beneficial effects:

[0029] This application can directly decouple the transient obtained during on-site scanning into a visible field and a hidden component, avoiding the use of auxiliary calibration devices such as mirrors or checkerboards, and supporting laboratory verification and actual deployment; after the user performs online calibration on the galvanometer and relay surface, they can adjust the scanning area and mode of the detection points. Through the Gamma map preview, any scanning mode can be defined to accurately measure different hidden scenes. A small number of detection points can be scanned, and the preview reconstruction result can be used to determine whether the current scanning settings are acceptable, and it supports the input transient of any scanning mode for NLOS reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It shows a top view schematic diagram of the conventional settings of the NLOS imaging acquisition system in an embodiment of this application.

[0031] Figure 2 It shows a flowchart schematic diagram of the on-site non-line-of-sight imaging method based on online calibration in an embodiment of this application.

[0032] Figure 3 It shows a schematic diagram of the structure of the acquisition system in an embodiment of this application.

[0033] Figure 4 It shows a schematic diagram of the scene of the coordinate system and the scanning area in an embodiment of this application.

[0034] Figure 5 It shows a schematic diagram of the scanning points in a scanning mode in an embodiment of this application.

[0035] Figure 6 It shows a model schematic diagram of reconstructing 'S' using the LCT, FK, and PF algorithms in an embodiment of this application.

[0036] Figure 7 It shows a model schematic diagram of the normalized transient data in the regular grid mode in an embodiment of this application.

[0037] Figure 8It shows a schematic model diagram of the normalized transient data in the regular grid pattern in another embodiment of the present application.

[0038] Figure 9 It shows a schematic module diagram of an electronic device in an embodiment of the present application.

[0039] Figure 10 It shows a schematic structural diagram of a computer device in an embodiment of the present application. Detailed implementation manners

[0040] The following uses specific specific examples to illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0041] It should be noted that the diagrams provided in the following embodiments only schematically illustrate the basic concept of the present application. Although only the components related to the present application are shown in the diagrams and are not drawn according to the number, shape, and size of the components in actual implementation, the type, quantity, and ratio of each component in actual implementation can be an arbitrary change, and the component layout type may also be more complex.

[0042] Throughout the specification, when it is said that a certain part is "connected" to another part, this includes not only the case of "direct connection", but also the case of "indirect connection" with other elements placed in between. In addition, when it is said that a certain part "includes" a certain constituent element, unless there is a particularly contrary record, it does not mean excluding other constituent elements, but means that other constituent elements can also be included.

[0043] The first, second, and third terms mentioned therein are used to illustrate various parts, components, regions, layers, and / or segments, but are not limited thereto. These terms are only used to distinguish a certain part, component, region, layer, or segment from other parts, components, regions, layers, or segments. Therefore, the first part, component, region, layer, or segment described below can refer to the second part, component, region, layer, or segment without exceeding the scope of the present application.

[0044] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should be further understood that the terms "comprising", "including" indicate the presence of the stated features, operations, elements, components, items, species, and / or groups, but do not preclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, species, and / or groups. The terms "or" and "and / or" used herein are to be construed as inclusive, or meaning any one or any combination. Thus, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". An exception to this definition occurs only when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0045] To solve the above problems, the present application proposes a field non-line-of-sight imaging method based on online calibration, which is applicable to customized reconstruction schemes for any sampling mode. This scheme uses the gradient descent method to iteratively reconstruct the NLOS scene. By supporting online calibration and on-site reconstruction evaluation, the present application enables more feasible laboratory verification and actual measurement, and will be open-sourced to the community.

[0046] As Figure 2 shown, it is a schematic flowchart of the field non-line-of-sight imaging method based on online calibration in an embodiment of the present application. As shown in the figure, the method includes:

[0047] Step S201: Decouple the transient data collected on-site by the acquisition system into a visible-field transient component and a hidden transient component based on the confocal imaging principle.

[0048] In an embodiment of the present application, the non-line-of-sight imaging model models the propagation process of light from the laser light source through the visible field and the hidden object and finally being received by the detector. As Figure 1 shown, the light source (pulsed laser beam) irradiates from the irradiation point o to the relay surface (wall) W and forms a light spot at l. After the light is scattered from this point, some photons are reflected from the reflection point p on the hidden object P and return to the reflection position (microfacet) s on the relay surface. The detection point d of the detector records the number of photons returned from the reflection position (microfacet) s received at a certain moment t. Based on the physics of light transmission, the forward model of imaging is defined as:

[0049] τ(t; o, l, s, d) = Γ(s) ∫ P δ(t - (t o→l + t l→p + t p→s + t s→d )) f(p; ω l→p , ω p→s ) g(p; l, s) dA p(1)

[0050]

[0051] where τ(t; o, l, s, d) records the 5-dimensional transient data, i.e., the histogram of the number of photons returned from the hidden object with respect to time. The integral ∫ P δ represents the sum of the number of photons returned from the microfacet A centered at p p . The Dirac delta function δ(·) relates the time t to the propagation times t o→l , t l→p , t p→s and t s→d . The function f(p; ω l→p , ω p→s ) describes the bidirectional reflectance distribution function (BRDF) with respect to the point p, the incident direction ω l→p and the outgoing direction ω p→s . The unit vector represents the direction vector from a to b. The function g(p; l, s) is the attenuation term, including distance, light and shadow, and occlusion effects. To simplify the model, equation (1) ignores the wave nature of light, such as interference, diffraction, etc.

[0052] Γ(s) models the intensity change of light reflected from the relay surface and can describe the field of view (FOV) of the detector. In equation (2), N o is the total number of photons emitted by the laser in one pulse from o. ρ is the reflectivity of the relay surface. A d , A s are the effective area of the detector and the detection area projected onto the relay surface, respectively. n s is the surface normal vector at s.

[0053] Under conventional settings, both the illumination point and the detection point can vary, or one point is fixed and the other point varies. The calibration procedure is complex because t o→l and t s→d are difficult to separate solely based on the measurement results. When the illumination point and the detection point coincide, i.e., l = s, o = d, the recorded transient data will be a 3-dimensional subset of τ(t; o, l, s, d). In the confocal setting, equation (1) can be simplified to:

[0054] τ(t; o, s) = Γ(s)∫ P δ(t - 2(t o→s + t s→p ))f(p; ω p→s )g(p; s)dA p (3)

[0055] The confocal imaging model has advantages in system calibration because the propagation time t between s and o o→s is relatively easy to determine. τ(t; o, s) consists of two components: τ LOS (t; o, s) along the direct optical path between the relay plane and the detector, and τ hidden (t; s) along the indirect optical path between the relay plane and the hidden object. The relationship between these two components is convolutional.

[0056] τ hidden (t; s) = ∫ P δ(t - 2t s→p ) f(p; ω p→s ) g(p; s) dA p (4)

[0057] τ LOS (t; o, s) = Γ(s) δ(t - 2t o→s ) (5)

[0058] In NLOS reconstruction, the LOS component is usually masked or removed to obtain the hidden component, which is equivalent to assuming a virtual light source at l and a virtual detector at s. τ LOS contains information on the relay plane, such as depth and reflectivity, enabling the present application to calibrate the acquisition system.

[0059] As Figure 3 shown, it is a schematic structural diagram of the acquisition system in an embodiment of the present application. As shown in the figure, the acquisition system includes an optical path and the optical path design of hardware devices. The acquisition system includes: a hardware module, a LOS module, and an NLOS module.

[0060] The hardware module includes: a pulsed laser Laser, a single-photon avalanche diode SPAD as a detector, a beam splitter Splitter, a galvanometer Galvanometer, and a controller Computer. At the same time, there are also a photon counter TCSPC, a delay line PSD, and a data acquisition device NI-DAQ.

[0061] The LOS module is used to connect the hardware module and the relay plane and corresponds to the transient LOS component; among them, the relay plane, the scanning mode, and the measurable bounding box are defined by using the LOS module, and the hidden object located within the bounding box can be detected.

[0062] The NLOS module is used to collect the spherical light waves propagating between the relay plane and the hidden object and contains information about the hidden object for reconstruction.

[0063] This application can be combined with application programs provided by hardware device manufacturers, such as TCSPC and galvanometric mirrors SPAD, and operate the entire program with minimal human intervention.

[0064] In an embodiment of this application, the measured transient data obtained from the acquisition system is affected by SPAD characteristics (including photon detection efficiency, afterpulsing, and pile-up effects), and the time jitter of the laser and SPAD. Time jitter describes the uncertainty in the time-resolved mechanism. This application uses an approximate model to describe time jitter. The number of photons generated by the SPAD at the detection point in the histogram of photon count versus time interval (4 ps) for each time interval has the characteristics of Poisson statistics. The background noise b is usually considered to be independent of time at a single detection point, and its sources include dark counts and ambient light. Considering these factors, this application models the data recorded by the SPAD (single-photon avalanche diode) as:

[0065] τ SPAD = Pois(τ * j + b) (6)

[0066] where τ represents the transient data; here j represents the time jitter of the entire system, which has two parts: a Gaussian peak and an exponential tail:

[0067] j(t; μ, σ, κ0, κ1, γ) = Gaus(t, μ, σ) + γExp(t; μ, κ0, κ1) (t > 0) (7)

[0068]

[0069]

[0070] where μ, σ, κ0, κ1 are the relevant parameters of the time jitter, γ is the weight of the exponential tail, and b represents the SPAD background noise. LOS transient measurement can be used to calculate the time jitter j of the acquisition system. It is optimized through the cross-entropy loss function:

[0071]

[0072] Calculate the time jitter at multiple detection points and take the average as the time jitter of the acquisition system.

[0073] Step S202: Calculate the Gamma map based on the visible field transient component, and re-perform online calibration and calibration of the acquisition system when at least one of the scene, relay surface setting, scanning area, and scanning mode changes.

[0074] In this application, an online calibration technique is proposed. This technique directly decouples the transient signal obtained from on-site scanning into the LOS component and the hidden component. The former contains the geometric structure and reflectivity information of the relay surface and the detection point. According to the Gamma map calculated from the LOS component, when the scene / obstacle configuration, scanning area, or scanning mode changes, the system is directly (re)calibrated and normalized. This also provides a useful reference for dynamic adjustment: this application can scan a small number of detection points and preview the reconstruction result to determine whether the current scanning settings are acceptable.

[0075] The purpose of calibration is to accurately measure by minimizing the difference between the transient data measured by the acquisition system of this application and the transient data theoretically calculated based on the confocal imaging model. The online calibration of this application can (re)adjust the hardware device with little human intervention to calibrate components including galvanometers, relay surfaces, and NLOS bounding boxes, and use the Gamma map to preview the calibration effect.

[0076] In an embodiment of this application, in formula (2), it can be observed that Γ(s) is related to the N of the laser o , related to the A of the detector d , related to the ρ of the relay surface, and related to the detection point s, and also related to the distance |s - d| from d to s. When the detector d is fixed, this application can define Γ(s) by determining the coordinates of the detection point s on the relay surface. Based on the confocal imaging model, considering that the integral of the δ function is 1, the Γ coefficient is calculated by summing the LOS part τ LOS (t; s) in formula (5) at each detection point. The Γ(s) coefficient, or Gamma map, is calculated for all detection points on the relay surface.

[0077] Γ(s) = ∫τ LOS (t; o, s)dt (11)

[0078] After online calibration, the Gamma map can be used to preview the scanning range, field of view, NLOS bounding box, etc. In equation (6) of the SPAD-based acquisition system, the integral of the time jitter j is also 1 like the δ function. Considering the additivity of the Poisson distribution and that the background count b is very small relative to the LOS transient measurement, this application can still calculate the Gamma map using the LOS transient data according to formula (11).

[0079] In one embodiment of the present application, for the method of calibrating a hardware module, the optical path is initially adjusted by observing the laser spot on the relay surface (such as a white paper). Then, the beam splitter and the galvanometer are adjusted so that the spot on the relay wall is clearly visible. Finally, the position of the SPAD is finely adjusted so that the primary reflected light received from the relay surface reaches the maximum. At high laser power settings, since the light directly reflected on the relay surface may flood the SPAD, in the present application, the position of the SPAD is slightly misaligned to reduce the intensity of the measured primary reflected light.

[0080] In one embodiment of the present application, the biaxial galvanometer used in the present application supports an optical scanning angle of about ±40°, which specifically depends on factors such as the laser beam diameter and the input voltage. Generally, the galvanometer is coupled with a servo motor, and the servo motor can feedback the deflection angle during scanning. The present application calibrates the relationship between the input voltage and the deflection angle, and further determines the position of the scanning point.

[0081] As Figure 4 shown, it illustrates the coordinate system and the scanning area. The two Cartesian coordinate systems include the XYZ coordinate system with the origin at o (where the laser emits into free space towards the relay surface), and the xyz coordinate system with the origin at the center o s of the detection area on the relay surface (the equation of the relay surface is z = 0). Scanning area: Area A is determined by the galvanometer, Area B takes into account the occlusion factor, and Area C is the determined scanning range.

[0082] In the present application, in order to calibrate the galvanometer, the present application assumes that the scanning system is linear, and models the relationship between the optical deflection angles θ X , θ Y and the input voltages V X , V Y :

[0083]

[0084] where, the initial deflection angles ∈ X and ∈ Y are determined by the offsets of the two mirrors without input voltage, and the β matrix represents the coefficients of a pair of angles relative to a pair of input voltages. The initial angles ∈ X , ∈ Y and the coefficient β may be provided by the manufacturer. For the sake of precision, the present application collects N groups of optical scanning angles θ n and the input voltages V n within a given voltage range to calculate them. The coefficients of β are first optimized using a multiple linear regression algorithm with a loss function :

[0085]

[0086] Then, the average error between the measured and calculated optical scanning angles is taken as ∈ X and ∈ Y .

[0087] Finally, the input voltage for each set of scanning angles is calculated to control the scanning points on any surface and define the scannable area of the galvanometer (the area A in Figure 4 ), note that the area A is not necessarily rectangular.

[0088] In an embodiment of the present application, for the relay surface, it plays a crucial role in the acquisition system. The transient LOS component, as shown in Equation (5), can be used to reconstruct the reflectivity and surface normal of the relay surface. Specifically, the present application extracts the histogram τ LOS (t; s) and its corresponding t for each detection point, and calculates the depth of the relay surface Then, the optical deflection angles θ X , θ Y of the galvanometer and the depth are used to estimate the three-dimensional coordinates of the detection points in XYZ as follows:

[0089]

[0090] The relay surface can be regarded as a plane W (W X , W Y , W Z ), and the equation is:

[0091] W: W X X + W Y Y + W Z Z + 1 = 0 (15)

[0092] It should be noted that the reflectivity and the direction of the relay surface are considered in the Gamma diagram. Therefore, the calibration technology does not require any additional equipment or textured targets. The plane is solved by minimizing the loss function i.e., the root mean square error (RMSE) of the distance from the point to the plane.

[0093]

[0094] In an embodiment of the present application, for the scanning mode, it is used to associate the hardware module with the relay surface and the NLOS bounding box, and the calibration effect of the entire system can be previewed. In the existing literature, the detection points are usually distributed in a regular grid, and the point distribution spacing on the relay surface is uniform. The acquisition system of the present application realizes any scanning mode by calculating the input voltage corresponding to the coordinates of each detection point on the relay surface. Specifically, the present application uses w (w X , wY , w Z ) Rewrite:

[0095]

[0096] In the effective scanning range restricted by the Gamma diagram, the present application defines a scanning area ( Figure 4 area C in s which the origin o is located at the center of the relay plane. A new set of orthogonal bases can be constructed through the orthogonal basis

[0097]

[0098] is the unit normal vector of the relay plane. Note that for the planar relay plane z = 0, the detection points can be respectively expressed in the two coordinate systems as:

[0099]

[0100] According to formulas (12) and (18), the galvanometer input voltage can be obtained by calculating the scanning angle of each detection point on the relay plane.

[0101] The present application previews the calibration effects of the galvanometer scanning area, the relay plane, and the scanning mode. First, a small number of detection points are scanned to outline the scannable area, and the Gamma diagram is calculated to adjust the position of the relay plane relative to the galvanometer. Then, the Gamma diagram is calculated to check the area blocked by the entire setup of the acquisition system. After selecting the scanning mode, the Gamma diagram can be calculated to verify the field of view defined by the scanning mode for NLOS measurement.

[0102] In an embodiment of the present application, for the NLOS bounding box: The reconstruction accuracy depends on several factors of the acquisition system, including the geometric setup of the hardware module and the scanning area on the relay plane. For effective measurement, the present application defines a bounding box to specify the position of the object that the acquisition system can detect in the hidden scene.

[0103] Scholars such as Ahn et al. pointed out that the orthogonal projection of the reconstruction result should be within the scanning range on the relay plane. In practice, the present application creates a free space by using a relay wall with adjustable direction and position to allow larger hidden objects. This setup is equivalent to adjusting the position of the entire hardware setup. The orthogonal projection of the scanning range limits the measurable bounding box of the NLOS scene with the maximum width and height of the scanning area. And the minimum depth of the bounding box is approximately:

[0104] z min = ct delay (20)

[0105] ​Where z represents the depth of the bounding box; t delay represents the time delay of the arrival time of photons returned from the relay plane and the hidden object, and c represents the speed of light.

[0106] Since the distance between the hidden object and the relay plane plays an important role in photon attenuation, this application believes that it determines the maximum depth of the bounding box:

[0107]

[0108] Where the minimum value Γ of the Gamma graph min limits the scanning range and the volume of the bounding box. The bounding box helps to preview the detectable size and position of the hidden object and a rough estimate of the reconstruction quality.

[0109] In an embodiment of this application, first, the histogram (transient data) of each detection point is realigned so that the arrival time of the photons directly reflected by the relay plane is at zero time. Using the Gamma graph, this application normalizes the transient τ(t; s) to obtain higher quality. In Equation (6), it can be noted that the time jitter j(t) is an important factor in the SPAD characteristic. Therefore, Wiener filtering can be applied to the transient data τ SPAD for noise reduction:

[0110]

[0111] Where and represent the Fourier transforms of τ(t; s) and τ SPAD (t; s) respectively, and v is the frequency. The Wiener filtering kernel in the frequency domain can be calculated by the Fourier transform of the time jitter and the signal-to-noise ratio η:

[0112]

[0113] Scholars such as O’Toole embedded Wiener filtering in the LCT algorithm, while Velten et al. applied Laplace filtering to the results of the backprojection algorithm for noise reduction. Different from them, this application uses the noise reduction algorithm before the reconstruction algorithm, so that different reconstruction algorithms can all be applied to the denoised data. Like these studies, this application ignores the role of the Poisson distribution in the noise, while the optimization algorithm of this application considers the impact of this on NLOS reconstruction.

[0114] Step S203: Process the hidden transient component based on the optimization algorithm of the confocal imaging model, and use the gradient descent method to perform on-site reconstruction of the non-line-of-sight scene of the hidden object.

[0115] Many algorithms for reconstructing hidden objects have been proposed. Voxel-based algorithms, such as the light cone transform (LCT), f-k migration, and phasor field methods, assume that the hidden scene is represented as 3D voxels and use the fast Fourier transform (FFT) to solve the inverse problem and reconstruct the scene. These algorithms require regular-form transients as input. On the other hand, optimization-based and learning-based algorithms (such as the Neural Transient Field, NeTF) support transient input for arbitrary scanning patterns, while the latter requires dozens of hours of training.

[0116] This application implements an optimization algorithm based on a confocal imaging model. Like most existing methods, this algorithm assumes that the hidden object is fully diffusely reflective, there is no occlusion between the hidden object and the relay surface, and mutual reflections between the hidden object surfaces are not considered. Based on these assumptions, the attenuation term g(p; s) in Equation (4) can be regarded as the bidirectional reflectance distribution function (BRDF) term f(p; ω p→s ) is independent of the viewing angle and can be written as f(p). Equation (4) can be simplified to a linear model

[0117] τ = Ψf (24)

[0118] where τ is the discretized transient measurement, Ψ is the linear measurement matrix containing g(p; s). The reflectivity f of the object can be solved by an optimization method that minimizes the difference between Ψf and τ. This application uses the Poisson likelihood function to evaluate the similarity. The loss function also includes the total variation (TV) as a regularization term.

[0119]

[0120] Here, e i is the i-th standard unit vector, τ i is the i-th element of τ, and λ is the weight of the total variation term. This application uses the gradient descent method to implement the optimization algorithm.

[0121] Since this application supports both uniform and non-uniform scanning patterns, this application inherits existing non-line-of-sight reconstruction algorithms to process hidden components based on spatial, frequency, or learning-based techniques. Many fast Fourier transform (FFT)-based algorithms, such as LCT, FK, and PF, require transient signals in a regular grid as input; while learning-based methods, such as the Neural Transient Field (NeTF), allow more flexible sampling patterns but require longer computation times to train the network.

[0122] Qualitative evaluation: As Figure 5As shown, it shows the calibration verification of this application under three different scanning methods (random scanning, regular grid, and concentric circles), so as to verify the accuracy of the online calibration of this application. Among them, a is the random method. Among them, the dots in a represent the scanning points.

[0123] For the random scanning mode, this application randomly scans several points in different directions, determines the coordinates of all detection points s, and the relay surface equation. Then, the corresponding new coordinates s are calculated according to the estimated relay surface equation. W And the input voltage. Using these voltages, the galvanometer scans again on the relay surface. Figure 5 Figure a shows two groups of detection points before and after: the squares represent the target detection points s, and the dots represent the reprojection points s. W .

[0124]

[0125] The system described in this application is also adapted to the uniform scanning mode. Scan N×N detection points within the range of L×L, and the coordinates of the detection point s(i,j) can be expressed as:

[0126]

[0127] Inspired by the circular scanning mode, this application further proposes a concentric circle scanning mode. In the scanning area with a radius of R, N r points are defined on each of the N φ concentric circles. The detection point s(i,j) can be expressed as:

[0128]

[0129] Similarly, 32 detection points are scanned on 4 circles, that is, 8 dots on each circle.

[0130] Figure 5 The results of the scanning modes in show that the scanning points, whether regular or irregular, equally spaced or unequally spaced, are highly consistent with the required positions.

[0131] Quantitative evaluation: In addition, this application quantitatively verified the calibration accuracy by changing the scanning points and the input voltage range. Table 1 shows the details of the calibration experiment.

[0132] Table 1. Quantitative verification of calibration accuracy based on root mean square error

[0133]

[0134] This application selects three sets of regular scanning points while keeping the input voltage within the same range. This application calculates the 3D coordinates of these scanning points and estimates the relay plane. Then, 4×4 scanning points are randomly selected within the specified voltage range. The distance from the coordinates of the new scanning points to the estimated relay plane is calculated. For a laser speckle with a diameter of 4 mm and a distance from the system origin to the relay plane of 1500 mm, the root mean square error of the distance is about 9 mm, and the calibration accuracy is about 9 mm.

[0135] Similarly, this application selects three sets of input voltage ranges with the same number of scanning points (5×5), and estimates the 3D coordinates of the scanning points and the relay plane. The root mean square error (RMSE) of the distance indicates that the calibration accuracy is about 7 mm. For NLOS imaging and reconstruction, the calibration error is small enough.

[0136] In some embodiments, this application further verifies this acquisition system and calibration technique through the reconstruction results of simulation data and measured data. The state-of-the-art (SOTA) methods (including LCT, FK, and PF) and the optimized algorithm of this application are used. Finally, an ablation study is conducted on how the calibration error affects the NLOS reconstruction quality through simulated and measured hidden scenes.

[0137] Simulation transient: This application develops an algorithm for generating simulation data based on a confocal imaging model. The illumination points on the relay plane match the detection points, and the light scatters towards the hidden object in the form of spherical waves. It is assumed that the luminance map L(u, v; s) and depth map D(u, v; s) are obtained at the detection point s, where u and v represent the corresponding pixels in the map. Then, the luminance and depth of each pixel are extracted, and the transient τ(t; s) is calculated at the corresponding time t. This process is carried out for each detection point.

[0138] This application renders the transient data τ(t; s) for a whiteboard of 0.6 m×0.6 m 'S' and 0.6 m×0.4 m with a time resolution of 4 ps and a spatial resolution of 64×64. Considering the time jitter, background count, and Poisson distribution characteristics in formula (6), noise is also added to the simulation data of the 'S' and the whiteboard.

[0139] Measured Transients: The present application uses a fast-gated SPAD from Micro Photon Devises, which uses a time-correlated single-photon counter PicoHarp 300 to provide a time resolution of 4 ps. The return light is focused onto the SPAD using an achromatic lens (Canon EF 50mm f / 1.8). A laser (SuperK EXTREME FUI-15) with a tunable wavelength filter (SuperK VARIA) emits collimated light through a polarization beam splitter cube (Thorlabs VA5-PBS251). Then, the light is guided by a 2D galvanometer (Thorlabs GVS012) controlled by a data acquisition device. The SPAD records the indirect light from the NLOS scene by gating the direct light using a delay line (PicoQuant PSD-065-A-MOD) with a gate width of 12 ns. The time jitter of the entire system is approximately 200 ps.

[0140] The hardware device of the present application is located 1.0 m away from the relay wall, which is a white melamine panel, and its position and orientation can be adjusted for NLOS objects of different sizes and shapes. The transients of the whiteboard and 'S' were measured, and their settings were similar to those in the simulation. Compared with the simulated transients, due to the characteristics of the SPAD, such as dark counts and pile-up effects, the smaller peaks of the measured transients decreased. Note that the denoised transients still contain noise from factors such as Poisson distribution and pile-up effects. Other NLOS scenarios include a chessboard, a mannequin, and a resolution board designed with stripes of different widths and lengths. All the measured objects are diffuse reflection materials of paper, cotton, or wood. In the system of the present application, transient data is automatically collected, and most experiments require 2.5 s to record the measurement results of a single detection point and transition to the next detection point.

[0141] In some embodiments, the present application uses simulated and measured transient data for experiments. The source code is in MATLAB language and includes publicly available LCT, FK, and PF algorithms, which are run on a personal computer equipped with an Intel i7-8750H CPU (2.2 GHz), 16 GB RAM, and a GPU 1050Ti.

[0142] Figure 6 The 'S' reconstructed using the LCT, FK, and PF algorithms is shown. Among them, (a) and (c) are noise-free and noisy simulated data, respectively, and (b) and (d) are original and denoised measured data. The original and denoised (original and denoised) measured data and (noise-free and noisy) simulated data are all measured in a regular grid manner. As expected, the 'S' was well reconstructed from the transients, and the data enhancement method of the present application can improve the reconstruction quality.

[0143] This application also measures checkerboards, resolution targets, and dolls using a regular grid pattern and reconstructs them using the normalized transient data. Figure 7 The reconstruction verification of the normalized transient data in the regular grid pattern is shown. From top to bottom, they are the checkerboard, resolution target, and doll models. The SOTA algorithm and the OPT algorithm are used to reconstruct the hidden objects. The checkerboard and resolution target match the photos or reference standards, and the algorithm has a great impact on the reconstruction quality. The SOTA algorithm takes several seconds to reconstruct each NLOS scene, while the optimized algorithm of this application takes several minutes to perform 1000 iterations.

[0144] Ablation experiment: The calibration error accumulates gradually during the calibration process of the entire acquisition system. For example, galvanometer calibration, relay plane calibration, and determination of scanning points will affect the reconstruction quality of the non-line-of-sight scene. Here, this application verifies how the error of relay plane calibration affects NLOS reconstruction. This application simulates an accurate relay plane (error of 0°), and then this application considers verifying the cases where the error ranges from 2.5° to 10°, and reconstructs the hidden rabbit through the mis-calibrated relay plane. Similarly, in the actual measurement system, this application performs scanning, data processing, and reconstruction at different deflection angles.

[0145] Figure 8 The reconstruction results of the rabbit and checkerboard and the calibration error are shown for deflections from 2.5° (2.0°) to 10.0° (20.0°). The reconstruction quality of the NLOS scene deteriorates rapidly with the increase of the relay plane calibration error.

[0146] In summary, this application proposes an on-site non-line-of-sight imaging method based on online calibration, which enables users to (re)-adjust the scanning area and mode of the detection points through Gamma map preview after online calibration of the galvanometer and relay plane. This application can define any scanning mode to accurately measure different hidden scenes and support the input transient of any scanning mode for NLOS reconstruction. A large number of experiments have demonstrated the efficiency and efficacy of the NLOS imaging solution of this application.

[0147] As Figure 9 shown, it is a schematic diagram of the modules of an electronic device in an embodiment of this application. As shown in the figure, the device 1000 includes:

[0148] A decoupling module 1001, configured to decouple the transient data collected on-site by the acquisition system into a visible-field transient component and a hidden transient component based on the principle of confocal imaging;

[0149] A calibration module 1002, configured to calculate a Gamma map based on the visible-field transient component and re-perform online calibration and calibration of the acquisition system when at least one of the scene, relay plane setting, scanning area, and scanning mode changes;

[0150] The reconstruction module 1003 is used to process the hidden transient components based on the optimization algorithm of the confocal imaging model, and the gradient descent method is used to reconstruct the hidden object in the non-visible field scene on-site.

[0151] It should be noted that for the information interaction, execution process, etc. among the above-mentioned device modules / units, since they are based on the same concept as the method embodiments of the present application, the technical effects brought by them are the same as those of the method embodiments of the present application. For the specific content, reference can be made to the descriptions in the method embodiments shown above in the present application, and details will not be repeated here.

[0152] It should also be noted that it should be understood that the division of each module of the above device is only a logical function division. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. And these units can all be implemented in the form of software called by a processing element; they can also all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the reconstruction module 1003 can be a separately established processing element, or can be integrated in a certain chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and the function of the above reconstruction module 1003 can be called and executed by a certain processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or can be independently implemented. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the integrated logic circuit in the processor element in hardware or in the form of instructions in software.

[0153] For example, the above modules can be one or more integrated circuits configured to implement the above method, such as: one or more application specific integrated circuits (ASICs), or, one or more digital signal processors (DSPs), or, one or more field programmable gate arrays (FPGAs), etc. Again, when a certain module above is implemented in the form of a program code scheduled by a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program code. Again, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0154] Such as Figure 10As shown, it is a schematic structural diagram of a computer device in an embodiment of the present application. As shown in the figure, the computer device 1100 includes: a memory 1101 and a processor 1102; the memory 1101 is used to store computer instructions; the processor 1102 runs the computer instructions to implement as Figure 2 the method described above.

[0155] In some embodiments, the number of the memories 1101 in the computer device 1100 can be one or more, and the number of the processors 1102 can also be one or more, and Figure 10 one of each is taken as an example herein.

[0156] In an embodiment of the present application, the processor 1102 in the computer device 1100 will, according to the steps as Figure 1 described above, load instructions corresponding to one or more application program processes into the memory 1101, and the processor 1102 runs the application programs stored in the memory 1101, so as to implement as Figure 2 the method described above.

[0157] The memory 1101 may include a random access memory (Random Access Memory, abbreviated as RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 1101 stores an operating system and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations. The operating system may include various system programs for implementing various basic services and processing hardware-based tasks.

[0158] The processor 1102 may be a general-purpose processor, including a central processing unit (Central Processing Unit, abbreviated as CPU), a network processor (Network Processor, abbreviated as NP), etc.; it may also be a digital signal processor (Digital Signal Processing, abbreviated as DSP), an application specific integrated circuit (Application Specific Integrated Circuit, abbreviated as ASIC), a field programmable gate array (Field-Programmable Gate Array, abbreviated as FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0159] In some specific applications, the components of the computer device 1100 are coupled together through a bus system, which may include a power bus, a control bus, a status signal bus, etc. in addition to the data bus. However, for the sake of clarity, in Figure 10 all kinds of buses are referred to as the bus system.

[0160] In an embodiment of the present application, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the method as Figure 2 described.

[0161] At any possible technical detail combination level, the present application can be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium, on which computer-readable program instructions for causing a processor to implement various aspects of the present application are loaded.

[0162] The computer-readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. The computer-readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punched card or raised structures in a groove storing instructions thereon, and any suitable combination of the above. The computer-readable storage medium used herein is not construed as an instantaneous signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0163] The computer-readable programs described herein can be downloaded from the computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. The network adapter or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0164] The computer program instructions for performing the operations of the present application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by using an Internet service provider to connect through the Internet). In some embodiments, by using the status information of the computer-readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute the computer-readable program instructions to implement various aspects of the present application.

[0165] In summary, a method, apparatus, device, and medium for on-site non-line-of-sight imaging based on online calibration provided by the present application decouple the transient data collected on-site by the acquisition system into a visible-field transient component and a hidden transient component based on the confocal imaging principle; calculate the Gamma map based on the visible-field transient component, and re-perform online calibration and calibration on the acquisition system when at least one of the scene, relay surface setting, scanning area, and scanning mode changes; process the hidden transient component by an optimization algorithm based on the confocal imaging model, and use the gradient descent method to perform on-site reconstruction of the non-visible-field scene of the hidden object.

[0166] The present application effectively overcomes various disadvantages in the prior art and has high industrial utilization value.

[0167] The above embodiments are only illustrative of the principles and effects of the present application and are not used to limit the present invention. Any person familiar with this technology may modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes completed by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by the present invention should still be covered by the claims of the present application.

Claims

1. An on-site non-line-of-sight imaging method based on online calibration, characterized in that The method includes: Decoupling the transient data collected on-site by the acquisition system into a visible-field transient component and a hidden transient component based on the principle of confocal imaging; Calculating a Gamma map based on the visible-field transient component, and re-performing on-line calibration and calibration of the acquisition system when at least one of the scene, relay surface setting, scanning area, and scanning mode changes; Processing the hidden transient component using an optimization algorithm based on the confocal imaging model, and performing on-site reconstruction of the non-visible-field scene of the hidden object using the gradient descent method; The transient data includes: τ(t;o,l,s,d) = Γ(s) ∫ P δ(t - (t o→l + t l→p + t p→s + t s→d )) f(p;ω l →p , ω p→s ) g(p;l, s) dA p ; Among them, τ represents transient data; t represents time; o represents the illumination point of the light source; l represents the spot position on the relay surface irradiated by the light source; s represents the reflection position on the relay surface after the light source is scattered from the relay surface and reflected back by the reflection point p on the hidden object P; d represents the detection point of the detector for detecting the number of returned photons; Γ(s) represents the intensity change coefficient after the light source is reflected on the relay surface and can describe the detector's field of view; the integral ∫ P δ represents the total number of photons reflected from the microfacet A centered on the reflection point p p The sum of the number of photons; → represents the light source path; the unit vector represents the direction vector from l to p; the unit vector represents the direction vector from p to s; f represents the incident direction ω corresponding to the reflection point p l→p and the outgoing direction ω p→s bidirectional reflection function; the function g represents an attenuation term that includes distance, light and shadow, and occlusion effects; N o represents the total number of photons emitted by the light source in one pulse from the illumination point o; ρ represents the reflectivity of the relay surface; A d represents the effective area of the detector; A s represents the detection area where the detector is projected onto the relay surface; n s represents the surface normal vector at the reflection position s.

2. The method according to claim 1, wherein The acquisition system includes: a hardware module, a LOS module, and an NLOS module; The hardware module includes: a pulsed laser, a SPAD as a detector, a beam splitter, a galvanometer, and a controller; The LOS module is used to associate the hardware module with the relay surface and corresponds to the transient LOS component; Among them, the relay surface, scanning mode, and measurable bounding box are defined using the LOS module to detect hidden objects located within the bounding box; The NLOS module is used to collect spherical light waves propagating between the relay surface and the hidden object and contains information about the hidden object for reconstruction.

3. The method according to claim 1, characterized in that, The method includes: Modeling the data recorded by the SPAD as: τ SPAD = Pois(τ*j + b); j(t; μ, σ, κ0, κ1, γ) = Gaus(t, μ, σ) + γExp(t; μ, κ0, κ1); Where, SPAD represents a single-photon avalanche diode as a detector; τ represents transient data; Pois represents the Poisson likelihood function; j represents the time jitter of the entire system, and the time jitter includes two parts: the Gaussian peak represented by Gaus and the exponential tail represented by Exp(t; μ, κ0, κ1); μ, σ, κ0, κ1 represent the relevant parameters of the time jitter; γ represents the weight of the exponential tail; b represents the SPAD background noise; Calculate the time jitter of multiple detection points and take the average as the time jitter of the acquisition system.

4. The method according to claim 3, wherein The method includes: Re-aligning the transient data of each detection point so that the arrival time of the photons directly reflected by the relay surface is at zero time; Using the Gamma map to normalize the transient data τ(t; s) to enhance the acquisition quality; Apply Wiener filtering to the transient data τ SPAD for noise reduction: Among them, and represent the Fourier transforms of τ(t; s) and τ SPAD (t; s) respectively; v is the frequency; the Wiener filtering kernel in the frequency domain is calculated by the Fourier transform of time jitter and the signal-to-noise ratio η:

5. The method according to claim 1, characterized in that The decoupling of the transient data collected on-site by the acquisition system into a visible-field transient component and a hidden transient component includes: When the irradiation point o and the detection point d coincide to form a confocal, the transient data is simplified to: τ(t;o,s) = Γ(s)∫ P δ(t - 2(t o→s + t s→p ))f(p;ω p→s )g(p;s)dA p ; τ hidden (t;s) = ∫ P δ(t - 2t s→p )f(p; ω p→s )g(p; s)dA p ; τ LOS (t; o, s) = Γ(s)δ(t - 2t o→s ) Among them, τ(t; o, s) includes: τ LOS (t; o, s) the visible field transient component along the direct optical path between the relay surface and the detector, τ hidden (t; s) the hidden transient component along the indirect optical path between the relay surface and the hidden object.

6. The method according to claim 5, characterized in that, The processing of the hidden transient component using an optimization algorithm based on the confocal imaging model and performing on-site reconstruction of the non-visible-field scene of the hidden object using the gradient descent method includes: Based on the fact that the hidden object P is completely diffusely reflective, there is no occlusion between the hidden object P and the relay surface, and mutual reflection between the surfaces of the hidden object P is not considered, the attenuation term g(p; s) can be regarded as The bidirectional reflection function f(p; ω p→s ) is independent of the viewing angle and can be written as f(p), τ hidden (t; s) can be simplified to: τ = Ψf; Where, τ represents transient data, Ψ represents a linear measurement matrix containing g(p; s), and f represents the reflectivity; Among them, NLOS represents the non-line-of-sight region; represents the loss function; e i represents the i-th standard unit vector, τ i represents the i-th element of τ, and λ is the weight of the total variation regularization term; TV represents the total variation regularization term; The gradient descent method is used to achieve on-site reconstruction of the non-visible-field scene of the hidden object.

7. The method according to claim 1, wherein The calculation of the Gamma map based on the visible-field transient component includes: When the detection point d is fixed, the intensity change coefficient Γ(s) of the light source after reflection on the relay surface is defined by determining the coordinates of the reflection position s on the relay surface; based on confocal imaging, the integral of the δ function is 1, and Γ(s) is expressed as: Γ(s) = ∫τ LOS (t; o, s) dt.

8. The method according to claim 2, wherein When at least one of the scene, relay plane setting, scanning area, and scanning mode changes, re - perform on - line calibration and calibration of the acquisition system, including: Adjust the optical path by observing the light spot on the relay plane; Adjust the beam splitter and galvanometer so that the light spot on the relay plane is clearly visible; Fine - tune the position of the SPAD so that the received primary reflected light from the relay plane reaches the maximum.

9. The method according to claim 2, wherein When at least one of the scene, relay plane setting, scanning area, and scanning mode changes, re - perform on - line calibration and calibration of the acquisition system, including: Assume the scanning system is linear and perform on - line calibration of the galvanometer; For the optical deflection angle θ X , θ Y and the input voltage V X , V Y to establish a model of their relationship: wherein, the initial deflection angle ∈ X and ∈ Y are determined by the offset of the two mirrors without input voltage; the β matrix represents the coefficients of a pair of angles with respect to a pair of input voltages; The coefficient of β is optimized using a multiple linear regression algorithm with a loss function corresponding to the galvanometer ; The average error between the measured and calculated optical scanning angles is taken as ∈ X and ∈ Y ; Calculate the input voltage for each set of scanning angles to control the scanning points on any surface and define the scannable area of the galvanometer.

10. The method according to claim 5, wherein When at least one of the scene, relay plane setting, scanning area, and scanning mode changes, re - perform on - line calibration and calibration of the acquisition system, including: Based on the transient component τ of the visible field LOS Reconstruct the reflectivity and surface normal vector of the relay surface; Extract τ at each detection point LOS (t; s) and its corresponding t, and calculate the depth l of the relay surface; Optical deflection angle θ of the galvanometer X , θ Y and depth to estimate the three-dimensional coordinates of the detection point in XYZ: Among them, the relay surface is regarded as a plane W (W X , W Y , W Z ), and the equation is: W:W X X + W Y Y + W Z Z + 1 = 0; By minimizing the loss function The root mean square error of the distance from a point to a plane is used to solve for the plane W of the relay surface (W X ,W Y ,W Z ):

11. The method according to claim 10, wherein When at least one of the scene, relay plane setting, scanning area, and scanning mode changes, re - perform on - line calibration and calibration of the acquisition system, including: Implement any scanning mode by calculating the input voltage corresponding to the coordinates of each detection point on the relay plane; Define a scanning area, with the origin o s Located at the center of the relay plane, through an orthogonal basis Construct a new set of orthogonal bases Among them, is the unit normal vector of the relay plane; for the plane relay plane z = 0, the detection points can be respectively expressed in the two coordinate systems as follows: Calculate the scanning angle of each detection point on the relay plane and then obtain the galvanometer input voltage.

12. The method according to claim 1, characterized in that, When at least one of the scene, relay plane setting, scanning area, and scanning mode changes, re - perform on - line calibration and calibration of the acquisition system, including: Create a free space by using a relay plane with adjustable direction and position to allow larger hidden objects; The orthogonal projection of the scanning range is used to limit the measurable bounding box of the NLOS scene with the maximum width and height of the scanning area. The minimum depth of the bounding box is approximately: z min = ct delay ; where z represents the depth of the bounding box; t delay represents the delay in the arrival time of photons returning from the relay surface and the hidden object, and c represents the speed of light; The photon attenuation between the hidden object and the relay plane determines the maximum depth of the bounding box; Among them, the minimum value Γ of the intensity change coefficient min is used to limit the scanning range and the volume of the bounding box; b represents the SPAD background noise.

13. An electronic device, characterized in that, Apply the on - site non - line - of - sight imaging method based on on - line calibration as described in any one of claims 1 to 12; The device includes: A decoupling module for decoupling the transient data collected on - site by the acquisition system into a visible - field transient component and a hidden transient component based on the confocal imaging principle; A calibration module for calculating a Gamma map based on the visible - field transient component and re - performing on - line calibration and calibration of the acquisition system when at least one of the scene, relay plane setting, scanning area, and scanning mode changes; A reconstruction module for processing the hidden transient component based on the optimization algorithm of the confocal imaging model and performing on - site reconstruction of the non - visible - field scene of the hidden object using the gradient descent method.

14. A computer device, characterized in that, The device includes: a memory and a processor; The memory is used to store computer instructions; The processor runs the computer instructions to implement the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, Store computer instructions, and when the computer instructions are run, execute the method as described in any one of claims 1 to 12.

Images