Two-stage active steady-state non-line-of-sight imaging method based on data-model driven

Abstract

The present invention discloses a two-stage active steady-state non-line-of-sight imaging method based on data-model driving, including: building an active steady-state non-line-of-sight imaging system; obtaining non-line-of-sight signals and mirror reflection signals reflected by the imaging target scene and constructing a training data set; constructing an imaging conversion model from non-line-of-sight signals to mirror reflection signals; constructing a first-stage transmission matrix estimation neural network, iteratively optimizing it using the training data set to obtain the transmission matrix; constructing a second-stage imaging scene estimation neural network, given the non-line-of-sight signal of the hidden scene to be imaged, iteratively optimizing it with the participation of the transmission matrix, and obtaining the mirror reflection estimation value. The hardware required by the present invention is simple in composition and low in cost, and only requires an ordinary continuous laser source and a steady-state optical sensor as an excitation and detection device; it has strong adaptability and is not limited by the ambient lighting or the self-luminous properties of the object; it has good generalization, and effective non-line-of-sight reconstruction can be achieved without relying on a large amount of data training.

Description

Technical Field

[0001] The present invention relates to the fields of computational photography and non-line-of-sight imaging, and in particular to a two-stage active steady-state non-line-of-sight imaging method based on data-model driving. Background Art

[0002] As an emerging optical sensing technology, non-line-of-sight imaging has attracted widespread attention in recent years because it can break the limitations of traditional line-of-sight imaging and detect scenes hidden in blind spots or corners. This technology is expected to be applied to autonomous driving, military reconnaissance, medical examination, industrial inspection and other fields, and has a bright future. Existing non-line-of-sight imaging settings can be divided into active non-line-of-sight imaging and passive non-line-of-sight imaging according to whether external optical excitation is provided.

[0003] Passive non-line-of-sight imaging usually relies on ambient lighting or scene self-luminescence, and is therefore relatively easy to deploy and implement. Existing passive non-line-of-sight imaging methods mainly utilize the intensity information of scene reflections, and the implementation methods mainly include reducing the condition number of the optical transmission matrix by introducing occlusions or learning the inverse mapping of light transmission through neural networks with the support of large-scale data. However, due to the special requirements for light sources, the application scenarios of passive non-line-of-sight imaging are limited, especially when the scene is dim or the object itself does not have self-luminous properties.

[0004] In contrast, active non-line-of-sight imaging directly illuminates the relay surface through the controllable optical excitation provided, and indirectly illuminates the hidden scene through the reflection of the relay surface, effectively improving the flexibility of the system and its adaptability to different scenes. According to the temporal resolution of the imaging system, the existing active non-line-of-sight imaging technology can be mainly divided into transient non-line-of-sight imaging and steady-state non-line-of-sight imaging.

[0005] Transient non-line-of-sight imaging usually uses a laser with an extremely short pulse width to scan the relay surface and capture the scene reflection information, especially the flight time information of photons, through a highly sensitive detector. Transient non-line-of-sight imaging has high requirements on the temporal resolution of the excitation and sensing equipment, which usually needs to reach the picosecond or even femtosecond level. Existing transient non-line-of-sight imaging methods mainly include ellipsoidal tomography, light field modulation, wave-based and learning-based methods. Although these methods have made important contributions to improving the quality of three-dimensional reconstruction of non-line-of-sight scenes, transient non-line-of-sight imaging still has some limitations, such as high cost, high system complexity, low photon efficiency, and long sampling time. These limitations affect the deployment and practicality of transient non-line-of-sight imaging systems.

[0006] Steady-state non-line-of-sight imaging has gradually gained favor among researchers due to its many advantages such as low cost, easy deployment, high efficiency and high stability. The system is simple in composition and usually only requires a continuous active light source and a steady-state detector. Existing steady-state non-line-of-sight imaging methods mainly use the intensity distribution or speckle information generated by active illumination on the relay wall after being reflected by the hidden scene to reconstruct the hidden scene. At present, learning-based methods have become one of the most popular methods in the field of steady-state non-line-of-sight imaging. Many methods train a deep model end-to-end in a data-driven manner to establish a direct mapping from optical measurement to hidden scenes. Although these methods are efficient and flexible, they usually rely on large-scale data sets to achieve good imaging effects and generalization performance. The establishment of large-scale data sets often requires high funds and a large amount of manpower investment, which is costly, cumbersome and time-consuming. Therefore, how to use a small amount of data to achieve effective non-line-of-sight reconstruction is an important problem that needs to be solved in the current field of active steady-state non-line-of-sight imaging. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the prior art and propose a two-stage active steady-state non-line-of-sight imaging method based on data-model driving.

[0008] The technical solution adopted by the present invention is as follows:

[0009] A two-stage active steady-state non-line-of-sight imaging method based on data-model driving includes the following steps:

[0010] S1. Build an active steady-state non-line-of-sight imaging system;

[0011] S2. using an active steady-state non-line-of-sight imaging system to obtain a non-line-of-sight signal and a mirror reflection signal reflected by an imaging target scene, and preprocessing the non-line-of-sight signal and the mirror reflection signal to obtain a training data set;

[0012] S3. Construct an imaging conversion model from non-line-of-sight signals to specular reflection signals. The formula is:

[0013] ,

[0014] in, is the denoised and vectorized non-viewing signal, is the overall transmission matrix, is the denoised and quantized specular reflection signal;

[0015] S4. According to the imaging conversion model, a one-stage transfer matrix estimation neural network is constructed, and the one-stage transfer matrix estimation neural network is iteratively optimized using the training data set to obtain the estimated transfer matrix ;

[0016] S5. Construct a two-stage imaging scene estimation neural network based on the imaging conversion model, given the non-viewing signal of the hidden scene to be imaged , in the estimated transmission matrix The two-stage imaging scene estimation neural network is iteratively optimized with the participation of to obtain the estimated value of the mirror reflection of the hidden scene to be imaged. , thereby achieving non-line-of-sight imaging of hidden scenes.

[0017] Further, the active steady-state non-line-of-sight imaging system includes an imaging target scene, an obstruction, an obstacle, a relay surface, a continuous laser source, and a steady-state optical sensor;

[0018] The continuous laser source and the steady-state optical sensor are located on the same side and are blocked by obstacles from the imaging target scene; the relay surface is parallel to the plane where the imaging target scene is located; an obstruction is provided between the relay surface and the imaging target scene; the imaging target scene includes a movable target, and the movable target can move freely within the range of the imaging target scene;

[0019] The illumination position of the laser emitted by the continuous laser source on the relay surface satisfies that the reflected light field completely covers the imaging target scene and is not hindered by any obstruction.

[0020] Furthermore, the laser emitted by the continuous laser source illuminates the imaging target scene through diffuse reflection of the relay surface, and undergoes secondary diffuse reflection on the imaging target scene. Part of the laser after the secondary diffuse reflection returns to the relay surface and is received by the steady-state optical sensor after a third reflection on the relay surface.

[0021] Furthermore, the non-field of view signal is: a signal received by the steady-state optical sensor when an obstruction exists and the relay surface where the third reflection occurs is a diffuse reflection surface; the mirror reflection signal is: a signal received by the steady-state optical sensor when the obstruction is removed and the relay surface where the third reflection occurs is a mirror surface.

[0022] Furthermore, the non-line-of-sight signal and the mirror reflection signal are preprocessed, and the specific steps are: collecting the signal when there is no movable target in the active steady-state non-line-of-sight imaging system as the background signal, and differentiating the non-line-of-sight signal and the mirror reflection signal with their respective background signals to obtain a denoised non-line-of-sight signal or a denoised mirror reflection signal.

[0023] Furthermore, the imaging conversion model of the non-viewing area signal to the specular reflection signal is constructed by the following specific steps:

[0024] The imaging conversion model is constructed based on the discretized transmission equation:

[0025] ,

[0026] in, Represents the vectorized non-viewing signal including noise, with a size of ; It represents the reflection intensity of the imaging target scene after receiving indirect illumination from the active light source, and its magnitude is ; represents the vectorized noise term, with a size of ; represents the light transmission matrix under this condition, and its size is ;

[0027] In direct access In difficult cases, the specular reflection intensity of the imaging target scene is indirectly obtained through specular reflection:

[0028] ,

[0029] in, Represents the vectorized mirror reflection signal, which is , Represents the conversion coefficient vector, size is , represents the Hadamard product;

[0030] Substituting the above formula into the discretized transmission equation and transforming it, it can be expressed as:

[0031] ,

[0032] in, Indicates size A vector of all 1s, express The transpose of represents the Kronecker product;

[0033] The above formula can be simplified as:

[0034] ,

[0035] in, Represents the denoised non-viewing signal, which is , represents the overall transmission matrix, , the size is .

[0036] Furthermore, the input of the first-stage transmission matrix estimation neural network is a random noise matrix, and the output is an estimated transmission matrix , the estimated transmission matrix The specular reflection signal in the training dataset Perform matrix multiplication and denoise the non-viewing signal Under the supervision of The generated transmission matrix Make optimizations.

[0037] Furthermore, the input of the two-stage imaging scene estimation neural network is a random noise matrix, and the output is the estimated value of the mirror reflection of the hidden scene to be imaged. , the estimated value of the specular reflection of the hidden scene to be imaged With the estimated transmission matrix Perform matrix multiplication to obtain the non-viewing signal of the hidden scene to be imaged. Under the supervision of Estimated specular reflection of the hidden scene to be imaged Make optimizations.

[0038] Compared with the prior art, the beneficial effects of the present invention are mainly reflected in:

[0039] (1) Compared with passive non-line-of-sight imaging settings, the present invention is not limited by ambient lighting or the self-luminous properties of objects, has a wide range of applications and strong adaptability; compared with active transient non-line-of-sight imaging settings, it does not rely on the time-resolved detection system to provide additional time-of-flight information; the composition structure is simple and does not require a complex calibration process, and is efficient, flexible and easy to implement and deploy.

[0040] (2) The present invention does not require explicit physical modeling of the non-viewing light field transmission model. By combining a small amount of data drive and physical constraints to compensate for the uncertainty of traditional explicit physical modeling, it can adapt to more complex and changeable scenes.

[0041] (3) Compared with the traditional end-to-end data-driven method, the present invention can achieve effective non-field of view reconstruction of hidden plane targets without relying on large-scale sample training. In addition, due to the incorporation of physical constraints, the reconstruction quality is higher and the generalization performance is better than that of the method based solely on data-driven. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the configuration of an active steady-state non-line-of-sight imaging system provided in an embodiment of the present invention.

[0043] Figure 2 A schematic top view and side view of the geometric configuration of an active steady-state non-line-of-sight imaging system provided by an embodiment of the present invention.

[0044] Figure 3 A schematic diagram of the settings for collecting active steady-state non-line-of-sight signals and their corresponding mirror reflection signals provided by an embodiment of the present invention.

[0045] Figure 4A schematic diagram of a one-stage transmission matrix calculation framework provided in an embodiment of the present invention.

[0046] Figure 5 A schematic diagram of a two-stage imaging scene calculation framework provided in an embodiment of the present invention.

[0047] Figure 6 A schematic diagram of training examples and test examples provided by an embodiment of the present invention.

[0048] Figure 7 A convergence curve diagram of a one-stage loss function provided in an embodiment of the present invention.

[0049] Figure 8 This is the transmission matrix estimation result provided by the embodiment of the present invention.

[0050] Fig. 9 A convergence curve diagram of the two-stage loss function provided in an embodiment of the present invention.

[0051] Fig.10 This is a diagram of non-field of view imaging effects of different methods provided in embodiments of the present invention. DETAILED DESCRIPTION

[0052] The technical solution of the present invention will be further explained in detail below in conjunction with the accompanying drawings and specific embodiments.

[0053] A two-stage active steady-state non-line-of-sight imaging method based on data-model driving includes the following steps:

[0054] S1. Build an active steady-state non-line-of-sight imaging system.

[0055] The active steady-state non-line-of-sight imaging system includes an imaging target scene, an obstruction, an obstacle, a relay surface, a continuous laser source and a steady-state optical sensor; the continuous laser source and the steady-state optical sensor are located on the same side and are blocked by an obstacle between them and the imaging target scene; the relay surface is parallel to the plane where the imaging target scene is located; an obstruction is arranged between the relay surface and the imaging target scene; the imaging target scene includes a movable target, and the movable target can move freely within the range of the imaging target scene.

[0056] The illumination position of the laser emitted by the continuous laser source on the relay surface satisfies that the reflected light field completely covers the imaging target scene and is not hindered by any obstruction.

[0057] The laser emitted by the continuous laser source illuminates the imaging target scene through diffuse reflection of the relay surface, and undergoes secondary diffuse reflection on the imaging target scene. Part of the laser after the secondary diffuse reflection returns to the relay surface and is received by the steady-state optical sensor after a third reflection on the relay surface.

[0058] S2. Use an active steady-state non-line-of-sight imaging system to obtain non-line-of-sight signals and mirror reflection signals reflected by the imaging target scene, pre-process the non-line-of-sight signals and mirror reflection signals, and obtain a training data set.

[0059] The non-field of view signal is: the signal received by the steady-state optical sensor when an obstruction exists and the relay surface where the third reflection occurs is a diffuse reflection surface; the mirror reflection signal is: the signal received by the steady-state optical sensor when the obstruction is removed and the relay surface where the third reflection occurs is a mirror surface.

[0060] The non-line-of-sight signal and the mirror reflection signal are preprocessed, and the specific steps are: collecting the signal when there is no movable target in the active steady-state non-line-of-sight imaging system as the background signal, and differentiating the non-line-of-sight signal and the mirror reflection signal with their respective background signals to obtain a denoised non-line-of-sight signal or a denoised mirror reflection signal.

[0061] S3. Construct an imaging conversion model from non-line-of-sight signals to specular reflection signals. The formula is:

[0062] ,

[0063] in, is the denoised and vectorized non-viewing signal, is the overall transmission matrix, is the denoised and quantized specular reflection signal.

[0064] The imaging conversion model from the non-viewing area signal to the specular reflection signal is constructed in the following specific steps:

[0065] The imaging conversion model is constructed based on the discretized transmission equation:

[0066] ,

[0067] in, Represents the vectorized non-viewing signal including noise, with a size of ; It represents the reflection intensity of the imaging target scene after receiving indirect illumination from the active light source, and its magnitude is ; represents the vectorized noise term, with a size of ; represents the light transmission matrix under this condition, and its size is ;

[0068] In direct access In difficult cases, the specular reflection intensity of the imaging target scene is indirectly obtained through specular reflection:

[0069] ,

[0070] in, Represents the vectorized mirror reflection signal, which is , Represents the conversion coefficient vector, size is , represents the Hadamard product;

[0071] Substituting the above formula into the discretized transmission equation and transforming it, it can be expressed as:

[0072] ,

[0073] in, Indicates size A vector of all 1s, express The transpose of represents the Kronecker product;

[0074] The above formula can be simplified as:

[0075] ,

[0076] in, Represents the denoised non-viewing signal, which is , represents the overall transmission matrix, , the size is .

[0077] S4. According to the imaging conversion model, a one-stage transfer matrix estimation neural network is constructed, and the one-stage transfer matrix estimation neural network is iteratively optimized using the training data set to obtain the estimated transfer matrix .

[0078] The input of the first-stage transmission matrix estimation neural network is a random noise matrix, and the output is an estimated transmission matrix , the estimated transmission matrix The specular reflection signal in the training dataset Perform matrix multiplication and denoise the non-viewing signal Under the supervision of The generated transmission matrix Make optimizations.

[0079] S5. Construct a two-stage imaging scene estimation neural network based on the imaging conversion model, given the non-viewing signal of the hidden scene to be imaged , in the estimated transmission matrix The two-stage imaging scene estimation neural network is iteratively optimized with the participation of to obtain the estimated value of the mirror reflection of the hidden scene to be imaged. , thereby achieving non-line-of-sight imaging of hidden scenes.

[0080] The input of the two-stage imaging scene estimation neural network is a random noise matrix, and the output is the estimated value of the specular reflection of the hidden scene to be imaged. , the estimated value of the specular reflection of the hidden scene to be imaged With the estimated transmission matrix Perform matrix multiplication to obtain the non-viewing signal of the hidden scene to be imaged. Under the supervision of Estimated specular reflection of the hidden scene to be imaged Make optimizations.

[0081] The hardware devices involved in the method of the present invention are mainly ordinary continuous laser sources and steady-state optical sensors. The proposed two-stage imaging model can achieve effective non-field of view reconstruction of hidden plane scenes without relying on a large amount of data training. Example

[0082] This embodiment provides a two-stage active steady-state non-line-of-sight imaging method based on data-model driving, and the specific steps are as follows:

[0083] S1. Build an active steady-state non-line-of-sight imaging system, and use the active steady-state non-line-of-sight imaging system to obtain the non-line-of-sight signal and mirror reflection signal reflected by the imaging target scene.

[0084] S1.1 Build an active steady-state non-line-of-sight imaging system including continuous laser and steady-state optical sensor.

[0085] Figure 1 The non-field of view imaging setting involved in the present invention is given. The steady-state sensor involved in the present invention is an ordinary visible light camera, which is placed on the same side as a continuous laser source with a wavelength in the visible light range, and on the opposite side from the target scene to be imaged, with an obstacle blocking it, so that the camera cannot directly detect the scene content. The process of non-field of view imaging involved in the present invention can be described as follows: the light emitted by the continuous laser source indirectly illuminates the imaging target scene through diffuse reflection of the relay surface, and undergoes secondary diffuse reflection on the imaging target scene, wherein a portion of the light will be blocked by an obstruction between the imaging target scene and the relay surface, while the light that is not blocked by the obstruction will return to the relay surface and be received by the camera after a third diffuse reflection on the relay surface, so the camera will capture an optical signal that has undergone at least three reflections.

[0086] S1.2 Determine the distance between the imaging target scene and the relay surface, as well as the geometric shape and coordinates of the obstruction between the imaging target scene and the relay surface, and determine the lighting position and the effective field of view of the non-viewing area imaging according to the positional relationship among the imaging target scene, the obstruction and the relay surface.

[0087] Figure 2 A schematic diagram of a geometric configuration of this embodiment is given in top view and side view. The active area of ​​the hidden imaging target is parallel to the relay surface and at a distance of Ignoring the support part of the occluder, the main part of the occluder is a square, placed parallel to the relay surface and at a distance of . The camera is placed facing the relay surface and its optical axis is perpendicular to the relay surface. In addition, optical excitation and signal reception need to meet two important conditions: first, the position of the laser illumination point on the relay surface satisfies the condition that the reflected light field is not obstructed by occlusions while completely covering the range of the imaging target scene; second, the effective field of view of non-field of view imaging should make the transfer matrix as sparse as possible under the conditions that the plane of the imaging target scene and the position of the occlusion are determined, that is, the determination of the effective field of view should be based on increasing the variability between the columns of the transfer matrix as much as possible, thereby increasing the rank of the transfer matrix to improve the ill-posedness of the problem. For the first point, Figure 2 The top view and side view in the figure show the critical position of the laser illumination point under the above conditions. Its reflected light field can not only cover the imaging range of the imaging target scene, but also completely avoid the obstruction. For the second point, Figure 2 The dotted lines in the top view and side view in the figure depict an example that satisfies the corresponding transfer matrix conditions. These dotted lines start from the edge of the imaging target scene plane, pass through the edge of the occluder and intersect with the relay surface. The area with all intersection points as vertices constitutes the effective field of view of the non-line of sight imaging. The field of view of the camera should cover or include the effective field of view of the non-line of sight imaging.

[0088] S2. using an active steady-state non-line-of-sight imaging system to obtain a non-line-of-sight signal and a mirror reflection signal reflected by an imaging target scene, and preprocessing the non-line-of-sight signal and the mirror reflection signal to obtain a training data set;

[0089] Figure 3 The settings for collecting the non-line-of-sight signal and its corresponding mirror reflection signal of this embodiment are given. The non-line-of-sight signal of the imaging target scene is captured under the conditions that the relay surface is a conventional diffuse reflection surface, the movable target position changes, and an obstruction exists. The corresponding mirror reflection signal is captured when the movable target position changes, the relay surface at the third reflection receiving position is converted from a conventional diffuse reflection surface to a mirror surface, and the obstruction is removed.

[0090] The relay surface used for training data collection can be composed of a conventional diffuse reflection surface and a mirror surface. One side is completely composed of a conventional diffuse reflection surface, and the other side is set as a conventional diffuse reflection surface of the same material in the laser illumination area and a mirror surface in the signal receiving area. By flipping the relay surface while keeping the target fixed, the non-viewing signal of the imaging target scene and its mirror reflection signal can be collected.

[0091] The signal when there is no movable target in the acquisition scene is taken as the background signal, and the captured non-viewing signal and its mirror reflection signal from the imaging target scene are respectively differentiated from their respective background signals to obtain the denoised signal.

[0092] S3. Construct an imaging conversion model from non-line-of-sight signals to specular reflection signals.

[0093] The imaging conversion model is constructed based on the discretized transmission equation:

[0094] ,

[0095] in, Represents the vectorized non-viewing signal including noise, with a size of , It represents the reflection intensity of the imaging target scene after receiving indirect illumination from the active light source, and its magnitude is , represents the vectorized noise term, with a size of , represents the light transmission matrix under this condition, and its size is ; In direct access In case of difficulties, the specular reflection intensity of the imaging target scene can be indirectly obtained through specular reflection:

[0096] ,

[0097] in, Represents the vectorized mirror reflection signal, which is , Represents the transformation vector, whose size is , represents the Hadamard product, which can be expressed as follows after substituting it into the discretized transmission equation and transforming it:

[0098] ,

[0099] in, Indicates size A vector of all 1s, express The transpose of represents the Kronecker product. Further, the formula can be simplified as follows:

[0100] ,

[0101] in, Represents the background difference by performing The denoised non-viewing signal obtained after that is of size , represents the overall transmission matrix , the size is .

[0102] and The size of the image depends on the required imaging resolution.

[0103] S4. Construct a one-stage transfer matrix estimation neural network and iteratively optimize the network using the training data set to obtain the estimated transfer matrix for the corresponding imaging scene. .

[0104] Figure 4 A one-stage transfer matrix computation framework is described, which consists of a transfer matrix estimation neural network that receives a A random noise matrix of size is taken as input and output is The size of the transfer matrix , the transmission matrix and the imaging target scene mirror reflection signal Perform matrix multiplication and denoise the non-viewing signal Under the supervision of The generated transmission matrix The optimization is performed until the loss function converges, and the data required for the network training in this step comes from the training data set collected in step S2.

[0105] S5. Construct a two-stage imaging scene estimation neural network, given the non-viewing signal of the hidden scene to be imaged ', the transmission matrix generated in step S4 The network is iteratively optimized with the participation of , so as to obtain the estimated value of the mirror reflection of the unknown hidden scene .

[0106] Figure 5 A two-stage hidden scene content computation framework is described, which consists of an imaging scene estimation neural network that receives a As input, a random noise matrix of size φ takes as output an estimate of the specular reflection of the hidden scene. , which is the same as the transmission matrix obtained in step S4 Perform matrix multiplication to obtain the given non-visual signal Under the supervision of Specular Estimation for Generated Unknown Hidden Scenes The optimization is performed until the loss function converges. The non-line-of-sight signal of the hidden scene to be imaged used in this step is any non-line-of-sight signal of the active steady-state non-line-of-sight imaging system that satisfies step S1.

[0107] The effect of the method of the present invention can be further illustrated by the following experiments.

[0108] (1) Experimental conditions

[0109] The present invention uses Blender and Python and other related tools to conduct simulation experiments on a CPU of Intel(R) Core(TM) i7-9700K @ 3.60GHz, memory of 32G, GPU of NVIDIA GeForce RTX 3090, and Windows 10 operating system. The training and test data used in the experiment are generated by Cycles, a physically based rendering engine in Blender, and the examples are as follows: Figure 6 shown.

[0110] (2) Experimental content

[0111] The method of the present invention was tested as follows:

[0112] 1) Construct an active steady-state non-line-of-sight imaging system according to the method described in step S1;

[0113] 2) Collect training data according to the method described in step S2. All training data are collected based on a square target, such as Figure 6 As shown in the training examples in , by controlling the movement of a square target within the imaging scene, the non-viewing signals and mirror reflection signals of the same target at different positions are collected, and a total of 256 sets of data at different positions are collected;

[0114] 3) Construct a one-stage transmission matrix calculation framework according to the method described in step S4. The random noise size input in the experiment is set to , the dimension of the output transmission matrix is , the training data collected in step S2 is used to optimize the transfer matrix estimation neural network, the optimizer uses Adam, the learning rate is set to 0.005, the number of iterations is 1000 rounds, and the convergence curve of the loss function in the first stage of this embodiment is as follows: Figure 7 As shown, the output estimated transmission matrix like Figure 8 As shown;

[0115] 4) Finally, the two-stage hidden scene content calculation framework is constructed according to the method described in step S5. The random noise size input in the experiment is set to , the imaging resolution is The experiment tested the reconstruction effect of the imaging scene estimation neural network on the uppercase letters 'C', 'J', and 'L'. The test samples are as follows Figure 6As shown, the optimizer in this stage uses Adam, the learning rate is set to 0.0005, and the TV regularization term weight is The value is set to 0.01, the number of iterations for each test sample is 2500, and the iteration time is about 24 seconds. The convergence curve of the second-stage loss function of this embodiment is as follows: Fig. 9 As shown, the reconstruction effect is Fig.10 shown.

[0116] (3) Qualitative analysis

[0117] This embodiment also compares the imaging performance of the end-to-end data-driven method and the two-stage active steady-state non-viewing-area imaging method based on data-model driven proposed by the present invention. The experimental results are as follows: Fig.10 As shown in the figure, the end-to-end data-driven method used as a comparative solution adopts the same architecture as the two-stage imaging scene estimation neural network of the present invention, and implements end-to-end training based on the same training data set. Fig.10 The results shown show that the end-to-end data-driven method cannot successfully reconstruct the complete hidden target. In comparison, the method proposed in the present invention achieves better results and can effectively reconstruct images close to the contours and basic features of the real target. The experimental results further show that when the scale of training data is small, the end-to-end method that relies on data-driven is difficult to obtain good generalization performance. On the basis of data-driven, the present invention introduces a physical model to further optimize the non-viewing signal processing. This data-model driven approach effectively improves the performance of the neural network and also increases the interpretability of the computational framework.

[0118] In summary, the two-stage active steady-state non-line-of-sight imaging method based on data-model driving proposed in the present invention has simple hardware composition and low cost. Compared with transient non-line-of-sight imaging, it only requires ordinary continuous laser sources and steady-state optical sensors as excitation and detection devices; it has strong adaptability and is not limited by ambient lighting or the self-luminous properties of objects compared with passive non-line-of-sight imaging; it has good generalization and can achieve effective non-line-of-sight reconstruction of hidden targets without relying on a large amount of data training compared with end-to-end data-driven methods.

[0119] The contents not described in detail in the present invention belong to conventional techniques known to those skilled in the art. The above embodiments are only examples of the present invention and should not be regarded as limitations of the present invention. Those skilled in the art may make various modifications, equivalent replacements or improvements thereto without violating the spirit and principles of the present invention, which should all be regarded as included in the protection scope of the present invention.

Claims

1. A two-stage active steady-state non-line-of-sight imaging method based on data-model driving, characterized in that: The following steps are involved: S1. Build an active steady-state non-line-of-sight imaging system; S2. using an active steady-state non-line-of-sight imaging system to obtain non-line-of-sight signals and specular reflection signals reflected in an imaging target scene, and preprocessing the non-line-of-sight signals and specular reflection signals to obtain a training data set; S3. Construct an imaging conversion model from non-line-of-sight signals to specular reflection signals. The formula is: ， in, is the denoised and vectorized non-viewing signal, is the overall transmission matrix, is the denoised and quantized specular reflection signal; S4. According to the imaging conversion model, a one-stage transfer matrix estimation neural network is constructed, and the one-stage transfer matrix estimation neural network is iteratively optimized using the training data set to obtain the estimated transfer matrix ; S5. Construct a two-stage imaging scene estimation neural network based on the imaging conversion model, given the non-viewing signal of the hidden scene to be imaged , in the estimated transmission matrix The two-stage imaging scene estimation neural network is iteratively optimized with the participation of to obtain the estimated value of the mirror reflection of the hidden scene to be imaged. , thereby achieving non-line-of-sight imaging of hidden scenes.

2. According to the data-model driven two-stage active steady-state non-line-of-sight imaging method of claim 1, it is characterized in that: The active steady-state non-line-of-sight imaging system includes an imaging target scene, an obstruction, an obstacle, a relay surface, a continuous laser source, and a steady-state optical sensor; The continuous laser source and the steady-state optical sensor are located on the same side and are blocked by obstacles from the imaging target scene; The relay surface is parallel to the plane where the imaging target scene is located; an obstruction is provided between the relay surface and the imaging target scene; the imaging target scene includes a movable target, and the movable target can move freely within the range of the imaging target scene; The illumination position of the laser emitted by the continuous laser source on the relay surface satisfies that the reflected light field completely covers the imaging target scene and is not hindered by any obstruction.

3. The data-model driven two-stage active steady-state non-line-of-sight imaging method according to claim 2, characterized in that: The laser emitted by the continuous laser source illuminates the imaging target scene through diffuse reflection of the relay surface, and secondary diffuse reflection occurs in the imaging target scene. Part of the laser after the secondary diffuse reflection returns to the relay surface and is received by the steady-state optical sensor after a third reflection on the relay surface.

4. The data-model driven two-stage active steady-state non-line-of-sight imaging method according to claim 3, characterized in that: The non-field of view signal is: the signal received by the steady-state optical sensor when an obstruction exists and the relay surface where the third reflection occurs is a diffuse reflection surface; the mirror reflection signal is: the signal received by the steady-state optical sensor when the obstruction is removed and the relay surface where the third reflection occurs is a mirror surface.

5. The data-model driven two-stage active steady-state non-line-of-sight imaging method according to claim 1, characterized in that: The non-line-of-sight signal and the mirror reflection signal are preprocessed, and the specific steps are: collecting the signal when there is no movable target in the active steady-state non-line-of-sight imaging system as the background signal, and differentiating the non-line-of-sight signal and the mirror reflection signal with their respective background signals to obtain a denoised non-line-of-sight signal or a denoised mirror reflection signal.

6. The data-model driven two-stage active steady-state non-line-of-sight imaging method according to claim 1, characterized in that: The imaging conversion model from the non-viewing area signal to the specular reflection signal is constructed in the following specific steps: The imaging conversion model is constructed based on the discretized transmission equation: ， in, Represents the vectorized non-viewing signal including noise, with a size of ; It represents the reflection intensity of the imaging target scene after receiving indirect illumination from the active light source, and its magnitude is ; represents the vectorized noise term, with a size of ; represents the light transmission matrix, whose size is ; In direct access In difficult cases, the specular reflection intensity of the imaging target scene is indirectly obtained through specular reflection: ， in, Represents the vectorized mirror reflection signal, which is , Represents the conversion coefficient vector, size is , represents the Hadamard product; Substituting the above formula into the discretized transmission equation and transforming it, it can be expressed as: ， in, Indicates size A vector of all 1s, express The transpose of represents the Kronecker product; The above formula can be simplified as: ， in, Represents the denoised non-viewing signal, which is , represents the overall transmission matrix, , the size is .

7. The data-model driven two-stage active steady-state non-line-of-sight imaging method according to claim 1, characterized in that: The input of the first-stage transmission matrix estimation neural network is a random noise matrix, and the output is an estimated transmission matrix , the estimated transmission matrix The specular reflection signal in the training dataset Perform matrix multiplication and denoise the non-viewing signal Under the supervision of The generated transmission matrix Make optimizations.

8. The data-model driven two-stage active steady-state non-line-of-sight imaging method according to claim 1, characterized in that: The input of the two-stage imaging scene estimation neural network is a random noise matrix, and the output is the estimated value of the specular reflection of the hidden scene to be imaged. , the estimated value of the specular reflection of the hidden scene to be imaged With the estimated transmission matrix Perform matrix multiplication to obtain the non-viewing signal of the hidden scene to be imaged. Under the supervision of Estimated specular reflection of the hidden scene to be imaged Make optimizations.

Images