Imaging system configured to perform line-of-sight imaging and non-line-of-sight imaging

Abstract

An imaging system (1) has: a light transmission system (5, 8) configured to generate and steer an output light signal (LO); a light receiving system (6, 12) configured to detect an input light signal (LI); and electronic resources (3, 4, 5, 16, 17, 18) configured to control the light transmission system and the light receiving system and to perform imaging based on a time-of-flight between emission of the output light signal and detection of the input light signal. The imaging system is configured to perform an imaging calibration (51) comprising: acquiring an environment map (ENV_MAP) that represents a three- dimensional image of an area scanned by the output light signal; and identifying, from the environment map, the presence of one or more objects (60, 61, 62) in the scanned area. The imaging system is further configured to: perform (53) a Line-of-Sight, LOS, imaging of a region of interest within the scanned area; and perform (54) a Non-Line-of- Sight, NLOS, imaging by illuminating, as a relay surface, at least one of the objects (61) identified within the scanned area.

Description

[0001] “IMAGING SYSTEM CONFIGURED TO PERFORM LINE-OF-SIGHT IMAGING AND NON-LINE-OF-SIGHT IMAGING”

[0002] Cross-Reference to Related Applications

[0003] This Patent Application claims priority from European Patent Application No. 24173931.7 filed on May 2, 2024 and from Italian Patent Application No. 102024000022602 filed on October 10, 2024 the entire disclosures of which are incorporated herein by reference.

[0004] Technical Field of the Invention

[0005] The present invention relates to an imaging system that is configured to perform Line-of-Sight (LOS) imaging and Non-Line-of-Sight (NLOS) imaging.

[0006] State of the Art

[0007] Imaging systems are known to be able of reconstructing a 3D volume in space by using Time-of-Flight (TOF) measurements.

[0008] The imaging system emits a laser pulse towards a direction in space by a steering system and detects a reflected return signal.

[0009] The returning signal is detected by a single -photon detector which has a sensitivity at the single-photon level and can time-stamp photons arrival time with ps-resolution, thereby allowing TOF measurements.

[0010] Current 3D imaging systems based on single-photon counting can be grouped into two categories: Line-of-Sight (LOS) systems and Non-Line-of-Sight (NLOS) systems.

[0011] Both LOS and NLOS systems exploit the principle of TOF measurement to reconstruct a 3D volume in space.

[0012] The known LOS systems are limited to the reconstruction of a target (scene, object or volume) that is placed along the line of sight of the system.

[0013] The known NLOS systems are limited to the reconstruction of a target (scene, object or volume) that is not placed along the line of sight of the system.

[0014] In particular, NLOS imaging is based on the illumination of a relay surface that is in the line-of-sight of the system and the detection of the photons that are scattered by the relay surface onto the target, reflected back by the target onto the relay surface and scattered back to the imaging system. In practice, NLOS imaging is based on detection of photons that undergo a three-bounce scattering process. Accordingly, known imaging systems are prepared and designed to work either in the LOS or NLOS modality; thus, known LOS systems cannot reconstruct a NLOS scene and known NLOS systems cannot reconstruct a LOS scene.

[0015] For example, WO 2021 / 232031 A2 discloses a system that is configured to perform only NLOS imaging of a hidden object.

[0016] Moreover, the known NLOS imaging systems are pre-calibrated to work in a fixed position; thus, if the NLOS imaging system moves, the NLOS imaging system is subject to malfunctions and errors.

[0017] Therefore, the known imaging systems have a low versatility of use in real life applications.

[0018] Subject and Summary of the Invention

[0019] The aim of the present invention is to overcome the disadvantages of the prior art.

[0020] The present invention relates to an imaging system, a method, and a computer program, as claimed in the appended claims.

[0021] Brief Description of the Drawings

[0022] Figure 1 shows a block diagram of an imaging system, according to an embodiment of the invention.

[0023] Figure 2 shows a flowchart of an imaging method, according to an embodiment of the invention.

[0024] Figures 3A-3D show examples of a reconstructed scene during an imaging calibration of the system of Figure 1.

[0025] Figure 4A shows an exemplificative schematic view of the system of Figure 1, when used for LOS imaging.

[0026] Figure 4B shows an exemplificative schematic view of the system of Figure 1, when used for NLOS imaging.

[0027] Detailed Description of Preferred Embodiments of the Invention

[0028] The following description is provided to enable a person skilled in the art to make and use the invention. Various modifications to the embodiments will be readily apparent to those skilled in the art, without departing from the scope of the claimed invention. Thus, the present invention is not intended to be limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and features disclosed herein and defined in the appended claims.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments disclosed belongs. In the case of conflict, the present specification, including definitions, will prevail. In addition, the examples are illustrative only and not intended to be limiting.

[0030] For the purposes of facilitating understanding of the embodiments described herein, reference will be made to certain embodiments and specific language will be used to describe the same. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the present disclosure.

[0031] Figure 1 shows an imaging system 1 that is configured to perform imaging based on time of flight (TOF).

[0032] The TOF-based imaging allows to reconstruct an image of an illuminated target. In particular, the TOF-based imaging allows to reconstruct an image of the arrangement of target in a spatial coordinate system X, Y, Z, i.e., to reconstruct the profiles and distance of the illuminated target.

[0033] The word target may be used hereinafter to indicate indistinctively, unless otherwise specified, any region of interest (ROI) to be imaged, such as a scene, a volume and in general any object, including physical objects, persons, animals, etc., for which it is desired to obtain an image.

[0034] The imaging system 1 comprises an electronic processing or control unit 3, hereinafter indicated as CPU 3, and a memory 4, mutually coupled with one another.

[0035] The imaging system 1 further comprises a light source 5 that is configured to emit an output light signal LO and a light detector 6 that is configured to detect an input light signal LI.

[0036] In the embodiment of Figure 1, the light source 5 is a laser source configured to emit a laser beam, in particular a pulsed laser beam.

[0037] The optical parameters of the laser output LO, such as wavelength, repetition rate, power of the pulses, duration of the pulses, etc., may depend on the specific type of the light source 5 and may be fixed or selectable by a user of the imaging system 1.

[0038] The light source 5 also provides a signal, in particular a digital signal, that is indicative of the emission of the light output signal LO. In other words, the digital signal is indicative of the time instant at which a laser pulse is emitted. The light detector 6 is a single-photon detector, i.e., having a single-photon sensitivity; in particular, the light detector 6 may be a Single Photon Avalanche Diode (SPAD).

[0039] The imaging system 1 also comprises an optical transmission system 8 that directs the output light signal LO provided by the light source 5 outside the imaging system 1 and towards specific directions in space, depending on the target to be imaged and the imaging mode (i.e., LOS or NLOS).

[0040] In detail, in the embodiment of Figure 1, the optical transmission system 8 comprises two mirrors TM1, TM2 configured to allow scanning of an area within the field of view of the imaging system 1.

[0041] By way of example, mirrors TM1, TM2 may be fast rotating mirrors, in particular galvo-mirrors.

[0042] Movement of the mirrors TM1, TM2 may be controlled by respective drivers 9, 10.

[0043] In practice, the optical transmission system 8 is configured to allow steering of the output light signal LO along two orthogonal axis X, Y.

[0044] The imaging system 1 comprises an optical receiving system 12 that collects the input light signal LI coming from the outside of the imaging system 1 and directs the photons of the input light signal LI onto the light detector 6.

[0045] In detail, in the embodiment of Figure 1, the optical receiving system 12 comprises two mirrors RM1, RM2 configured to allow collection of photons from the two- dimensional scanned area.

[0046] By way of example, mirrors RM1, RM2 may be fast steering mirrors, in particular galvo-mirrors.

[0047] Movement of the mirrors RM1, RM2 may be controlled by respective drivers 13, 14.

[0048] In practice, the optical receiving system 12 is configured to allow collection of returning photons from two-orthogonal axis X, Y.

[0049] The optical transmission system 8 and / or the optical receiving system 12 may further comprise other optical elements, depending on the specific application and configuration, such as for example one or more optical lenses.

[0050] The light detector 6 is also coupled to a respective driver 15 that is configured to regulate functioning of the light detector 6.

[0051] In detail, the driver 15 may control the activation of the light detector 6 and may read the signal, for example voltage or current signals, generated by the light detector 6 in response to the detection of one or more photons of the input light signal LI.

[0052] A pulse generator 16 controls the optical transmission system 8 and the optical receiving system 1, in particular, it controls the movement of the respective mirrors TM1, TM2 and RM1, RM2.

[0053] In detail, the pulse generator 16 may provide analogic signals to the drivers 9, 10 and 13, 14 for controlling movement of mirrors TM1, TM2 and RM1, RM2.

[0054] The pulse generator 16 may also provide signals, in particular digital signals, that are configured to tag acquisition time instants. For example, the digital signals may be indicative of the initial time instant of a frame acquisition.

[0055] A time tagger 17 is configured to receive the signals (e.g., digital signals) provided by the light source 5, the detector driver 15 and the pulse generator 16, and to time stamp the arrival time of said signals.

[0056] In other words, the time tagger 17 provides a time label to the emission instant of a pulse by the light source 5, a time label to the reception of one or more photons by the light detector 6, and a time label to the initial instant of the frame acquisition.

[0057] Signals indicative of the time stamps are provided by the time tagger 17 to the CPU 3 for further processing; that is, to obtain LOS or NLOS imaging, as discussed hereinafter.

[0058] The imaging system further has a time-delaying block or delayer 18 which is coupled to the detector driver 15 of the light detector 6 in order to control a time delay TD for the activation of the light detector 6.

[0059] The time delay TD may be counted with respect to the time instant at which a light pulse has been emitted by the light source 5. In detail, the delayer 18 may receive the digital signal provided by the light source 5 in response to the emission of a light pulse and count the time delay TD in response to the arrival of said signal.

[0060] In practice, the light source 5 and the optical transmission system 8 form a light transmission system that is configured to generate the output light signal LO and steer the output light signal LO.

[0061] In practice, the light detector 6 and the optical receiving system 12 form a light receiving system that is configured to detect the input light signal LI incoming on the imaging system 1 and originated from a reflection of the output light signal LO on the illuminated scene.

[0062] In practice, CPU 3, memory 4, driver 15, pulse generator 16, time tagger 17 and delayer 18 form electronic resources that are configured to control the light transmission and receiving systems, in order to perform the TOF-based imaging. The imaging system 1 further comprises a motion sensing system 20, in particular a position sensor or an attitude sensor that is configured to detect a movement of the imaging system 1.

[0063] The motion sensing system 20 is configured to monitor a displacement, such as a rotation or a translation, of the whole imaging system 1.

[0064] In detail, the motion sensing system 20 may be configured to monitor the displacement of the light receiving system 6, 12 and / or light transmission system 5, 8. In other words, to monitor the displacement of those elements of the imaging system 1 whose positioning may affect the imaging accuracy.

[0065] Preferably, the motion sensing system 20 may comprise one or more inertial sensors (e.g., accelerometers and gyroscopes). However, in addition or in alternative, the motion sensing system 20 may comprise one or more cameras and / or geo-positioning sensors (e.g., GPS sensors) configured to detect a movement of the imaging system 1.

[0066] The motion sensing system 20 is coupled to the CPU 3 and provides to the CPU 3 motion data indicative of the movement of the imaging system 1.

[0067] The imaging system 1 may further comprise a user interface 21, in particular a screen or monitor, for allowing interactions with a user, such as an operator, of the imaging system 1.

[0068] The imaging system 1 is configured to perform a LOS imaging or a NLOS imaging, based on selection data that are indicative of an input received by a user of the imaging system 1.

[0069] With reference to Figure 2, a method 50 is described for controlling the imaging system of Figure 1 so that the imaging system 1 can be used for both LOS and NLOS imaging.

[0070] One or more of the steps of method 50 may be executed by the CPU 3.

[0071] The method 50 may start, step SI, in response to the reception of a start signal by a user of the imaging system 1.

[0072] For example, step SI may be performed at a first use of the imaging system 1, for example after turning on the imaging system 1. Alternatively, step SI may be forced by the user during functioning of the imaging system 1, so as to restart an imaging calibration of the imaging system 1.

[0073] At step SI, the CPU 3 may control the interface 21, for example by controlling a GUI of the interface 21, so as to allow the user to define initial configuration parameters of the imaging system 1. For example, at step SI, the user may set the optical parameters of the light source 5, such as laser repetition rate and optical power.

[0074] The user may also set configuration parameters of the delayer 18 and the detector driver 15.

[0075] After step SI, the method 50 proceeds on performing an imaging calibration 51 wherein the imaging system 1 acquires an environment map ENV_MAP and identifies, from the environment map ENV_MAP, a group of features that will be used to perform LOS or NLOS imaging, as discussed in detail hereinafter.

[0076] The environment map ENV_MAP represents a three-dimensional image of an area that is within the field of view of the imaging system 1, that is the area that can be scanned by the optical transmission system 8.

[0077] In other words, the environment map ENV_MAP is acquired without moving the imaging system 1, but only by deflecting the mirrors RM1, RM2 and TM1, TM2 so as to scan the area within the line-of-sight of the imaging system 1.

[0078] The environment map ENV_MAP is acquired by performing a LOS imaging of the scanned area.

[0079] The acquisition of step S2 is based on a LIDAR technique and may be classified, depending on the specific principle that is used for estimating the distance of the target, for example as TOF-based LIDAR.

[0080] In detail, at a step S2, the imaging system 1 controls the light source 5, the optical transmission system 8 and the pulse generator 16 so as to steer the output light signal LO over a scanning area, in particular over the full scene within the area that can be scanned with the optical transmission system 8.

[0081] At step S2, the imaging system 1 also controls the optical receiving system 12, the light detector 6 and respective driver 15, and the pulse generator 16 so as to detect the input light signal LI.

[0082] At step S2, the imaging system 1 measures the TOF as a function of the time difference between the emission of a pulse by the light source 5 and the reception of photons by the light detector 6.

[0083] Based on the TOF measurements, the imaging system 1 may thus obtain, for each scanning point within the scanning area, data indicative of the distance of a target. In fact, the distance may be estimated by multiplying half of the travel time of the photons from the source to the target and back to the imaging system 1, by the speed of light.

[0084] In detail, for each scanning point, at step S2: - the CPU 3 controls the light source 5 so as to emit a light pulse (output light signal

[0085] LO);

[0086] - the output light signal LO travels in a free space optical channel until eventually interacts with an object along the propagation direction thereof, which may reflect, at least partially, the impinging output light signal LO based on the optical properties of the object material;

[0087] - the reflected light signal propagates back to the imaging system 1 (input light signal LI), wherein it is directed, by the optical receiving system 12, onto the light detector 6 which, in response, provides an electrical signal at output indicative of the received signal, in particular indicative of the intensity of the received signal;

[0088] - the time tagger 17 calculates the ToF as the difference between the time of the emission of the output light pulse LO and detection of the input light signal LI.

[0089] Thus, for each scanning point, the imaging system 1 acquires an intensity histogram as a function of the TOF, wherein the difference between one light pulse emission and the other may be considered as the maximum TOF.

[0090] After a sufficient acquisition time to have enough statistics, which may be defined during initialization or calibration of the imaging system 1, the intensity histogram of each scanning point is characterized by a peak whose intensity is related to the reflectivity of the target and whose TOF is related to its distance from the imaging system 1.

[0091] Preferably, at step S2, the optical transmission system 8 and the optical receiving system 12 may be set confocal one with the other.

[0092] Preferably, for each scanning position, the TOF for which the intensity histogram has the maximum value may be considered for calculating the distance of the target. However, other metrics may be used, depending on the specific application.

[0093] Thus, at the end of step S2, a dataset representing a 3D reconstruction of the scanned scene is obtained. In other words, the dataset may be a 3D point cloud.

[0094] The dataset output from step S2 may be a raw 3D dataset.

[0095] By way of example only, an example of a dataset generated at step S2 is shown in Figure 3A. For example, the raw dataset may have a first and a second axis that are indicative of an x and, respectively, y-coordinate of the scanned area, and a third axis that is indicative of the measured TOF.

[0096] As shown in Figure 3A, the first and second axis may be indicative of the voltages Vx and Vy that are used for driving the mirrors TM1, TM2 and RM1, RM2.

[0097] The raw 3D dataset may be converted, step S3, to a processed 3D dataset in a spatial reference system (X, Y, Z) of the imaging system 1.

[0098] The conversion may be performed by the CPU 3 by running a dedicated algorithm on the raw 3D dataset, which may be stored in the memory 4, and previously calibrated based on the optical properties of the imaging system 1.

[0099] The pre-calibrated function that is used for the conversion of step S3 is based on a physical model of the beam steering system (i.e., optical transmission system 8 and optical receiving system 12), and is configured to enable mapping of the beam direction and beam observation point (i.e., propagation direction and observation point of output light signal LO and input light signal LI) onto the coordinates (x,y,z) of the spatial reference system of the imaging system 1.

[0100] For example, the pre-calibrated function that is employed at step S3 may be the one described in the scientific article by Manakov et al, “A mathematical model and calibration procedure for galvanomtric laser scanning systems”, published in International Symposium on Vision, Modeling, and Visualization (2011), DOL / 10.2312 / PE / VMV / VMV11 / 207-214, in particular in the respective sections 2.1 and 3.

[0101] By way of example only, Figure 3B shows an example of the processed 3D dataset that may be obtained at step S3 by converting the raw dataset of Figure 3A.

[0102] The processed dataset obtained at step S3, which may be stored in the memory 4, forms the environment map ENV_MAP.

[0103] Then, step S4, the CPU 3 identifies, from the environment map ENV_MAP, the presence of one or more objects within the area scanned at step S2. Preferable, the identification is performed based on the converted map obtained at step S3; this may allow to improve the accuracy of the imaging steps described hereinafter, in particular the NLOS imaging.

[0104] In detail, the CPU 3 performs a feature analysis based on the acquired environment map ENV_MAP and, in response, provide LOS feature data ENV_LOS and / or NLOS feature data ENV_NLOS.

[0105] The LOS feature data ENV_LOS is useful for performing LOS imaging and the NLOS feature data ENV_NLOS is useful for performing NLOS imaging, as discussed in detail hereinafter.

[0106] For providing the LOS feature data ENV_LOS, the feature analysis may comprise identifying one or more objects in the environment map ENV_MAP that are relevant environment features in the scanned scene. For example, the object to be identified may be buildings, doors, walls, etc.

[0107] In detail, the processed 3D dataset may be provided at input of a known object recognition algorithm, such as for example a fast R-CNN or a Y OLO algorithm.

[0108] Thus, the LOS feature data ENV_LOS includes the identified objects (e.g., the points of the 3D processed map that have been associated to the identified object) and the respective label (e.g., building, wall, etc.).

[0109] The LOS feature data ENV_LOS may be stored in the memory 4.

[0110] By way of example only, Figure 3C shows an example of the relevant environment features that have been recognized in the processed 3D dataset of Figure 3B; each relevant environment feature is identified in Figure 3C by a surface coloured with a respective shade of grey. For example, in Figure 3C, three relevant environment features ENV_LOS have been identified, each corresponding to a respective wall.

[0111] For providing the NLOS feature data ENV_NLOS, the feature analysis of step S4 may comprise identifying one or more surfaces in the environment map ENV_MAP that are suitable for being used as relay or scattering surface for performing NLOS imaging. For example, the surfaces to be identified may be walls, ceilings, or other planar surfaces.

[0112] In detail, the processed 3D dataset may be provided at input of a known surface recognition algorithm, for example which exploits a plane segmentation algorithm for identifying the surfaces. For example, the surface recognition algorithm may be based on a RANSAC method or any other known method.

[0113] Thus, the NLOS feature data ENV_NLOS include the identified surfaces (e.g., the points of the 3D processed map that have been associated to the identified surface), the position of the identified surfaces and the orientation of the identified surfaces.

[0114] The NLOS feature data ENV_NLOS may be stored in the memory 4.

[0115] By way of example only, Figure 3D shows an example of the suitable relay surfaces (here four walls numbered from 1 to 4 and each coloured with a respective shade of grey) that have been identified in the processed 3D dataset of Figure 3B.

[0116] In practice, with steps S2 to S4, the imaging system 1 has been calibrated for the subsequent imaging steps.

[0117] Then, step S5, the imaging system 1 may ask the user to select the desired type of imaging method to be performed, that is to perform either LOS imaging or NLOS imaging.

[0118] For example, the CPU 3 may control the monitor interface 21 to display a pop-up to the user for selection of the desired imaging technique. Moreover, at step S5, the user may select, from the environment map ENV_MAP, one of the features on which perform the subsequent imaging.

[0119] In practice, the CPU 3 may receive selection data that are indicative of the selected type of imaging.

[0120] If the user selects to perform LOS imaging (left branch 53 from step S5), then the user may select a region of interest (ROI) to be imaged from the environment map ENV_MAP. In detail, the imaging system 1 may show to the user, through the interface 21, the LOS feature data ENV_LOS that have been identified at step S4 and stored in the memory 4, so that the user may select one of the environment features included in the LOS feature data ENV_LOS to be imaged.

[0121] In practice, the selected ROI may be within the area that was scanned during the imaging calibration 51. Therefore, the LOS imaging of step S7 may be based on the imaging calibration 51 (in particular based on the LOS feature data ENV_LOS or, more generally, based on the environment map ENV_MAP acquired during the imaging calibration 51).

[0122] Then, step S6, the time delay TD is defined. The time delay TD may be chosen by a user, for example through the interface 21, or defined automatically based on the target to be imaged, depending on the specific application.

[0123] The time delay TD is defined based on the distance of the target for which LOS imaging must be performed.

[0124] For example, as shown with reference to Figure 4A, if the selected ROI is a scene behind an obscurant 60, such as for example smoke, fog, or a camouflage net, that is placed at a distance L from the imaging system 1 and that partially allows transmission of an impinging light signal, then the time delay TD may be selected as a function of the distance L. By doing so, it is possible to increase the sensitivity for imaging of targets T placed beyond the obscurant 60.

[0125] In detail, the time delay TD may be a function of the time of flight of photons from the imaging system 1 to the target T and back to the imaging system 1.

[0126] Moreover, at step S6, a time gate TG may be selected; for example, the time gate TG may be chosen by a user, in particular through the interface 21. The time gate TG indicates the duration of an integration time window (for example, the time gate TG may be equal to the duration of the integration time window). During the integration time window, the light detector 6 is active; in other words, during the integration time window, the light detector 6 is enabled so as to acquire the incoming photons. Then, step S7, LOS imaging towards the selected ROI is performed. The LOS imaging is configured to obtain a profile and / or volumetric reconstruction of targets placed in line-of-sight of the system 1 within the scanned area.

[0127] The imaging technique of step S7 may be the same as the one described with reference to step S2.

[0128] In practice, also the acquisition of step S7 is based on a LIDAR technique and may be classified, depending on the specific principle that is used for estimating the distance of the target, for example as TOF-based LIDAR.

[0129] In detail, at a step S7, the imaging system 1 controls the light source 5, the optical transmission system 8 and the pulse generator 16 to steer the output light signal LO over a scanning area that encompasses the selected ROI to be imaged.

[0130] In other words, preferably, the output light signal LO is not scanned over the entire area that can be scanned with the optical transmission system 8, but only over the area associated to the selected ROI, as defined in the LOS feature data ENV_LOS.

[0131] The scanned area at step S7 comprises a plurality of scanning positions so that the area illuminated by the light output signal LO covers the area wherein the ROI is placed.

[0132] At step S7, the imaging system 1 also controls the optical receiving system 12, the light detector 6 and respective driver 15, and the pulse generator 16 to detect the input light signal LI.

[0133] At step S7, the imaging system 1 measures the TOF as a function of the time difference between the emission of a pulse by the light source 5 and the reception of photons by the light detector 6.

[0134] In detail, for each scanning position, at step S7:

[0135] - the CPU 3 controls the light source 5 so as to emit a light pulse (output light signal LO);

[0136] - the output light signal LO travels in a free space optical channel until it interacts with the obscurant 60, part of the output light signal LO may be reflected and part may pass though and propagate towards the target T;

[0137] - part of the light reaching the target T is reflected back and propagates back to the imaging system 1, through the obscurant 60; the resulting input light signal LI is directed, by the optical receiving system 12, onto the light detector 6 which, in response, provides an electrical signal at output indicative of the received signal, in particular indicative of the intensity of the received signal. The detector 6 is kept active during the integration time window of duration TG defined at step S6; - the CPU 3 calculates the ToF as the difference between the time of the emission of the output light pulse LO and detection of the input light signal LI.

[0138] Thus, for each scanning point, the imaging system 1 acquires an intensity histogram as a function of the TOF, wherein the difference between one light pulse emission and the other may be considered as the maximum TOF.

[0139] After a sufficient acquisition time to have enough statistics, which may be defined during initialization or calibration of the imaging system 1, the intensity histogram of each scanning point is characterized by a peak whose intensity is related to the reflectivity of the target and whose TOF is related to its distance from the imaging system 1.

[0140] The CPU 3, based on the intensity peak and TOF, may thus reconstruct intensity and depth contour map of the objects in line-of-sight of the imaging system 1.

[0141] In particular, with reference to the example of Figure 4A, the imaging system 3 may obtain LOS imaging of the obscurant 60 and / or target T.

[0142] Preferably, during the acquisition of step S7, the delayer 18 controls the light detector 6 and respective driver 15 so that detection is delayed by the time delay TD from emission of the output light signal LO. In particular, for each scanning position, detection of the input signal LI is delayed by the time delay TD with respect to the emission of the respective light output signal LO.

[0143] By doing so, the light detector 6 is not saturated by received light signal that has back reflected from possible obstacles that are placed at a distance smaller than L from the imaging system 1. Therefore, this allows to increase the sensitivity to image targets beyond the obscurant 60.

[0144] Finally, the reconstructed profile obtained at step S7 is provided to the user (step S8), for example is displayed on the interface 21.

[0145] Back with reference to step S5, if the user selects to perform NLOS imaging (right branch 54 from step S5), then the user may select a region of interest (ROI) to be used as a relay surface, from the environment map ENV_MAP.

[0146] In detail, the imaging system 1 may show to the user, through the interface 21, the NLOS feature data ENV_NLOS that have been identified at step S4 and stored in the memory 4, so that the user may select one of the relay surfaces in the NLOS feature data ENV_NLOS.

[0147] For example, Figure 4B shows an example wherein a relay wall 61 has been selected as the relay surface of the NLOS feature data ENV_NLOS to be used for NLOS imaging, to verify the presence of a target T that is placed behind an obstacle 62. Then, step S10, the CPU 3 defines an area, also called grid, to be scanned on the selected relay surface (for example the relay wall 61 of Figure 4B), which will be used as scanning area for the subsequent NLOS imaging.

[0148] In other words, the grid is formed by a set of points, each representing a position on the relay surface towards the output light signal LO will be pointed during the scan process.

[0149] Preferably, the grid may cover only a portion of the selected relay surface. However, the grid may also correspond to the whole selected relay surface.

[0150] The grid may have a regular or irregular geometry, depending on the specific application and algorithm used for image reconstruction. Preferably, the grid may have a regular shape.

[0151] Density of the points of the grid may be regularly spaced or irregularly spaced within the grid, depending on the specific application and algorithm used for scanning and image reconstruction. Preferably, the points may be regularly spaced.

[0152] Depending on grid shape, density and size the resolution of the NLOS imaging may be improved.

[0153] In detail, at step S 10 the CPU 3 may:

[0154] - identify a centre point of the grid; for example, the CPU 3 may provide to the user, through the interface 21, an image of the selected relay wall 61 and allow the user to manually select the centre point, for example with a click on the interface 21. Alternatively, the centre point may be automatically identified by the CPU 3 on the interface 21 ; and

[0155] - generate the grid around the centre point. For example, the CPU 3 may provide a group of possible grids and allow the user to select the grids manually. Alternatively, the CPU 3 may automatically generate the grid around the centre point.

[0156] The so generated grid has physical dimensions Sx along first axis X and Sy along second axis Y, wherein X and Y may be two orthogonal axes in the reference system of the imaging system 1.

[0157] Then, step SI 1, NLOS acquisition is started based the grid defined at step S10.

[0158] The imaging system 1 raster scans the grid by providing the output light signal LO and directing the output light signal LO towards the relay surface.

[0159] In practice, each point of the grid forms a pixel having coordinates (x_pixel, y_pixel) which is illuminated for a certain amount of time, which may be defined by a user. The projected grid is guaranteed to be the one desired in the relay surface reference frame thanks to the pre-calibration of the laser beam steering (e.g., the imaging calibration 51). For this purpose, the best fitting plane equation of the designated relay surface is determined from a LOS scan and converted in the device reference frame, as described for the imaging calibration 51. The calibration of the steering system is then exploited to determine the direction of the beam that point to the pre-determined grid points in the relay surface reference frame.

[0160] In detail, given the tuple (x_pixel, y_pixel), the pulse generator 16 generates an analogue signal with waveform either defined by the user or pre-defined in the memory 4. The function mapping the target positions on the wall to the required electrical signals, such as voltages, that control the beam steering system 8 is obtained based on the previously computed surface reference frame.

[0161] For each pixel of the grid, the returning NLOS signal is recorded by the light detector 6.

[0162] For each pixel of the grid, the CPU 3 records time-resolved histograms.

[0163] After having scanned the whole grid, a 3D dataset is obtained which is formed by time-resolved histogram for each pixel of the grid. The 3D dataset has two spatial coordinates x, y and a time coordinate t.

[0164] Then, the CPU 3 converts the acquired 3D dataset by means of a specific NLOS reconstruction algorithm, to map the acquired 3D dataset into a 3D volume having spatial coordinates x-y-z which represents the hidden target T.

[0165] In fact, the NLOS algorithm function exploits the fact that light impinging on the relay surface 61 is scattered also over the hidden target T that is placed behind the obstacle 62, and light reflected by the target T can be scattered again from the relay surface 61 towards the imaging system 1 and thus detected by the light detector 6. In practice, NLOS imaging is based on a three-bounce scattering process.

[0166] The CPU 3 may be configured to run any known NLOS reconstruction algorithm. For example, the CPU 3 may be configured to run the NLOS algorithm described in the scientific article by Lindell et al, titled “Wave-based non-line-of- sight imaging using fast fk migration”, ACM Transactions on Graphics (ToG) 38.4 (2019): 1-13.

[0167] Finally, the reconstructed image may be displayed (step S8) to the user.

[0168] For example, the CPU 3 may control the interface 21 to display a matrix of voxels, wherein each voxel is coloured with a grayscale that is indicative of the probability to have a target in that voxel. After LOS or NLOS imaging has been performed, method 50 may proceed to a step S12, wherein the CPU 3 decides whether to restart the acquisition (i.e., repeat step S7 or Si l depending on the selected application) or proceed to an initial step S 13.

[0169] For example, at step S12, the user is allowed to decide if they want to restart the acquisition by displaying, on the interface 21, a pop-up GUI.

[0170] By default, the restart conditioning may always be set to “false”, so that the current acquisition is not interrupted.

[0171] However, when the user selects to restart the acquisition process, the method 50 moves to step S13.

[0172] Step S13 may allow to recursively repeat LOS and NLOS imaging steps. For example, method 50 may return to step S13 during use, to form a functional loop for controlling use of the imaging system 1.

[0173] According to the embodiment of Figure 2, method 50 further provides to verify whether the imaging system 1 has moved, before performing the subsequent LOS or NLOS imaging.

[0174] In detail, the CPU 3 receives (step S20) motion data, from the motion sensors 20, which are indicative of a movement, such as for example a rotation and / or a translation, of the imaging system 1.

[0175] Then, step S21, the CPU 3 determines whether the imaging system 1 has moved with respect to a previous iteration.

[0176] Step S21 may be performed in the auto-start mode, that is after step S 13.

[0177] In detail, the CPU 3 verifies if the imaging system 1 has moved with respect to the time at which the imaging calibration 51 was last performed. In other words, the CPU 3 verifies if the imaging system 1 has moved with respect to the time at which the current environment map ENV_MAP, as stored in the memory 4, has been acquired.

[0178] For example, the CPU 3 may verify, based on the motion data, if the imaging system 1 has undergone a rotation and / or a translation that is higher than one or more thresholds. The use of thresholds may help to filter out small movements that would not affect the current calibration.

[0179] If the CPU 3 verifies that the imaging system 1 has moved, then acquisition of an updated environment map ENV_MAP is needed. Therefore, method 50 proceeds to step S2 and repeats the imaging calibration 51.

[0180] If the CPU 3 verifies that the imaging system 1 has not moved, then acquisition of an updated environment map ENV_MAP is not needed. Therefore, method 50 proceeds to step S5, so that LOS or NLOS imaging of another area can be performed.

[0181] From the above description, the advantages of the invention emerge clearly.

[0182] In fact, the imaging system 1 allows to perform, with the same hardware, both LOS and NLOS imaging. Therefore, the imaging system 1 has a high versatility of applications.

[0183] In particular, the imaging calibration 51 allows to store a reconstructed map of the scene in the line-of-sight of the imaging system 1 , and the reconstructed map may be used for performing both LOS and NLOS imaging with a high accuracy.

[0184] Moreover, the fact that the imaging calibration 51 may be repeated automatically every time that the imaging system 1 moves, may guarantee to follow the relay wall in case the imaging system 1 has moved, and thus to correctly perform NLOS imaging with high accuracy.

[0185] Use of a single-photon detector as light detector 6 and / or hardware (light detector 6, light source 5, delayer 18, pulse generator 16, etc.) with very low time resolution and jitter, for example picosecond time resolution, may allow to further increase the resolution of the imaging system 1.

[0186] The resolution in the depth of the target is in fact directly associated to the convolution of the system temporal uncertainties as the duration of the light pulses (LO) and the jitter of the light detector 6 and light source. The sensitivity of the imaging is instead related to the detector quantum efficiency at the used wavelength and pupil size of the receiving optics.

[0187] The proposed imaging system 1 exploits the latest technological capability in order to have the sensitivity at the single-photon level and a millimetre resolution

[0188] In particular, the gating capability (time gate TG) described with reference to step S6 and used during the LOS imaging described with reference to step S7 may further improve noise rejection of the imaging system 1.

[0189] The proposed method and system thus allow to significantly improve imaging capabilities in non-conventional scenarios, both for LOS and NLOS scenarios, in particular where the target is hidden by obscurants or placed behind obstacles.

[0190] This allows the method and system of the invention to have a wide variety of applications.

[0191] The automotive market would benefit from the invention for the envisioned vehicles capability to perform autonomous guidance, active safety break and pedestrian detection even when fog or haze are present. Furthermore, the employment of the system on unmanned vehicles could be critical for rescue mission in dangerous or inaccessible environments where, for instance, the smoke of a fire limits the visibility of the rescuers.

[0192] Finally, it is clear that modifications and variations may be made to what has been described and illustrated herein, without thereby departing from the scope of the present invention, as defined in the annexed claims.

[0193] For example, the electronics control and processing resources described with reference to Figure 1 (that is, elements 3, 4, 15, 16, 17, 18) may comprise different elements from what shown and described, depending on the specific application.

[0194] Similarly, the light transmission system (5, 8) and light receiving system (6, 12) may comprise different elements from what shown and described, depending on the specific application.

[0195] For example, the optical transmission system 8 may comprise a number of mirrors lower or higher than two; the mirrors may not be rotating-mirrors but instead based on a different kind of displacement; and / or the optical transmission system 8 may be a different type of steering system, for example not based on reflective optical elements, but based (in addition or in alternative to reflective optical elements) on transmission optical elements.

[0196] For example, the optical receiving system 12 may comprise a number of mirrors lower or higher than two; the mirrors may not be rotating -mirrors but instead based on a different kind of displacement; and / or the optical receiving system 12 may be a different type of steering system, for example not based on reflective optical elements, but based (in addition or in alternative to reflective optical elements) on transmission optical elements.

[0197] For example, memory 4 and CPU 3 may be wired or wirelessly coupled with the circuitry and optics of the imaging system 1. In other words, motion sensors 20, light source 5, light detector 6 and respective driver 15, delayer 18, pulse generator 16, optical transmission system 8 and optical receiving system 12, may be integrated in the same device, whereas memory 4 and CPU 3 may be integrated in a separate device. Thus, CPU 3 and memory 4 may be configured to control the electronics and optics of the imaging system 1 remotely.

[0198] For example, the imaging system 1 may be fixed or portable, depending on the specific application.

[0199] Some of the blocks described with reference to Figure 1 may be implemented via hardware, software, or as mixed-signal electronics, depending on the specific application. For example, delayer 18, time tagger 17 and driver 15 may be implemented either via hardware or software.

[0200] Method 50 of Figure 2 may comprise different steps or the same steps but performed in a different order with respect to those illustrated and described above, depending on the specific application.

[0201] Step S3 may allow to increase accuracy of the imaging calibration 51 but may be optional. In this case, the raw dataset acquired at step S2 may be used directly as input to the feature analysis of step S4.

[0202] For example, a gating analogous to the one described with reference to step S6 may be used also during step S 11 of NLOS imaging. In this case, the time delay TD may be set as a function of the distance of the imaging system 1 with respect to the relay surface 61 and / or with respect to the obstacle 62.

[0203] For example, the LOS imaging described with reference to step S7 may be performed on a ROI that is not placed behind an obscurant, depending on the specific scenario of application.

[0204] Finally, the described embodiments may be combined to provide further solutions.

[0205] The above disclosure may be summarized by an imaging system (1) that comprises:

[0206] - a light transmission system (5, 8) configured to generate and steer an output light signal (LO);

[0207] - a light receiving system (6, 12) configured to detect an input light signal (LI); and

[0208] - electronic resources (3, 4, 15, 16, 17, 18) configured to control the light transmission system and the light receiving system and to perform imaging based on a time of flight between emission of the output light signal and detection of the input light signal, wherein the imaging system is configured to perform an imaging calibration (51) comprising:

[0209] - acquiring an environment map (ENV_MAP) that represents a three-dimensional image of an area scanned by the output light signal; and

[0210] - identifying, from the environment map, the presence of one or more objects (ENV_LOS, ENV_NLOS, 60, 61, 62) in the scanned area, wherein the imaging system is further configured to:

[0211] - perform (53) a Line-of-Sight, LOS, imaging of a region of interest within the scanned area (that is, the area that was scanned during the imaging calibration; in other words, the region of interest may be at least one of the objects identified within the scanned area); and

[0212] - perform (54) a Non-Line-of-Sight, NLOS, imaging by illuminating, as a relay surface, at least one of the objects (61) identified within the scanned area.

[0213] Moreover, the above disclosure may be summarized also by a corresponding method (and related computer program) for controlling the imaging system, wherein the method comprises performing an imaging calibration (51) that includes:

[0214] - acquiring an environment map (ENV_MAP) that represents a three- dimensional image of an area scanned by the output light signal; and

[0215] - identifying, from the environment map, the presence of one or more objects (ENV_LOS, ENV_NLOS, 60, 61, 62) in the scanned area, wherein the method further comprises receiving selection data indicative of a type of imaging to be performed and, based on the selection data: performing (53) a Line-of-Sight, LOS, imaging of a region of interest within the scanned area (that is, the area that was scanned during the imaging calibration; in other words, the region of interest may be at least one of the objects identified within the scanned area); or performing (54) a Non-Line-of-Sight, NLOS, imaging by illuminating, as a relay surface, at least one of the objects (61) identified within the scanned area.

[0216] Therefore, in other words, according to the above system and method, the LOS imaging of the region of interest may be performed after (e.g., in response to) the imaging calibration and may be based on the imaging calibration (in particular based on the environment map acquired during the imaging calibration and / or based on at least one of the objects identified in the scanned area).

Claims

CLAIMS1. An imaging system (1) comprising:- a light transmission system (5, 8) configured to generate and steer an output light signal (LO);- a light receiving system (6, 12) configured to detect an input light signal (LI); and- electronic resources (3, 4, 15, 16, 17, 18) configured to control the light transmission system and the light receiving system and to perform imaging based on a time of flight between emission of the output light signal and detection of the input light signal, wherein the imaging system is configured to perform an imaging calibration (51) comprising:- acquiring an environment map (ENV_MAP) that represents a three-dimensional image of an area scanned by the output light signal; and- identifying, from the environment map, the presence of one or more objects (ENV_LOS, ENV_NLOS, 60, 61, 62) in the scanned area, the imaging system being further configured to:- perform (53) a Line-of-Sight, LOS, imaging of a region of interest within the scanned area; and- perform (54) a Non-Line-of-Sight, NLOS, imaging by illuminating, as a relay surface, at least one of the objects (61) identified within the scanned area.

2. The imaging system according to claim 1, further comprising a motion sensing system (20) configured to detect a movement of the light transmission system (5, 8, 16) and / or of the light receiving system (6, 15, 12, 16, 18), the imaging system being configured to perform the imaging calibration in response to the detection of the movement.

3. The imaging system according to claim 1 or 2, wherein identifying the presence of one or more objects comprises extracting, from the environment map, a first set of features (ENV_NLOS) indicative of position and / or orientation of at least one surface, within the scanned area, that is suitable for being used as a relay surface during NLOS imaging, wherein performing the NLOS imaging comprises illuminating the at least onesurface with the output light signal.

4. The imaging system according to any of the preceding claims, wherein the imaging calibration further comprises converting the environment map based on a physical model of the light transmission system and / or light receiving system, so that the environment map is expressed in a spatial reference system (x-y-z) of the imaging system.

5. The imaging system according to the preceding claim, wherein the one or more objects are identified from the converted environment map.

6. The imaging system according to any of the preceding claims, wherein the imaging system is configured, for performing the LOS imaging, to: steer the output light signal over a plurality of scanning positions so as to illuminate an area that includes the region of interest; and delay detection of the input light signal by a time delay (TD) from emission of the output light signal, the time delay being a function of a distance (L) of the region of interest, in particular the time delay being equal to or higher than the time of flight corresponding to the distance of the region of interest.

7. The imaging system according to any of the preceding claims, wherein the imaging system is configured, for performing the NLOS imaging, to:- define a grid to be illuminated on the relay surface; and- illuminate the relay surface with the output light signal, based on the defined grid.

8. A method for controlling an imaging system (1) that is configured to perform time of flight-based imaging and comprises:- a light transmission system (5, 8) configured to generate and steer an output light signal (LO); and- a light receiving system (6, 12) configured to detect an input light signal (LI), the method comprising performing an imaging calibration (51) that includes:- acquiring an environment map (ENV_MAP) that represents a three-dimensional image of an area scanned by the output light signal; and- identifying, from the environment map, the presence of one or more objects (ENV_LOS, ENV_NLOS, 60, 61, 62) in the scanned area, the method further comprising receiving selection data indicative of a type of imaging to be performed and, based on the selection data: performing (53) a Line-of-Sight, LOS, imaging of a region of interest within the scanned area; or performing (54) a Non-Line-of-Sight, NLOS, imaging by illuminating, as a relay surface, at least one of the objects (61) identified within the scanned area.

9. The method according to the preceding claim, further comprising: detecting a movement of the imaging system, in particular a movement of the light transmission system and / or of the light receiving system; and perform the imaging calibration, in response to the detection of a movement of the imaging system.

10. The method according to claim 8 or 9, wherein identifying the presence of one or more objects comprises running, starting from the environment map, a surface recognition algorithm, for example a plane segmentation algorithm, configured to recognize a surface, within the scanned area, that is suitable for being used as a relay surface during NLOS imaging.

11. The method according to any of claims 8-10, wherein identifying the presence of one or more objects comprises running, starting from the environment map, an object recognition algorithm.

12. The method according to any of claims 8-11, wherein performing the imaging calibration, further comprises converting the environment map based on a physical model of the light transmission system and / or light receiving system, so that the environment map is expressed in a spatial reference system (x-y-z) of the imaging system.

13. The method according to any of claims 8-11, wherein performing the LOS imaging comprises: steer the output light signal over a plurality of scanning positions so as toilluminate an area that includes the region of interest; and for each scanning position, delay detection of the input light signal by a time delay (TD) from emission of the output light signal, the time delay being a function of a distance (L) of the region of interest.

14. The method according to any of claims 8-13, wherein performing the NLOS imaging comprises:- determining a grid to be illuminated on the relay surface; and- illuminating the relay surface with the output light signal, based on the defined grid.

15. A computer program comprising instructions that, when executed by electronic resources of an imaging system (1) that is configured to perform time of flightbased imaging, cause the imaging system to: perform an imaging calibration (51) that includes:- acquiring an environment map (ENV_MAP) that represents a three-dimensional image of an area scanned by an output light signal of the imaging system; and- identifying, from the environment map, the presence of one or more objects (ENV_LOS, ENV_NLOS, 60, 61, 62) in the scanned area, the instructions, when executed by the electronic resources, further causing the imaging system to receive selection data indicative of a type of imaging to be performed and, based on the selection data: to perform (53) a Line-of-Sight, LOS, imaging of a region of interest within the scanned area; or perform (54) a Non-Line-of-Sight, NLOS, imaging by illuminating, as a relay surface, at least one of the objects (61) identified within the scanned area.

Images