System and method for 3d-imaging using reconfigurable TOF sensor and peripheral sensor

Abstract

A system, wherein the system includes: a transmitter configured to illuminate a scene by emitting modulated light; a receiver configured to detect modulated light reflected in the scene to generate time-of- flight data; at least one peripheral sensor, each being configured to generate sensor data; a processor unit configured to: run an application that uses the time-of-flight data and the sensor data, and control, based on at least one of the time-of-flight data and the sensor data, a configuration of the transmitter and the receiver depending on the application.

Description

[0001] SYSTEM AND METHOD FOR 3D-IMAGING USING RECONFIGURABLE TOF SENSOR AND PERIPHERAL SENSOR

[0002] TECHNICAL FIELD

[0003] The present disclosure generally pertains to a system and a method.

[0004] TECHNICAL BACKGROUND

[0005] There is a need for three-dimensional (“3D”) imaging for allowing electronic devices to understand the environment to support various applications, for example, mobile electronic devices such as smart phones, smart watches and smart glasses may run an augmented reality (“AR”), a virtual reality (“VR”) or an extended reality (“XR”) application relying on 3D environment detection.

[0006] Generally, time-of-flight (“ToF”) systems are known which may be used for 3D imaging, since a ToF system is able to measure a distance to objects in a scene based on reflected illumination light.

[0007] Basically, two different techniques for ToF systems are known: direct ToF (“dToF”) and indirect ToF (“iToF”). In dToF systems, the distance is determined based on a time-of-arrival of a light pulse reflected at objects in the scene. In iToF systems, the scene is illuminated with a periodic light wave and a phase difference between emitted and reflected light wave is indicative for the distance.

[0008] Although there exist techniques for 3D imaging, it is generally desirable to improve the existing techniques.

[0009] SUMMARY

[0010] According to a first aspect, the disclosure provides a system comprising: a transmitter configured to illuminate a scene by emitting modulated light; a receiver configured to detect modulated light reflected in the scene to generate time-of- flight data; at least one peripheral sensor, each being configured to generate sensor data; a processor unit configured to: run an application that uses the time-of-flight data and the sensor data, and control, based on at least one of the time-of-flight data and the sensor data, a configuration of the transmitter and the receiver depending on the application.

[0011] According to a second aspect, the disclosure provides a method comprising: illuminating, by a transmitter, a scene by emitting modulated light; detecting, by a receiver, modulated light reflected in the scene to generate time-of-flight data; generating, by at least one peripheral sensor, sensor data; running an application that uses the time-of-flight data and the sensor data; and controlling, based on at least one of the time-of-flight data and the sensor data, a configuration of the transmitter and the receiver depending on the application.

[0012] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0013] BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0015] Fig. 1 schematically illustrates in Figs. 1 A-F in block diagrams various embodiments of a system;

[0016] Fig. 2 schematically illustrates in a block diagram an embodiment of a system;

[0017] Fig. 3 schematically illustrates in a block diagram an embodiment of a system;

[0018] Fig. 4 schematically illustrates in a block diagram an embodiment of a system;

[0019] Fig. 5 schematically illustrates in a block diagram an embodiment of a system;

[0020] Fig. 6 schematically illustrates in a block diagram an embodiment of a system;

[0021] Fig. 7 schematically illustrates in a block diagram an embodiment of a system; and

[0022] Fig. 8 schematically illustrates in a flow diagram an embodiment of a method.

[0023] DETAILED DESCRIPTION OF EMBODIMENTS

[0024] Before a detailed description of the embodiments under reference of Fig. 2 is given, general explanations are made.

[0025] As mentioned in the outset, there is a need for three-dimensional (“3D”) imaging for allowing electronic devices to understand the environment to support various applications, for example, mobile electronic devices such as smart phones, smart watches and smart glasses may run an augmented reality (“AR”), a virtual reality (“VR”) or an extended reality (“XR”) application relying on 3D environment detection. As further mentioned in the outset, generally, time-of-flight (“ToF”) systems are known which may be used for 3D imaging, since a ToF system is able to measure a distance to objects in a scene based on reflected illumination light.

[0026] For enhancing the general understanding of the present disclosure, various embodiments of a system 1-1 to 1-6 are discussed in the following under reference of Fig. 1, which schematically illustrates in Fig. 1 A-F the various embodiments in a block diagram, wherein the various embodiments also apply to other embodiments of the present disclosure.

[0027] Each of the systems 1-1 to 1-6 includes a transmitter 2 (which may also be referred to as ToF transmitter or intelligent illuminator in some embodiments), a receiver 3 (which may also be referred to as ToF receiver or programmable macropixel ToF sensor in some embodiments), at least one peripheral sensor 4 and a processor unit 5.

[0028] Generally, the transmitter 2 and the receiver 3 are able to perform a ToF measurement to generate ToF data, wherein the transmitter 2 illuminates a scene 6 by emitting modulated light (for example, an intensity of the emitted light may be temporally and spatially modulated) and the receiver 3 detects modulated light reflected in the scene 6 to generate the ToF data.

[0029] The transmitter 2 and the receiver 3 thus correspond to a ToF system which may be configured as a direct ToF (“dToF”) system or as an indirect ToF (“iToF”) system.

[0030] In some embodiments, the transmitter 2 includes a plurality of emitters, wherein each emitter is configured to emit temporally modulated light to the scene 6. Each emitter may be or may include, for example, a semiconductor laser such as a Vertical-Cavity Surface-Emitting Laser (“VCSEL”) or an edge emitting laser. Each emitter may be individually activated, and an amplitude of the emitted temporally modulated light may be individually controlled.

[0031] In some of such embodiments, the transmitter 2 illuminates the scene with a plurality of light spots. In other words, the transmitter 2 illuminates the scene 6 with a spatial light pattern of high- intensity and low-intensity areas.

[0032] In some embodiments, the receiver 3 includes a plurality of light detection pixels, wherein each light detection pixel is configured to detect temporally modulated light reflected in the scene 6. Each light detection pixel may be or may include, for example, an avalanche photodiode (“APD”), a single-photon avalanche diode (“SPAD”) or a current-assisted photonic demodulator (“CAPD”). Each light detection pixel may be individually activated. Each light detection pixel may be part of one of a plurality of macropixels, wherein each macropixel includes adjacent light detection pixels. In dToF systems, in some embodiments, the distance is determined based on a time-of-arrival of a light pulse emitted by an emitter of the transmitter 2 towards the scene 6 where the light pulse is at least partially reflected at objects in the scene 6. A time between two consecutive light pulses is typically divided into time intervals with equal spacing. In such embodiments, ToF data is generated by the receiver 3 in the form of a histogram for each light detection pixel of the receiver 3. The histogram represents a number of light detection events (e.g., detected photons) arrived in a particular time interval. This process may be repeated several times to increase a signal-to-noise (“SNR”) ratio.

[0033] In iToF systems, in some embodiments, an emitter of the transmitter 2 illuminates the scene 6 with a periodic light wave and a light detection pixel of the receiver 3 detects a phase difference between emitted and reflected light wave which is indicative for the distance. In some of such embodiments, ToF data is generated by the receiver 3 in four frames corresponding to four correlation measurements with different phase shifts (e.g., 0 degrees, 90 degrees, 180 degrees and 270 degrees) between a periodic light modulation signal applied to the emitter and a corresponding periodic demodulation signal applied to the light detection pixel. In some of such embodiments, the ToF data include pixel values of the plurality of light detection pixels of the receiver 3 of the four frames. Based on the captured four frames, component data (IQ values: Q is quadrature component, I is in-phase component) may be calculated which may be used to determine the phase and the distance. In some embodiments, ToF data includes component data.

[0034] The scene 6 is typically sampled all at once - i.e. with full possible spatial resolution of the transmitter 2 (all emitters are activated) and the receiver 3 (all light detection pixels are activated) - in known ToF systems.

[0035] The at least one peripheral sensor includes at least one of an image sensor (e.g., a red-green-blue (“RGB”) image sensor which may be configured as an active pixel sensor or a charge-coupled device (“CCD”) sensor), an event vision sensor (“EVS”), an inertial measurement unit (“IMU”) and a gaze detector or the like. Each peripheral sensor generates sensor data. In some cases, the at least one peripheral sensor may not be physically separated, but the additional data may be acquired by the ToF system itself. In other words, in some embodiments, the at least one peripheral sensor may be not be present or used.

[0036] The processor unit 5 may be or may include a CPU (“Central Processing Unit”), an application processor, a GPU (“Graphical Processing Unit”), an NPU (“Neural Processing Unit”), a DSP (“Digital Signal Processor”), an ASIC (“Application Specific Integrated Circuit”), a FPGA (“Field-Programmable Gate Array”), a microcontroller or the like. The processor unit 5 may include memory and one or more data bus interfaces.

[0037] The data exchange between the transmitter 2, the receiver 3, the at least one peripheral sensor 4 and the processer unit 5 may be implemented via various data bus interfaces such as I2C, I3C, SPI (“Serial Peripheral Interface”) or the like or an internal data bus (between integrated components).

[0038] As depicted in Fig. 1 A, the system 1-1 includes all components (i.e., the transmitter 2, the receiver 3, the at least one peripheral sensor 4 and the processer unit 5) as separate or individual integrated circuits.

[0039] As depicted in Fig. IB, the system 1-2 includes the transmitter 2 and the receiver 3 as a ToF system 7 implemented as a single integrated circuit. The system 1-2 further includes the at least one peripheral sensor 4 and the processer unit 5 as separate or individual integrated circuits.

[0040] As depicted in Fig. 1C, the system 1-3 includes the receiver 3 and the at least one peripheral sensor 4 as a receiver system 8 implemented as a single integrated circuit. The system 1-3 further includes the transmitter 2 and the processer unit 5 as separate or individual integrated circuits.

[0041] As depicted in Fig. ID, the system 1-4 includes the receiver 3, the at least one peripheral sensor 4 and the processor unit 5 as a receiver and processing system 9 implemented as a single integrated circuit. The system 1-4 further includes the transmitter 2 as a separate or individual integrated circuit.

[0042] As depicted in Fig. IE, the system 1-5 includes all components on a single integrated system circuit 10.

[0043] As depicted in Fig. IF, the system 1-6 includes the transmitter 2 and a transmitter part of the at least one peripheral sensor implemented as a single integrated transmitter circuit I la. The system 1-6 further includes the receiver 3 and a receiver part of the at least one peripheral sensor implemented as a single integrated receiver circuit 1 lb. The system 1-6 further includes the processor unit 5 as a separate or individual integrated circuit.

[0044] Returning to the general explanations, as mentioned above, a scene is typically sampled all at once - i.e. with full possible spatial resolution of the transmitter (all emitters are activated) and the receiver (all light detection pixels are activated) - in known ToF systems.

[0045] However, mobile electronic devices may require a low power consumption and the ToF system may be a power-hungry system due to the illumination. In some cases, the illumination may take above 50 percent of the total power consumption of the ToF system. Moreover, applications running on the mobile electronic device such as AR, VR and XR applications may require a high frame rate for fast response and high immersion depth. Additionally, an eye-safety limit may be reached or exceeded with when all emitters are activated.

[0046] It has been recognized that not all applications using ToF data may require the same precision and amount and type of information.

[0047] It has thus been recognized that sampling the scene with full possible spatial resolution at maximum power may generate redundant information or information which is not relevant in a particular situation for a particular application that uses the ToF data. Thereby, the mobile electronic device may unnecessarily waste power and more time than actually needed to keep the performance of the application.

[0048] Some systems may use maximum power for all dots as defined according to maximum specification of the device (i.e., maximum reachable range at lowest reflectivity to be supported by the specification). On the other hand, objects may be present at a distance smaller than maximum device specified range and / or with higher reflectivity. Hence, in some cases, some devices may consume redundant power that may be saved with control and device support.

[0049] It has thus been recognized to exploit the specificity of the application to reduce power consumption in some embodiments.

[0050] It has further been recognized that the information for the specificity may be gathered ToF data and sensor data from at least one peripheral sensor which enables dynamic mapping of the scene in spatial information and time information. Based on the specificity information, in some embodiments, a transmitter is programmed where to illuminate, and a receiver is programmed accordingly, in some embodiments, where to detect reflected light spots to optimize the power consumption by arranging emitted light spots with activated macropixels.

[0051] Hence, some embodiments pertain to a system, wherein the system includes: a transmitter configured to illuminate a scene by emitting modulated light; a receiver configured to detect modulated light reflected in the scene to generate ToF data; at least one peripheral sensor, each being configured to generate sensor data; a processor unit configured to: run an application that uses the time-of-flight data and the sensor data, and control, based on at least one of the time-of-flight data and the sensor data, a configuration of the transmitter and the receiver depending on the application.

[0052] The system may be implemented in or may be any form of device which is not particularly limited.

[0053] The system may be implemented in or may be, for example, a mobile electronic device which may be battery-powered, for example, a smart phone, smart glasses, a smart watch or a headmounted display.

[0054] The at least one peripheral sensor may be or may include at least one of an image sensor, an EVS, an IMU and a gaze detector.

[0055] The application may be or may be based on or may include an application for simultaneous localization and mapping (“SLAM”).

[0056] The application may be or may be based on or may include an automotive application such as an in-cabin monitoring application or an application for providing advanced driver assistance system (“ADAS”) functionality.

[0057] The application may be or may be based on or may include an AR, VR or VR application.

[0058] The application may be or may be based on or may include a facial or gesture recognition or authentication application.

[0059] The processor unit controls, based on at least one of the ToF data and the sensor data, a configuration of the transmitter and the receiver depending on the application.

[0060] As mentioned above under reference of Fig. 1, in some embodiments, the transmitter includes a plurality of emitters, wherein each emitter is configured to emit temporally modulated light to the scene.

[0061] In some embodiments, the control of the configuration of the transmitter includes determining and activating at least a subset of the plurality of emitters.

[0062] As mentioned above under reference of Fig. 1, in some embodiments, the receiver includes a plurality of light detection pixels, wherein each light detection pixel is configured to detect temporally modulated light reflected in the scene.

[0063] In some embodiments, the control of the configuration of the receiver includes determining and activating at least a subset of the plurality of light detection pixels. The processor unit may run or execute a control process in addition to the application process, for example, in an own thread to control the configuration of the transmitter and the receiver.

[0064] The control process may use an application identifier indicting the application that is executed to decide how the configuration of the transmitter and the receiver is to be determined.

[0065] The control process may use at least one of the ToF data and the sensor data on its own to determine the configuration of the transmitter and the receiver or the control process may pass at least one of the ToF data and the sensor data to the application process to determine the configuration of the transmitter and the receiver.

[0066] It has been recognized that different applications may have different requirements regarding the needed ToF information in a particular situation.

[0067] For some applications all objects in the scene may be of interest, while for other applications only specific objects may be of interest. For some applications only near or far objects are of interest. For some applications the changes in the scene may be of interest, for example, the changes of positions of the objects for object tracking.

[0068] The processor unit may thus be configured to perform at least object detection based on at least one of the ToF data and the sensor data. The processor unit may further be configured to perform, based on the results of the object detection, feature extraction or object recognition to filter the results of the object detection depending on the application. The processor unit may be configured to predict a movement of the detected objects based on at least one of the ToF data and the sensor data, wherein the data of a current frame or a past processed frame may be used.

[0069] The processor unit may thus be configured to use static and dynamic information about the scene to determine the configuration of the transmitter and the receiver.

[0070] Generally, different sensor data may be useful for different applications to determine the configuration of the transmitter and the receiver.

[0071] Hence, in some embodiments, the circuitry is further configured to select, depending on the application, the peripheral sensor whose sensor data are to be used for controlling the configuration of the transmitter and the receiver.

[0072] For example, some SLAM-based applications may require image data (e.g., representing a RGB image of the scene) to perform feature extraction and image segmentation. Some applications for analyzing and predicting dynamic scenes may require or may further use event data, since event data are complementary data to image data as the event data indicate changes in the scene and may thus be suitable, e.g., for predicting a movement of objects such that the configuration of the transmitter and the receiver may be adapted accordingly. Some applications may require a gaze direction of a user to determine which detected objects are of interest. Some applications may require a pose of the system to determine whether detected objects are aligned, for example, vertically or horizontally.

[0073] As mentioned above, different applications may have different requirements regarding the needed ToF information in a particular situation.

[0074] Hence, in some embodiments, the processor unit is further configured to: determine, based on at least one of the time-of-flight data and the sensor data, a region- of-interest depending on the application, and control the configuration of the transmitter and the receiver according to the region-of- interest.

[0075] Basically, the region-of-interest corresponds to a part of the scene that is associated with particular pixel coordinates (spatial information) of at least one of the receiver, the image sensor and the EVS. Moreover, the region-of-interest is further associated with a particular ToF and thus a particular distance or depth value (time information). The region-of-interest is thus associated with a particular position relative to the transmitter and the receiver.

[0076] The processor unit may determine a region-of-interest for each detected object or only for specific detected objects depending on the application.

[0077] As mentioned above, in some embodiments, the control of the configuration of the transmitter includes determining and activating at least a subset of the plurality of emitters and at least a subset of the plurality of light detection pixels.

[0078] In some embodiments, the subset of the plurality of emitters and the subset of the plurality of light detection pixels is determined according to the region-of-interest.

[0079] As each region-of-interest is associated with a particular position relative to the transmitter and the receiver, only some of the emitters emit modulated in the direction of the region-of-interest and only some of the light detection pixels detect the modulated light reflected in the scene.

[0080] Thus, in some embodiments, the processor unit is further configured to set an illumination region of the transmitter and a detection region of the receiver according to the region-of-interest. In some embodiments, the processor unit is further configured to set at least one of a spatial resolution and an amplitude of the emitted modulated light of the transmitter according to the region-of-interest.

[0081] The spatial resolution of the transmitter is basically given by the fraction of activated emitters relative to the total number of emitters that would be able to emit modulated light towards the region-of-interest.

[0082] In some embodiments, the circuitry is further configured to identify relevant geometrical shapes and configure the transmitter and the receiver with a reduced number of sampling points enabling reliable reconstruction.

[0083] In some embodiments, the processor unit is further configured to control the transmitter and the receiver to perform a first time-of-flight measurement with a first spatial resolution for determining a region-of-interest depending on the application and to perform a second time-of- flight measurement with a second spatial resolution in the region-of-interest for densifying a generated depth map, wherein the first spatial resolution is lower (or different) than the second spatial resolution.

[0084] In some embodiments, the processor unit is further configured to control the transmitter and the receiver to perform a time-of-flight measurement with a variable spatial resolution for determining a variety of regions-of-interest depending on the application.

[0085] The configuration of the receiver may further be controlled to account only for specific distances according to the region-of-interest to increase the SNR.

[0086] Hence, in some embodiments, the processor unit is further configured to set a time-gating window of the receiver according to the region-of-interest.

[0087] The SNR may be also increased after acquisition of the ToF data. The processor unit may exploit, for example, the ToF data in the region-of-interest to determine whether a signal intensity is low or whether the object is far away to determine whether the corresponding ToF data should be integrated.

[0088] Thus, in some embodiments, the processor unit is further configured to integrate the time-of- flight data according to the region-of-interest.

[0089] Hence, in some embodiments, the processor unit is further configured to determine groups of light detection pixels according to the region-of-interest and to integrate the time-of-flight data of each group of light detection pixels. In some embodiments, the processor unit is further configured to determine groups of light detection pixels according to an estimated required illumination time, an illumination result of a past group (regrouping), considerations of best resource usage or the like.

[0090] In the case of dToF systems, the integration corresponds to a bin-wise summation of the respective histograms. In the case of iToF systems, the integration corresponds to an average of, for example, the respective IQ values or phase values.

[0091] Some embodiments pertain to a (corresponding) method, wherein the method includes: illuminating, by a transmitter, a scene by emitting modulated light; detecting, by a receiver, modulated light reflected in the scene to generate time-of-flight data; generating, by at least one peripheral sensor, sensor data; running an application that uses the time-of-flight data and the sensor data; and controlling, based on at least one of the time-of-flight data and the sensor data, a configuration of the transmitter and the receiver depending on the application.

[0092] The method may be performed by the system as described herein.

[0093] In some embodiments, the method further includes: determining, based on at least one of the time-of-flight data and the sensor data, a region- of-interest depending on the application, and controlling the configuration of the transmitter and the receiver according to the region- of-interest.

[0094] In some embodiments, the method further includes setting an illumination region of the transmitter and a detection region of the receiver according to the region-of-interest.

[0095] In some embodiments, the method further includes setting at least one of a spatial resolution and an amplitude of the emitted modulated light of the transmitter according to the region-of-interest.

[0096] In some embodiments, the method further includes setting a time-gating window of the receiver according to the region-of-interest.

[0097] In some embodiments, the method further includes integrating the time-of-flight data according to the region-of-interest.

[0098] In some embodiments, the method further includes identifying relevant geometrical shapes and configuring the transmitter and the receiver with a reduced number of sampling points enabling reliable reconstruction. In some embodiments, the method further includes controlling the transmitter and the receiver to perform a first time-of-flight measurement with a variable first spatial resolution for determining a variety of regions-of-interest depending on the application.

[0099] In some embodiments, the method further includes controlling the transmitter and the receiver to perform a first time-of-flight measurement with a first spatial resolution for determining a region-of-interest depending on the application and to perform a second time-of-flight measurement with a second spatial resolution in the region-of-interest for densifying a generated depth map, wherein the first spatial resolution is lower than the second spatial resolution.

[0100] In some embodiments, the method further includes selecting, depending on the application, the peripheral sensor whose sensor data are to be used for controlling the configuration of the transmitter and the receiver.

[0101] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0102] Returning to Fig. 2, there is schematically illustrated in a block diagram an embodiment of a system 1-7, which is discussed in the following.

[0103] The system 1-7 may be based on any of the systems 1-1 to 1-6 of Fig. 1.

[0104] The system 1-7 includes the transmitter 2 (in Fig. 2 depicted as ToF transmitter), the receiver 3 (in Fig. 2 depicted as ToF receiver), the at least one peripheral sensor 4 and the processor unit 5.

[0105] The transmitter 2 and the receiver 3 perform a ToF measurement to obtain distances to objects in the scene 6, wherein the receiver 3 outputs the raw ToF data in data 13a to the processor unit 5 which may further output the raw ToF data or a depth map as data 13d to another processor unit or to a memory for storage for example.

[0106] In some embodiments, the transmitter 2 and the receiver 3 may have a relationship according to an auto-exposure capability (e.g., as described in European patent application EP22215978.2), which may also be referred to as JEP (“Just Enough Power”).

[0107] The at least one peripheral sensor 4 outputs the sensor data in data 13a to the processor unit 5, for example, the sensor data include image data representing an image of the scene 6. The processor unit 5 determines a first region-of-interest (“ROI”) 12a, a second ROI 12b and a third ROI 12c in the scene 6, based on at least one of the ToF data and the sensor data.

[0108] The determined ROIs 12a-c may be different depending on the application.

[0109] Then, the processor unit 5 determines data 13b and data 13c to control the configuration of the transmitter 2 and the receiver 3.

[0110] The data 13b indicate the transmitter 2 which emitters are to be activated in the next ToF measurement to illuminate the ROIs 12a-c by emitting modulated light.

[0111] In some embodiments, the processor unit 5 sets an illumination region of the transmitter 2 according to the ROIs 12a-c. In such embodiments, other regions of the scene 6 may not be illuminated by the transmitter 2.

[0112] In some embodiments, the processor unit 5 sets at least one of a spatial resolution and an amplitude of the emitted modulated light of the transmitter 2 according to the ROIs. In such embodiments, the spatial resolution in the ROIs may be higher or lower than in other regions of the scene 6 which may depend on the application. Moreover, in such embodiments, the amplitude of the emitted modulated light in the ROIs may be higher or lower than in other regions of the scene 6 which may depend on the application.

[0113] The data 13c indicate the receiver 3 which light detection pixels are to be activated in the next ToF measurement to detect modulated light reflected in the ROIs 12a-c.

[0114] Thus, the processor unit 5 defines accuracy, precision and spatial resolution in the ROIs 12a-c by controlling the transmitter 2 and the receiver 3 regarding where to expose the modulated light and how to set the light detection pixels, respectively.

[0115] Fig. 3 schematically illustrates in a block diagram an embodiment of a system 1-8, which is discussed in the following.

[0116] The system 1-8 may be based on any of the systems 1-1 to 1-7.

[0117] In this embodiment, the processor unit 5 is configured to control the transmitter 2 and the receiver 3 to perform a time-of-flight measurement with a variable spatial resolution for determining a variety of ROIs 12a-c depending on the application.

[0118] The transmitter 2 thus generates different spot densities in different regions of the scene. The receiver 3 then detects some ROIs 12a-c intensely and some sparsely.

[0119] The intelligent illuminator 2 uses reconfigurable emitters where the densification can be done programmatically and then acquired in a single shot or in multiple shots (ToF measurements). In some other embodiments, the processor unit 5 is configured to control the transmitter 2 and the receiver 3 to perform a first ToF measurement with a first spatial resolution for determining ROIs 12a-c depending on the application and to perform a second ToF measurement with a second spatial resolution in the ROIs 12a-c for densifying a generated depth map, wherein the first spatial resolution is lower than the second spatial resolution.

[0120] The processor unit 5 may, for example, determine where planes are located in the scene 6 which may be determined based on the data 13a and set the spatial resolution of the transmitter 2 higher in regions (here, the ROIs 12a-c) other than the planes.

[0121] Fig. 4 schematically illustrates in a block diagram an embodiment of a system 1-9, which is discussed in the following.

[0122] The system 1-9 may be based on any of the systems 1-1 to 1-8.

[0123] In this embodiment, the processor unit 5 is configured to set a time-gating window of the receiver 3 according to the determined ROIs 12a-c, wherein the determined ROIs 12a-c depend on the application.

[0124] Moreover, the processor unit 5 is configured to integrate the ToF data according to the determined ROIs 12a-c.

[0125] For example, the processor unit 5 exploits the ToF data to determine, e.g., whether spots are on an object with weak return light intensity, or on a far object which may require too long integration for a reasonable SNR. In such cases, the processor unit 5 may determine spot grouping and apply, for example, a binning of the measured information or a specific time-gating in order to look at a specific ToF window or distance range to increase the SNR.

[0126] Here, only the ToF data of the far object according to ROI 12c is integrated, e.g., binned in the case of dToF systems.

[0127] Hence, in some embodiments, the processor unit 5 is configured to determine groups of light detection pixels according to the ROIs 12a-c and to integrate the ToF data of each group of light detection pixels.

[0128] In the case of dToF systems, the integration corresponds to a bin-wise summation of the respective histograms. In the case of iToF systems, the integration corresponds to an average of, for example, the respective IQ values or phase values.

[0129] Fig. 5 schematically illustrates in a block diagram an embodiment of a system 1-10, which is discussed in the following. The system 1-10 may be based on any of the systems 1-1 to 1-9.

[0130] In this embodiment, the application is based on SLAM which is able to position a user in the environment and track the user’s trajectory.

[0131] In this embodiment, the processor unit 5 performs SLAM based on image data 13a-l from an image sensor as the at least one peripheral sensor 4, wherein the image data 13a-l represent a conventional image (e.g., RGB image). The processor unit 5 further performs the SLAM based on ToF data 13a-2 from the receiver 3 which is obtained in a ToF measurement performed by the transmitter 2 and the receiver 3. Various other possibilities to perform exist and the present disclosure is not limited to this particular form of SLAM.

[0132] Here, the SLAM is based on the identification of features in the image that are positioned in 3D space. The extracted features define the ROIs 12a-c.

[0133] The processor unit 5 activates emitters and light detection pixels in regions corresponding to the ROIs 12a-c that contain the features extracted and identified by the SLAM algorithm.

[0134] A typical ToF transmitter - such as the transmitter 2 - may have up to 1200 spots for example (more than 1200 spots may be used in other embodiments). With SLAM optimized for mobile electronic devices, the number of feature points may be in the order of 200-500 depending on the scene complexity. Considering that these features may typically be extracted in a VGA (“Video Graphics Array”) image (640x480), overlapping spots may exist for roughly 50% of the features. Therefore, from 100 to 250 spots may be necessary. A saving of 1200 / 250. . . 100 ~ 5x. . . lOx in power may be expected.

[0135] Fig. 6 schematically illustrates in a block diagram an embodiment of a system 1-11, which is discussed in the following.

[0136] The system 1-11 may be based on any of the systems 1-1 to 1-10.

[0137] In this embodiment, the application is a 3D scene reconstruction application.

[0138] A 3D reconstruction of the scene 6 may exploit the regularity in order to reduce the number of acquired points and as a result reduce power consumption. In particular, it is possible to exploit segmentation on a conventional image (represented by image data 13a-l, e.g., a RGB image) in order to identify possible planes.

[0139] The planes may further be identified based on the ToF data 13a-2 or in addition to the image data 13a-l. Planes need, in theory, only three ToF points to be univocally identified in space.

[0140] The identified planes correspond in this embodiment to determined ROIs 12a-c. The processor unit 5 activates emitters and light detection pixels in regions corresponding to the ROIs 12a-c that contain the identified planes.

[0141] The emitter and light detection pixel selection may be done by sampling a reduced number of points where planes are identified.

[0142] A typical ToF transmitter - such as the transmitter 2 - may have up to 1200 spots. With plane detection, for each segmented plane a number of three spots is the geometrical minimum, but roughly five to ten may be needed for better accuracy. Typical scenes segment up to ten major planes (up to 100 spots for mapping those planes). Adding the non-segmented areas (e.g., details, borders, small objects, etc.) the total needed spot number may add up to roughly 250 spots. A saving of 1200 / 250 ~ 5x in power may be expected. This approach may be extended to any other classifiable geometrical shape (even curved ones) that can be identified with the peripheral sensor and then intelligently sampled with the ToF sensor.

[0143] Fig. 7 schematically illustrates in a block diagram an embodiment of a system 1-12, which is discussed in the following.

[0144] The system 1-11 may be based on any of the systems 1-1 to 1-11.

[0145] In this embodiment, the application is a human vision reconstruction application.

[0146] Human eyes have different resolutions according to angular distance.

[0147] The Fovea region 12a has the highest resolution and corresponds to the region where the human directly looks at.

[0148] The Paracentral region 12b has less resolution than the Fovea region 12a.

[0149] The Peripheral region 12c has the lowest resolution.

[0150] To exploit the difference in resolution for improving the system efficiency, it is possible to use the so-called foveated detection, by identifying the point-of-view with a gaze detector (as the at least one peripheral sensor 4 providing gaze detector data 13a-3) in order to reconstruct images as seen by a human. Indeed, in the example of a smart glasses device, the display is rigid to the frame and oriented together with the head, while the used can aim his / her gaze at positions different from the center: therefore, by reconstructing with lower precision the scene far from the gaze point-of-view, a power / time saving is obtained without perceived reduction of accuracy.

[0151] The Fovea region 12a, the Paracentral region 12b and the Peripheral region 12c correspond in this embodiment to determined ROIs 12a-c. The processor unit 5 activates emitters and light detection pixels in regions corresponding to the ROIs 12a-c.

[0152] The emitter and light detection pixel selection may be done by reducing the density of the scene sampling both in the Paracentral region 12b and in the Peripheral region 12c of the effective eye field-of-view.

[0153] A typical ToF transmitter - such as the transmitter 2 - may have up to 1200 spots for example (more than 1200 spots may be used in other embodiments). With foveated segmentation of the spot patterns it may be possible to define about six regions (even though, perception-wise, three to four regions may be enough) having a maximum spatial sampling in the central part and progressively decreasing in density and / or required exposure time, e.g., 100 percent, 50 percent, 25 percent, etc. bringing to an equivalent of about 300-500 spots. A saving of 1200 / 500. . .300 ~ 2.5x...4x in power may be expected.

[0154] Fig. 8 schematically illustrates in a flow diagram an embodiment of a method 200, which is discussed in the following.

[0155] The method may be performed by the system as described herein.

[0156] At 201, by a transmitter, a scene is illuminated by emitting modulated light, as discussed herein.

[0157] At 202, by a receiver, modulated light reflected in the scene is detected to generate ToF data, as discussed herein.

[0158] At 203, by at least one peripheral sensor, sensor data is generated, as discussed herein.

[0159] At 204, an application is run that uses the ToF data and the sensor data, as discussed herein.

[0160] At 205, based on at least one of the ToF data and the sensor data, a configuration of the transmitter and the receiver is controlled depending on the application, as discussed herein.

[0161] The order of the method 200 may be changed, for example, 203 may be before 201.

[0162] Returning to the general explanations, some embodiments of the systems and methods as described herein may provide or may achieve at least one of the following:

[0163] The system may be implemented in or may be a mobile electronic device which may be battery- powered, for example, a smart phone, smart glasses, a smart watch or a head-mounted display.

[0164] The system may be implemented in or may be any form of device which is not particularly limited.

[0165] The application may be based on SLAM. Applications that use ToF data may include AR, VR or XR reality applications on battery- powered lightweight mobile electronic devices.

[0166] Applications that use ToF data may include automotive applications, real-time mapping applications and facial recognition or authentication applications.

[0167] Typical use cases include gaming, navigation and a virtual interface.

[0168] Power consumption may be reduced.

[0169] A number of exposures from the transmitter may be decreased which may reduce power consumption of the transmitter.

[0170] A smaller number of exposures may lead to less data which may reduce power consumption of a data bus interface.

[0171] A data throughput may be reduced and optimized, since data may be sampled only when and where needed.

[0172] Eye-safety may be increased.

[0173] Less laser power average may be used which may increase the eye-safety.

[0174] A flexibility in configuration may increase the eye-safety.

[0175] Changing an illumination density over the scene may allow further increase of density in some areas compared to typical illuminator that may pose challenges to eye safety.

[0176] A tradeoff with frame rate (higher) or accuracy / precision (better) is exploited.

[0177] A reduction of the number of exposures may complement a frame rate by increasing the number of frames by using the decreased the number of exposures; may allow an increase or optimization of the number of repetitions for each frame to sort out the peak more clearly; or may allow to optimize in terms of the power consumption, frame rate and SNR.

[0178] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding.

[0179] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software. In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0180] Note that the present technology can also be configured as described below.

[0181] (1) A system, wherein the system includes: a transmitter configured to illuminate a scene by emitting modulated light; a receiver configured to detect modulated light reflected in the scene to generate time-of- flight data; at least one peripheral sensor, each being configured to generate sensor data; a processor unit configured to: run an application that uses the time-of-flight data and the sensor data, and control, based on at least one of the time-of-flight data and the sensor data, a configuration of the transmitter and the receiver depending on the application.

[0182] (2) The system of (1), wherein the processor unit is further configured to: determine, based on at least one of the time-of-flight data and the sensor data, a region- of-interest depending on the application, and control the configuration of the transmitter and the receiver according to the region-of- interest.

[0183] (3) The system of (2), wherein the processor unit is further configured to set an illumination region of the transmitter and a detection region of the receiver according to the region-of- interest.

[0184] (4) The system of (2) or (3), wherein the processor unit is further configured to set at least one of a spatial resolution and an amplitude of the emitted modulated light of the transmitter according to the region-of-interest.

[0185] (5) The system of anyone of (2) to (4), wherein the processor unit is further configured to set a time-gating window of the receiver according to the region-of-interest.

[0186] (6) The system of anyone of (2) to (5), wherein the processor unit is further configured to integrate the time-of-flight data according to the region-of-interest.

[0187] (7) The system of anyone of (1) to (6), wherein the processor unit is further configured to control the transmitter and the receiver to perform a time-of-flight measurement with a variable spatial resolution for determining a variety of regions-of-interest depending on the application. (8) The system of anyone of (1) to (7), wherein the application includes an application for simultaneous localization and mapping.

[0188] (9) The system of anyone of (1) to (8), wherein the at least one peripheral sensor includes at least one of an image sensor, an event vision sensor, an inertial measurement unit and a gaze detector.

[0189] (10) The system of anyone of (1) to (9), wherein the circuitry is further configured to identify relevant geometrical shapes and configure the transmitter and the receiver with a reduced number of sampling points enabling reliable reconstruction.

[0190] (11) A method, wherein the method includes: illuminating, by a transmitter, a scene by emitting modulated light; detecting, by a receiver, modulated light reflected in the scene to generate time-of-flight data; generating, by at least one peripheral sensor, sensor data; running an application that uses the time-of-flight data and the sensor data; and controlling, based on at least one of the time-of-flight data and the sensor data, a configuration of the transmitter and the receiver depending on the application.

[0191] (12) The method of (11), further including: determining, based on at least one of the time-of-flight data and the sensor data, a region- of-interest depending on the application, and controlling the configuration of the transmitter and the receiver according to the region- of-interest.

[0192] (13) The method of (12), further including setting an illumination region of the transmitter and a detection region of the receiver according to the region-of-interest.

[0193] (14) The method of (12) or (13), further including setting at least one of a spatial resolution and an amplitude of the emitted modulated light of the transmitter according to the region-of- interest.

[0194] (15) The method of anyone of (12) to (14), further including setting a time-gating window of the receiver according to the region-of-interest.

[0195] (16) The method of anyone of claim (12) to (15), further including integrating the time-of- flight data according to the region-of-interest. (17) The method of anyone of (11), further including controlling the transmitter and the receiver to perform a time-of-flight measurement with a variable spatial resolution for determining a variety of regions-of-interest depending on the application.

[0196] (18) The method of anyone of (11) to (17), wherein the application includes an application for simultaneous localization and mapping.

[0197] (19) The method of anyone of (11) to (18), wherein the at least one peripheral sensor includes at least one of an image sensor, an event vision sensor, an inertial measurement unit and a gaze detector.

[0198] (20) The method of anyone of (11) to (19), further including identifying relevant geometrical shapes and configuring the transmitter and the receiver with a reduced number of sampling points enabling reliable reconstruction.

[0199] (21) A computer program comprising program code causing a computer to perform the method according to anyone of (11) to (20), when being carried out on a computer.

[0200] (22) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to anyone of (11) to (20) to be performed.

Claims

CLAIMS1. A system comprising: a transmitter configured to illuminate a scene by emitting modulated light; a receiver configured to detect modulated light reflected in the scene to generate time-of- flight data; at least one peripheral sensor, each being configured to generate sensor data; a processor unit configured to: run an application that uses the time-of-flight data and the sensor data, and control, based on at least one of the time-of-flight data and the sensor data, a configuration of the transmitter and the receiver depending on the application.

2. The system of claim 1, wherein the processor unit is further configured to: determine, based on at least one of the time-of-flight data and the sensor data, a region- of-interest depending on the application, and control the configuration of the transmitter and the receiver according to the region-of- interest.

3. The system of claim 2, wherein the processor unit is further configured to set an illumination region of the transmitter and a detection region of the receiver according to the region-of-interest.

4. The system of claim 2 or 3, wherein the processor unit is further configured to set at least one of a spatial resolution and an amplitude of the emitted modulated light of the transmitter according to the region-of-interest.

5. The system of claim 2, wherein the processor unit is further configured to set a timegating window of the receiver according to the region-of-interest.

6. The system of claim 2, wherein the processor unit is further configured to integrate the time-of-flight data according to the region-of-interest.

7. The system of claim 1, wherein the processor unit is further configured to control the transmitter and the receiver to perform a time-of-flight measurement with a variable spatial resolution on the scene for determining a variety of regions-of-interest depending on the application.

8. The system of claim 1, wherein the application includes an application for simultaneous localization and mapping.

9. The system of claim 1, wherein the at least one peripheral sensor includes at least one of an image sensor, an event vision sensor, an inertial measurement unit and a gaze detector.

10. The system of claim 1, wherein the circuitry is further configured to identify relevant geometrical shapes and configure the transmitter and the receiver with a reduced number of sampling points enabling reliable reconstruction.

11. A method comprising: illuminating, by a transmitter, a scene by emitting modulated light; detecting, by a receiver, modulated light reflected in the scene to generate time-of-flight data; generating, by at least one peripheral sensor, sensor data; running an application that uses the time-of-flight data and the sensor data; and controlling, based on at least one of the time-of-flight data and the sensor data, a configuration of the transmitter and the receiver depending on the application.

12. The method of claim 11, further comprising: determining, based on at least one of the time-of-flight data and the sensor data, a region- of-interest depending on the application, and controlling the configuration of the transmitter and the receiver according to the region- of-interest.

13. The method of claim 12, further comprising setting an illumination region of the transmitter and a detection region of the receiver according to the region-of-interest.

14. The method of claim 12 or 13, further comprising setting at least one of a spatial resolution and an amplitude of the emitted modulated light of the transmitter according to the region-of-interest.

15. The method of claim 12, further comprising setting a time-gating window of the receiver according to the region-of-interest.

16. The method of claim 12, further comprising integrating the time-of-flight data according to the region-of-interest.

17. The method of claim 11, further comprising controlling the transmitter and the receiver to perform a time-of-flight measurement with a variable spatial resolution on the scene for determining a variety of regions-of-interest depending on the application.

18. The method of claim 11, wherein the application includes an application for simultaneous localization and mapping.

19. The method of claim 11, wherein the at least one peripheral sensor includes at least one of an image sensor, an event vision sensor, an inertial measurement unit and a gaze detector.

20. The method of claim 11, further comprising identifying relevant geometrical shapes and configuring the transmitter and the receiver with a reduced number of sampling points enabling reliable reconstruction.

Images