Controller, time-of-flight device, mobile electronic device and control method

Abstract

A controller for a time-of-flight device, wherein the controller includes circuitry configured to select, based on confidence values of current depth accumulation buffer data, a subset of light emitters from a plurality of light emitters of a transmitter of the time-of-flight device for activation in a next time-of-flight measurement.

Description

[0001] CONTROLLER, TIME-OF-FLIGHT DEVICE, MOBILE ELECTRONIC DEVICE AND CONTROL METHOD

[0002] TECHNICAL FIELD

[0003] The present disclosure generally pertains to a controller for a time-of-flight device, a time-of- flight device, a mobile electronic device and a control method.

[0004] TECHNICAL BACKGROUND

[0005] Some applications which are run by mobile electronic devices may use a range sensor in addition to other sensor input to understand the environment, for example, augmented reality (“AR”), virtual reality (“YR”) and simultaneous localization and mapping (“SLAM”) may use depth information from a range sensor in addition to image information from an image sensor.

[0006] Generally, time-of-flight (“ToF”) devices are known range sensors which provide depth information, since a ToF device is able to measure a distance to objects in a scene based on directly reflected illumination light.

[0007] Basically, two different techniques for ToF devices are known: direct ToF (“dToF”) and indirect ToF (“iToF”). In dToF devices, the distance is determined based on a time-of-arrival of a light pulse reflected at objects in the scene. In iToF systems, the scene is illuminated with a periodically modulated light wave and a phase difference between emitted and directly reflected light wave is indicative for the distance.

[0008] ToF devices may have desirable characteristics in terms of accuracy, ambient light robustness and depth detection range.

[0009] However, ToF devices require an active illumination which may, in some cases, consumes a large fraction of the overall consumed power of the ToF devices.

[0010] Although there exist techniques for time-of-flight devices, it is generally desirable to improve the existing techniques.

[0011] SUMMARY

[0012] According to a first aspect, the disclosure provides a controller for a time-of-flight device, wherein the controller comprises circuitry configured to select, based on confidence values of current depth accumulation buffer data, a subset of light emitters from a plurality of light emitters of a transmitter of the time-of-flight device for activation in a next time-of-flight measurement. According to a second aspect, the disclosure provides a time-of-flight device, comprising: a transmitter, wherein the transmitter includes a plurality of light emitters, and wherein each light emitter is configured to emit a spot of temporally modulated light to a scene; a receiver, wherein the receiver includes a plurality of light detection pixels, and wherein each light detection pixel is configured to detect temporally modulated light reflected in the scene for generating time-of-flight data based on which depth accumulation buffer data are to be updated; a controller, wherein the controller includes circuitry configured to select, based on confidence values of current depth accumulation buffer data, a subset of light emitters from the plurality of light emitters of a transmitter of the time-of-flight device for activation in a next time-of-flight measurement.

[0013] According to a third aspect, the disclosure provides a mobile electronic device, comprising a time-of-flight device, wherein the time-of-flight device includes: a transmitter, wherein the transmitter includes a plurality of light emitters, and wherein each light emitter is configured to emit a spot of temporally modulated light to a scene; a receiver, wherein the receiver includes a plurality of light detection pixels, and wherein each light detection pixel is configured to detect temporally modulated light reflected in the scene for generating time-of-flight data based on which depth accumulation buffer data are to be updated; a controller, wherein the controller includes circuitry configured to select, based on confidence values of current depth accumulation buffer data, a subset of light emitters from the plurality of light emitters of a transmitter of the time-of-flight device for activation in a next time-of-flight measurement.

[0014] According to a fourth aspect, the disclosure provides a control method for a time-of-flight device, wherein the method comprises selecting, based on confidence values of current depth accumulation buffer data, a subset of light emitters from a plurality of light emitters of a transmitter of the time-of-flight device for activation in a next time-of-flight measurement.

[0015] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0016] BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0018] Fig. 1 schematically illustrates in a block diagram an embodiment of a mobile electronic device; Fig. 2 schematically illustrates in a block diagram an embodiment of a mobile electronic device;

[0019] Fig. 3 schematically illustrates in a flow diagram an embodiment of a method for merging previous depth accumulation buffer data and current depth buffer data to obtain current depth accumulation buffer data; and

[0020] Fig. 4 schematically illustrates in a flow diagram an embodiment of a control method.

[0021] DETAILED DESCRIPTION OF EMBODIMENTS

[0022] Before a detailed description of the embodiments under reference of Fig. 2 is given, general explanations are made.

[0023] As mentioned in the outset, some applications which are run by mobile electronic devices may use a range sensor in addition to other sensor input to understand the environment, for example, augmented reality (“AR”), virtual reality (“VR”) and simultaneous localization and mapping (“SLAM”) may use depth information from a range sensor in addition to image information from an image sensor.

[0024] Generally, time-of-flight (“ToF”) devices are known ranges sensors which provide depth information, since a ToF device is able to measure a distance to objects in a scene based on directly reflected illumination light.

[0025] Basically, two different techniques for ToF devices are known: direct ToF (“dToF”) and indirect ToF (“iToF”). In dToF devices, the distance is determined based on a time-of-arrival of a light pulse reflected at objects in the scene. In iToF systems, the scene is illuminated with a periodically modulated light wave and a phase difference between emitted and directly reflected light wave is indicative for the distance.

[0026] ToF devices may have desirable characteristics in terms of accuracy, ambient light robustness and depth detection range.

[0027] However, ToF devices require an active illumination which may, in some cases, consumes a large fraction of the overall consumed power of the ToF devices.

[0028] For enhancing the general understanding of the present disclosure, an embodiment of a mobile electronic device 1 is discussed in the following under reference of Fig. 1, which schematically illustrates the embodiment, wherein the discussion generally applies also to other embodiments of the present disclosure.

[0029] The mobile electronic device 1 includes a ToF device 2, a processor 6, a memory 7 and a sensor 8. The ToF device 2 includes a controller 3, a transmitter 4 and a receiver 5.

[0030] The ToF device 2 may be configured as a dToF device or as an iToF device.

[0031] The ToF device 2 is able to exchange data with the processor 6 via a data bus interface 9 (e.g., a data bus interface in accordance with MIPI (“Mobile Industry Processor Interface”) specifications).

[0032] Generally, the transmitter 4 and the receiver 5 are able to perform a ToF measurement to generate ToF data under the control of the controller 3, wherein the transmitter 4 illuminates a scene 12 by emitting spots of temporally modulated light to the scene 12. The spots are sometimes known as dots.

[0033] For example, an intensity of the emitted light may be temporally modulated in the form of a train of single light pulses in the case of a dToF device or the emitted light intensity may be periodically and continuously modulated in the case of an iToF device.

[0034] The transmitter 4 thus illuminates the scene 12 with a spatial light pattern of high-intensity 14 and low intensity 15 areas, wherein the intensity is temporally modulated.

[0035] The transmitter 4 thus includes a plurality of light emitters, wherein each light emitter is configured to emit a spot of temporally modulated light to the scene 12. Each light emitter may be or may include, for example, a semiconductor laser such as a Vertical -Cavity Surface- Emitting Laser (“VCSEL”) or an edge emitting laser. Each light emitter may be individually activated, and an amplitude or intensity of the emitted light may be individually controlled. The plurality of light emitters may be arranged in an array or matrix, wherein each light emitter may have an address in the array or matrix.

[0036] The receiver 5 detects the temporally modulated light reflected at objects 13a and 13b in the scene 12.

[0037] The receiver 5 thus includes a plurality of light detection pixels, wherein each light detection pixel is configured to detect temporally modulated light reflected in the scene 12 for generating ToF data. Each light detection pixel may be or may include, for example, an avalanche photodiode (“APD”), a single-photon avalanche diode (“SPAD”) or a current-assisted photonic demodulator (“CAPD”). Each light detection pixel may be individually activated. The plurality of light detection pixels may be arranged in an array or matrix, wherein each light detection pixel may have an address in the array or matrix.

[0038] Each light emitter of the transmitter 4 is associated with one or more different light detection pixels of the receiver 5. The one or more different light detection pixels may be spatially adjacent light detection pixels, e.g., a block of 1x2, 2x2, 2x3, 3x3 adjacent light detection pixels. Each light emitter covers a certain field-of-illumination (“FOI”) with respect to the ToF device 2 and each light detection pixel covers a certain field-of-view (“FOV”) with respect to the ToF device 2. Typically, a parallax between transmitter 4 and receiver 5 may be small such that a FOI of a certain light emitter may almost entirely overlap with a FOV of one or more certain light detection pixels such that, when temporally modulated light emitted by the certain light emitter is reflected by a typical diffuse scattering object in the scene 12, the reflected light may typically be incident on the one or more certain light detection pixels. Accordingly, the certain light emitter and the one or more certain light detection pixels may be activated and controlled synchronously in a ToF measurement, since they are associated with each other. The association is preset according to the layout and may be calibrated experimentally by a calibration measurement or by simulation.

[0039] As mentioned above, in dToF devices, in some embodiments, a distance is determined based on a time-of-arrival of a light pulse emitted by a light emitter of the transmitter 4 towards the scene 12 where the light pulse is at least partially reflected at the objects 13a and 13b in the scene 12.

[0040] A time between two consecutive light pulses is typically divided into time intervals with equal spacing. In such embodiments, ToF data is generated by the receiver 5 in the form of a histogram for each light detection pixel of the receiver 5. The histogram represents a number of light detection events (e g., detected photons) arrived in a particular time interval. This process may be repeated several times to increase a signal-to-noise (“SNR”) ratio.

[0041] As mentioned above, in iToF devices, in some embodiments, a light emitter of the transmitter 4 illuminates the scene 12 with a periodic light wave and a light detection pixel of the receiver 5 detects a phase difference between emitted and reflected light wave which is indicative for the distance.

[0042] In some of such embodiments, ToF data is generated by the receiver 5 in four frames corresponding to four correlation measurements with different phase shifts (e.g., 0 degrees, 90 degrees, 180 degrees and 270 degrees) between a periodic light modulation signal applied to the light emitter and a corresponding periodic demodulation signal applied to the light detection pixel. This process may be repeated several times to increase a signal-to-noise (“SNR”) ratio.

[0043] In some of such embodiments, the ToF data include pixel values of the plurality of light detection pixels of the receiver 5 of the four frames. Based on the captured four frames, component data (IQ values: Q is the quadrature component, I is the in-phase component) may be calculated which may be used to determine the phase and the distance. In some embodiments, ToF data includes component data.

[0044] The controller 3 includes a depth accumulation buffer 10 for storing depth accumulation buffer data and a depth buffer 11 for storing current depth buffer data.

[0045] The current or previous depth accumulation buffer data include accumulated depth values and accumulated confidence values for each light detection pixel for a current or previous FOV of the ToF device 2, respectively.

[0046] The current depth buffer data include depth values and confidence values for each light detection pixel for the current FOV of the ToF device 2 which are obtained based on ToF data obtained in the current (i.e., the latest) performed ToF measurement.

[0047] A(n) (accumulated) depth value may be a distance (e g., in millimeter, centimeter, meter or the like) or may be a digital value representing a fraction of a maximum distance range. A(n) (accumulated) depth value may be a certain digital value indicating that a distance could not be obtained, for example, if the corresponding confidence value is too low.

[0048] A(n) (accumulated) confidence value may be a value between zero and one, wherein a zero may indicate low confidence and one may indicate high confidence.

[0049] In some embodiments, the controller 3 obtains the depth accumulation buffer data based on the depth buffer data and the previous depth buffer accumulation data which are obtained based on ToF data in previous performed ToF measurements, wherein the controller 3 performs a spot accumulation algorithm in such embodiments as will be discussed further below.

[0050] In other embodiments, the controller 3 obtains the depth accumulation buffer data from an external source, e.g., from the processor 6 which has performed the spot accumulation algorithm instead of the controller 3.

[0051] In other embodiments, the controller 3 stores the current depth buffer data as the current depth accumulation buffer data and a spot accumulation algorithm is not performed.

[0052] As mentioned above, each light emitter of the transmitter 4 is associated with one or more different light detection pixels of the receiver 5. Thus, the depth accumulation buffer data include a plurality of different data sections, wherein each data section includes depth values and confidence values for one or more light detection pixels, wherein the one or more light detection pixels are associated with a certain light emitter. A data section may relate to spatially adjacent light detection pixels, but when represented in a buffer a data section may not be stored as adjacent or sequential data and may be referred to, for example, by a pointer.

[0053] The processor 6 may execute an application (e.g., SLAM application) which uses the depth information and accumulates the depth information globally, for instance, in a point cloud with a fixed coordinate system initialized at a starting point in time and space, for example, the starting point in time may relate to the time point when the application is started and the starting point in space may relate to the position and the pose of the mobile electronic device 1 at that time point, wherein the starting point in space may correspond to the origin of the fixed coordinate system.

[0054] The processor 6 performs a pose change estimation, based on or further based on information from the sensor 8 (e.g., an image sensor, an acceleration sensor, a gyroscope, an Inertial Measurement Unit (“IMU”) or a combination thereof) of the mobile electronic device 1 and thus of the ToF device 2 for accumulating the depth information in the point cloud.

[0055] The processor 6 transmits the pose change estimation to the controller 3, in some embodiments, for performing the spot accumulation algorithm to accumulate the depth information in a current local coordinate system corresponding to the current FOV.

[0056] As depicted in Fig. 1, the ToF device 2 performs the first ToF measurement of a sequence of ToF measurements with all light emitters activated.

[0057] Returning to the general explanations, typically, however, the scene is sampled in its entirety for a given FOV in each ToF measurement such that all light emitters of the transmitter and all light detection pixels of the receiver are activated in each ToF measurement, which may lead to a high power consumption in some cases.

[0058] It has been recognized that a high power consumption of a ToF device may be mitigated by accumulating depth information and by refreshing the accumulation based on a selection of areas of a FOV for which depth information is to be reassessed.

[0059] In other words, it has been recognized that the transmitter may emit light with only a (real) subset of the light emitters for reducing the power consumption, wherein such light emitters are selected which may result in sufficient information for keeping the quality of depth information at or above a certain level.

[0060] Hence, some embodiments pertain to a controller for a ToF device, wherein the controller includes circuitry configured to select, based on confidence values of current depth accumulation buffer data, a subset of light emitters from a plurality of light emitters of a transmitter of the ToF device for activation in a next ToF measurement.

[0061] Some embodiments pertain to a ToF device, wherein the ToF includes: a transmitter, wherein the transmitter includes a plurality of light emitters, and wherein each light emitter is configured to emit a spot of temporally modulated light to a scene; a receiver, wherein the receiver includes a plurality of light detection pixels, and wherein each light detection pixel is configured to detect temporally modulated light reflected in the scene for generating ToF data based on which depth accumulation buffer data are to be updated; a controller, wherein the controller includes circuitry configured to select, based on confidence values of current depth accumulation buffer data, a subset of light emitters from the plurality of light emitters of a transmitter of the ToF device for activation in a next ToF measurement.

[0062] Some embodiments pertain to a mobile electronic device, wherein the mobile electronic device includes a ToF device, wherein the ToF device includes: a transmitter, wherein the transmitter includes a plurality of light emitters, and wherein each light emitter is configured to emit a spot of temporally modulated light to a scene; a receiver, wherein the receiver includes a plurality of light detection pixels, and wherein each light detection pixel is configured to detect temporally modulated light reflected in the scene for generating ToF data based on which depth accumulation buffer data are to be updated; a controller, wherein the controller includes circuitry configured to select, based on confidence values of current depth accumulation buffer data, a subset of light emitters from the plurality of light emitters of a transmitter of the ToF device for activation in a next ToF measurement.

[0063] The mobile electronic device may be a smartphone, smart glasses, a virtual reality device, a head mounted display, a tablet computer, a laptop or the like.

[0064] The ToF device may be a dToF device or an iToF device.

[0065] The controller may be separate circuitry or may be embedded in the transmitter or receiver or may be provided externally to the ToF device.

[0066] The circuitry may be or may be implemented by or may include one or more processors. A processor may be or may include an application processor, a central processing unit (“CPU”), a graphical processing unit (“GPU”), a digital signal processor (“DSP”), a field-programmable gate array (“FPGA”), an application specific integrated circuit (“ASIC”) etc. The circuitry may be or may be implemented by or may include one or more memory components. The depth buffer and depth accumulation buffer as described herein may be configured by one or more of such memory components, and the depth buffer data and depth accumulation buffer data as described herein may be stored by such memory components. A memory component may be or may include volatile and non-volatile memory such as static random-access memory (“SRAM”), dynamic RAM (“DRAM”), non-volatile RAM (“NVRAM”), read-only memory (“ROM”), programmable ROM (“PROM”), electrically PROM (“EPROM”), electrically erasable PROM (“EEPROM”), flash memory (e.g., NOR flash or NAND flash) etc. A memory component may be or may include one or more registers, latch circuits, flip-flops, caches, main memories, hard disk drives, solid-state drives etc.

[0067] The circuitry may include one or more data bus interfaces configured to exchange signals or data, e.g., with other components of the circuitry. The data bus interface may, for example, be configured in accordance with MIPI specifications, I2C (“Inter-Integrated Circuit”), 13 C or the like. The circuitry may include one or more communications or network interfaces configured to communicate with external information processing devices, for example, via a wired connection such as Ethernet or a wireless connection such as a local area network (“LAN”), a wireless local area network (“WLAN”), a mobile telecommunications system (“GSM”, “UMTS”, “LTE”, “NR” etc.), Bluetooth, etc.

[0068] The circuitry may be or may be implemented by or may include typical electronic components configured to achieve the functions as described herein. The functionality may be implemented by hardware. The functionality may be implemented in parts by hardware and in parts by software. The functionality may be implemented by software.

[0069] The control method is based on a spot selection algorithm or light emitter selection algorithm, as will be discussed in the following.

[0070] The spot selection algorithm uses the confidence values of the current depth accumulation buffer data for selecting such light emitters for activation in a next ToF measurement which may provide new depth information or improve reliability of existing depth information, for example, where the confidence is low or where depth information is missing.

[0071] The spot selection algorithm, in some embodiments, includes using the confidence values associated with a first ToF measurement and determining, based on the confidence values associated with the first ToF measurement, the subset of light emitters for a second ToF measurement. Some embodiments pertain to a controller for a ToF device, wherein the controller includes circuitry configured to determine, based on confidence values associated with a first ToF measurement, a subset of light emitters from a plurality of light emitters of a transmitter of the ToF device for activation in a second ToF measurement.

[0072] In some of such embodiments, the second ToF measurement immediately follows the first ToF measurement without an intermediate ToF measurement.

[0073] In some embodiments, the circuitry is further configured to: compute a confidence score for each different data section of the current depth accumulation buffer data, wherein each data section is associated with a different light emitter; select, based on the computed confidence score, a subset of data sections to select the subset of light emitters.

[0074] The confidence score may be an average confidence value, a minimum confidence value or the like.

[0075] In some embodiments, the circuitry selects the data sections with confidence scores below a preset confidence threshold. In some of such embodiments, the circuitry selects the data sections with the lowest confidence scores below the preset confidence threshold.

[0076] Hence, in some of such embodiments, the subset of light emitters is selected which illuminates parts of the scene where the confidence of the acquired depth information may be further improved or where depth information is missing or not available. In the other parts of the scene the depth information may have already been acquired with high confidence such that the corresponding light emitters may not need to be activated in order to decrease a power consumption.

[0077] However, it has been recognized that some high confidence spots may be required to be measured as well, for example, the confidence must be as high as possible or for robustness to verify the estimated depth information in the previous depth accumulation buffer data.

[0078] Hence, in some embodiments, the circuitry is further configured to select a second subset of data sections of the current depth accumulation buffer data with confidence scores above the preset confidence threshold to select the subset of light emitters.

[0079] In some of such embodiments, the circuitry selects the data sections with the highest average confidence values above the preset confidence threshold.

[0080] It has been recognized that, however, some parts of the scene do not reflect much or any light, since, for example, the object is too far away or too dark or there is no object present at all. In such cases the controller may tend to always select the corresponding light emitters, since the corresponding data sections may tend to have the lowest confidence. Thus, the controller may tend to always select the same light emitters which may waste power and may not provide further depth information.

[0081] It has thus been recognized that a cooldown mechanism may be introduced.

[0082] Hence, in some embodiments, every time a light emitter has been previously selected, an extra boost is added to the confidence score of the corresponding data section in the current selection procedure. In this way, even if the data section has a low confidence score, it may not be selected in the current selection procedure due to the boost in response to the previous selection.

[0083] Accordingly, in some embodiments, the circuitry is further configured to add a preset value to the confidence score of a previously selected data section before selecting the subset of data sections.

[0084] In some embodiments, the preset value decays over time.

[0085] Hence, the preset value added to the confidence score is reduced before each further ToF measurement until it has cooled down (for example reaches zero).

[0086] In some embodiments, the circuitry selects a preset number of light emitters.

[0087] In such embodiments, the selected number of light emitters is constant in every ToF measurement.

[0088] In some embodiments, the circuitry selects a varying number of light emitters, wherein a moving average of the selected numbers converges towards a preset number.

[0089] In such embodiments, the selected number of light emitters may not be constant in every ToF measurement.

[0090] In some embodiments, the circuitry is further configured to adapt a number of selected light emitters based on the confidence values of the current depth accumulation buffer data.

[0091] In such embodiments, the selected number of light emitters may not be constant in every ToF measurement.

[0092] Moreover, in such embodiments, the (moving) average number of selected light emitters may decrease over time, for example, in the case where there is little dynamic in the scene such that many high confidence data sections may be present. Similar, if there are many low confidence data sections, the (moving) average number of selected light emitters may increase over time. In some embodiments, the circuitry is further configured to select all of the plurality of light emitters for the first time-of-flight measurement.

[0093] In some embodiments, the circuitry is further configured to select periodically all of the plurality of light emitters or a preset subset of light emitters. The periodic selection may be according to a regular time interval or may be from time to time not necessarily according to a regular time interval.

[0094] It has been recognized that the power consumption may be further reduced by synchronizing transmitter and receiver operation, i.e. by activating only the light detection pixels associated with the selected light emitters.

[0095] Hence, in some embodiments, the circuitry is further configured to select a subset of light detection pixels from a plurality of light detection pixels of a receiver of the time-of-flight device for activation in the next ToF measurement according to the selected subset of light emitters.

[0096] The control method is, in some embodiments, further based on a depth accumulation algorithm, as will be discussed in the following.

[0097] The depth accumulation algorithm may provide a full-FOV depth estimate based on the sparse data, since data of previous ToF measurements may be combined with the data of a current ToF measurement, based on a pose change estimation of the ToF device.

[0098] Moreover, typically, the depth measurements may not vary so fast such that a previously measured depth value of a light detection pixel may be present in the vicinity on another light detection pixel in a following ToF measurement.

[0099] Hence, in some embodiments, the circuitry is further configured to obtain the current depth accumulation buffer data based on previous depth accumulation buffer data and current depth buffer data.

[0100] The previous depth accumulation buffer data, as mentioned above under reference of Fig. 1, include accumulated depth values and accumulated confidence values for each light detection pixel for a previous FOV of the ToF device.

[0101] The current depth buffer data, as mentioned above under reference of Fig. 1, include depth values and confidence values for each light detection pixel for the current FOV of the ToF device which are obtained based on ToF data obtained in the current (i.e., the latest) performed ToF measurement.

[0102] In some embodiments, the circuitry is further configured to: obtain a pose change estimation of the time-of-flight device; reproject the previous depth accumulation buffer data according to the obtained pose change estimation; and obtain the current depth accumulation buffer data by merging the reprojected previous depth accumulation buffer data and the current depth buffer data.

[0103] The reprojection of the previous depth accumulation buffer data is generally known and will be discussed further below under reference of Fig. 3.

[0104] In some embodiments, the merging includes computing a weighted average of depth values of the reprojected previous depth accumulation buffer data and depth values of the current depth buffer data for overlapping pixel coordinates to obtain the depth values of the current depth accumulation buffer data, wherein the weights are based on the respective confidence values, as will be discussed further below under reference of Fig. 3.

[0105] The confidence values of the reprojected previous depth accumulation buffer data may be decayed, e.g. reduced by a preset decay value, before computing the weighted average of depth values due to the uncertainty of the pose change estimation.

[0106] In some embodiments, the circuitry is further configured to obtain current ToF data and to compute confidence values in the current depth buffer data in accordance with a confidence metric, a confidence metric threshold and a distance to a detected spot pixel coordinate before the merging, as will be discussed further below under reference of Fig. 3.

[0107] The confidence metric is based on ToF data obtained in the current ToF measurement, i.e. the latest performed ToF measurement.

[0108] In some embodiments, the circuitry is further configured to: obtain current ToF data, detect, based on a confidence metric, spots at spot pixel coordinates in the current ToF data; and interpolate depth values of the detected spots to obtain the depth values in the current depth buffer data, as will be discussed further below under reference of Fig. 3

[0109] Some embodiments pertain to a (corresponding) control method for a ToF device, wherein the method includes selecting, based on confidence values of current depth accumulation buffer data, a subset of light emitters from a plurality of light emitters of a transmitter of the ToF device for activation in a next ToF measurement. Some embodiments pertain to a (corresponding) control method for a ToF device, wherein the control method includes determining, based on confidence values associated with a first ToF measurement, a subset of light emitters from a plurality of light emitters of a transmitter of the ToF device for activation in a second ToF measurement.

[0110] The control method may be performed by the controller as described herein.

[0111] In some embodiments, the control method further includes outputting a control signal indicating which light emitters are selected and to be activated in the next ToF measurement.

[0112] In some embodiments, the control method further includes obtaining an input describing the arrangement of the plurality of light emitters and the address of each light emitter in the arrangement. For example, the plurality of light emitters may be arranged in an array of size MxN (M is an Integer and corresponds to the number of rows; N is an Integer and corresponds to the number of columns) and the address represents the data for activating and driving the respective light emitter in the array.

[0113] In some embodiments, the control method further includes outputting a control signal indicating which light detection pixels are selected and to be activated in the next ToF measurement.

[0114] In some embodiments, the control method further includes obtaining an input describing the arrangement of a plurality of light detection pixels and the address of each light detection pixel in the arrangement. For example, the plurality of light detection pixels may be arranged in an array of size MxN (M is an Integer and corresponds to the number of rows; N is an Integer and corresponds to the number of columns) and the address represents the data for activation and readout of the respective light emitter in the array.

[0115] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0116] Returning to Fig. 2, which schematically illustrates in a block diagram an embodiment of the mobile electronic device 1, which is discussed under reference of Fig. 2, Fig. 3 and Fig. 4.

[0117] The mobile electronic device 1 corresponds to the mobile electronic device 1 of Fig. 1, wherein the Fig. 2 illustrates a ToF measurement with a subset of the light emitters of the transmitter 4. The controller 3 has selected, based on confidence values of the current depth accumulation buffer data, the subset of light emitters from the plurality of light emitters of the transmitter 4 of the ToF device 2 for activation in the next ToF measurement.

[0118] The controller 3 performs a control method including a spot accumulation algorithm and a spot selection algorithm, as will be discussed in the following under reference of Fig. 3 and Fig. 4.

[0119] An embodiment of a method 40 for merging previous depth accumulation buffer data and current depth buffer data to obtain current depth accumulation buffer data is schematically illustrated in a flow diagram in Fig. 3, which is discussed in the following.

[0120] At 41, the controller 3 obtains the current ToF data.

[0121] At 42, the controller 3 obtains the current depth buffer data, wherein the current depth buffer data is obtained based on the ToF data obtained in the current ToF measurement.

[0122] The controller 3 calculates a confidence metric for each light detection pixel such as, for example, an intensity of the reflected temporally modulated light signal on the respective light detection pixel. For example, the intensity may be calculated based on the IQ values in the case of an iToF device as generally known or based on the number of detected light detection events in a bin of the histogram in the case of a dToF device as generally known.

[0123] When the confidence metric is above a first confidence metric threshold, a spot is detected at certain spot pixel coordinates (e.g., the pixel coordinates may correspond to a row number and a column number in an array of light detection pixels) in the current ToF data.

[0124] When the confidence metric is further above a second confidence metric threshold, the confidence value for the spot may be set by the controller 3 to one in the current depth buffer data. If not, the confidence value in the current depth buffer data is set by the controller 3 between zero and one depending on a ratio of the confidence metric and the second confidence metric threshold.

[0125] The confidence values for light detection pixels between detected spots may be computed based on distance to the detected spot. In this way, the confidence values are densified and spatially decayed for larger distances to a detected spot.

[0126] Moreover, the controller 3 calculates, based on the current ToF data, depth values for the detected spots at the spot pixel coordinates.

[0127] Then, the controller 3 interpolates the depth values (e.g., Delaunay Triangulation) of the detected spots to obtain the depth values in the current depth buffer data for each light detection pixel. However, if the confidence value is too low, a depth value may not be rendered for a light detection pixel.

[0128] At 43, the controller 3 obtains the previous depth accumulation buffer data.

[0129] At 44, the controller 3 obtains a pose change estimation and reprojects the previous depth accumulation buffer data according to the obtained pose change estimation.

[0130] Generally, the reprojection method is known.

[0131] The pose change estimation is represented by a rotation R and a translation t.

[0132] The controller 3 computes, for each light detection pixel, a position of the pixel in the three- dimensional space based on the pixel coordinate, the depth value and the camera intrinsic parameters.

[0133] Then, the controller 3 applies the transformation: p' = R ■ p + t, wherein p is the three- dimensional position of the respective point before the transformation.

[0134] Then, the controller 3 projects the point back into the two-dimensional space. As a result, the controller 3 obtains reprojected pixel coordinates in the current FOV for the previous depth accumulation buffer data.

[0135] Then, at 45, the controller 3 determines whether the reprojected pixel coordinates overlap with the pixel coordinates of the current depth buffer data.

[0136] Moreover, the reprojected pixel coordinates may be rounded and dilated by one pixel before checking whether there is an overlap with the pixel coordinates of the current depth buffer data.

[0137] Then, the controller 3 computes a weighted average of depth values of the reprojected previous depth accumulation buffer data and depth values of the current depth buffer data for overlapping pixel coordinates to obtain the depth values of the current depth accumulation buffer data, wherein the weights are based on the respective confidence values.

[0138] The depth values for non-overlapping pixel coordinates correspond to the depth values of the current depth buffer data.

[0139] In some embodiments, the merging further includes computing the confidence values of the current depth accumulation buffer data based on the confidence values of the reprojected previous depth accumulation buffer data and the current depth buffer data for overlapping pixel coordinates. For example, the controller 3 may compute an average or choose the higher confidence value of the both. The confidence values for non-overlapping pixel coordinates correspond to the confidence values of the current depth buffer data.

[0140] Thus, at 46, the controller 3 obtains the current depth accumulation buffer data.

[0141] An embodiment of a control method 50 is schematically illustrated in a flow diagram in Fig. 4, which is discussed in the following.

[0142] At 51, the controller 3 obtains the current depth accumulation buffer data, for example, according to the method 40 of Fig. 3.

[0143] Then, at 52, the controller 3 obtains the confidence values for each data section of the current depth accumulation buffer data.

[0144] At 53, the different data sections of the current depth accumulation buffer data are associated with different light emitters of the transmitter 4. The process 53 is typically performed during calibration before 51 and 52 and typically performed only once. However, the association data is stored in the controller 3 such that the controller 3 can obtain the association information on-the- fly.

[0145] At 54, the controller 3 computes a confidence score for each different data section, for example, an average confidence value for each data section based on the confidence values in the respective data section.

[0146] At 55, the controller 3 adds a preset value to the confidence scores of a previously selected data section.

[0147] Moreover, the controller 3 reduces a previously added preset value in the respective data sections. Thereby, the added preset value decays over time.

[0148] At 56, the controller 3 sorts the data sections according to the confidence score.

[0149] At 57, the controller 3 selects, based on the confidence score, a subset of the data sections. For example, the controller 3 selects a preset number of the data sections with the lowest confidence score below a preset confidence threshold.

[0150] At 58, the controller 3 selects the subset of light emitters according to the selected subset of data sections.

[0151] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding. All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0152] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0153] Note that the present technology can also be configured as described below.

[0154] (1) A controller for a time-of-flight device, wherein the controller includes circuitry configured to select, based on confidence values of current depth accumulation buffer data, a subset of light emitters from a plurality of light emitters of a transmitter of the time-of-flight device for activation in a next time-of-flight measurement.

[0155] (2) The controller of (1), wherein the circuitry is further configured to: compute a confidence score for each different data section of the current depth accumulation buffer data, wherein each data section is associated with a different light emitter; select, based on the computed confidence score, a subset of data sections to select the subset of light emitters.

[0156] (3) The controller of (2), wherein the circuitry selects the data sections with confidence scores below a preset confidence threshold.

[0157] (4) The controller of (3), wherein the circuitry is further configured to select a second subset of data sections of the current depth accumulation buffer data with confidence scores above the preset confidence threshold to select the subset of light emitters.

[0158] (5) The controller of anyone of (1) to (4), wherein the circuitry is further configured to select a subset of light detection pixels from a plurality of light detection pixels of a receiver of the time-of-flight device for activation in the next time-of-flight measurement according to the selected subset of light emitters.

[0159] (6) The controller of anyone of (2) to (5), wherein the circuitry is further configured to add a preset value to the confidence scores of a previously selected data section before selecting the subset of data sections.

[0160] (7) The controller of (6), wherein the preset value decays over time. (8) The controller of anyone of (1) to (7), wherein the circuitry selects a preset number of light emitters.

[0161] (9) The controller of anyone of (1) to (8), wherein the circuitry selects a varying number of light emitters, wherein a moving average of the selected numbers converges towards a preset number.

[0162] (10) The controller of anyone of (1) to (9), wherein the circuitry is further configured to adapt a number of selected light emitters based on the confidence values of the current depth accumulation buffer data.

[0163] (11) The controller of anyone of (1) to (10), wherein the circuitry is further configured to select all of the plurality of light emitters for the first time-of-flight measurement.

[0164] (12) The controller of anyone of (1) to (11), wherein the circuitry is further configured to select periodically all of the plurality of light emitters or a preset subset of light emitters.

[0165] (13) The controller of anyone of (1) to (12), wherein the circuitry is further configured to obtain the current depth accumulation buffer data based on previous depth accumulation buffer data and current depth buffer data.

[0166] (14) The controller of (13), wherein the circuitry is further configured to: obtain a pose change estimation of the time-of-flight device; reproject the previous depth accumulation buffer data according to the obtained pose change estimation; and obtain the current depth accumulation buffer data by merging the reprojected previous depth accumulation buffer data and the current depth buffer data.

[0167] (15) The controller of (14), wherein the merging includes computing a weighted average of depth values of the reprojected previous depth accumulation buffer data and depth values of the current depth buffer data for overlapping pixel coordinates to obtain the depth values of the current depth accumulation buffer data, wherein the weights are based on the respective confidence values.

[0168] (16) The controller of (14) or (15), wherein the circuitry is further configured to obtain current ToF data and to compute confidence values in the current depth buffer data in accordance with a confidence metric, a confidence metric threshold and a distance to a detected spot pixel coordinate before the merging.

[0169] (17) The controller of anyone of (14) to (16), wherein the circuitry is further configured to: obtain current ToF data; detect, based on a confidence metric, spots at spot pixel coordinates in the current ToF data; and interpolate depth values of the detected spots to obtain the depth values in the current depth buffer data.

[0170] (18) A time-of-flight device, wherein the time-of-flight device includes: a transmitter, wherein the transmitter includes a plurality of light emitters, and wherein each light emitter is configured to emit a spot of temporally modulated light to a scene; a receiver, wherein the receiver includes a plurality of light detection pixels, and wherein each light detection pixel is configured to detect temporally modulated light reflected in the scene for generating time-of-flight data based on which depth accumulation buffer data are to be updated; a controller, wherein the controller includes circuitry configured to select, based on confidence values of current depth accumulation buffer data, a subset of light emitters from the plurality of light emitters of a transmitter of the time-of-flight device for activation in a next time-of-flight measurement.

[0171] (19) A mobile electronic device, wherein the mobile electronic device includes a time-of-flight device, wherein the time-of-flight device includes: a transmitter, wherein the transmitter includes a plurality of light emitters, and wherein each light emitter is configured to emit a spot of temporally modulated light to a scene; a receiver, wherein the receiver includes a plurality of light detection pixels, and wherein each light detection pixel is configured to detect temporally modulated light reflected in the scene for generating time-of-flight data based on which depth accumulation buffer data are to be updated, a controller, wherein the controller includes circuitry configured to select, based on confidence values of current depth accumulation buffer data, a subset of light emitters from the plurality of light emitters of a transmitter of the time-of-flight device for activation in a next time-of-flight measurement.

[0172] (20) A control method for a time-of-flight device, wherein the method includes selecting, based on confidence values of current depth accumulation buffer data, a subset of light emitters from a plurality of light emitters of a transmitter of the time-of-flight device for activation in a next time-of-flight measurement.

[0173] (21) A computer program comprising program code causing a computer to perform the control method of (20), when being carried out on a computer. (22) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the control method of (20) to be performed.

Claims

CLAIMS1. A controller for a time-of-flight device, wherein the controller comprises circuitry configured to select, based on confidence values of current depth accumulation buffer data, a subset of light emitters from a plurality of light emitters of a transmitter of the time-of-flight device for activation in a next time-of-flight measurement.

2. The controller of claim 1, wherein the circuitry is further configured to: compute a confidence score for each different data section of the current depth accumulation buffer data, wherein each data section is associated with a different light emitter; select, based on the computed confidence score, a subset of data sections to select the subset of light emitters.

3. The controller of claim 2, wherein the circuitry selects the data sections with confidence scores below a preset confidence threshold.

4. The controller of claim 3, wherein the circuitry is further configured to select a second subset of data sections of the current depth accumulation buffer data with confidence scores above the preset confidence threshold to select the subset of light emitters.

5. The controller of claim 1, wherein the circuitry is further configured to select a subset of light detection pixels from a plurality of light detection pixels of a receiver of the time-of-flight device for activation in the next time-of-flight measurement according to the selected subset of light emitters.

6. The controller of claim 2, wherein the circuitry is further configured to add a preset value to the confidence scores of a previously selected data section before selecting the subset of data sections.

7. The controller of claim 6, wherein the preset value decays over time.

8. The controller of claim 1, wherein the circuitry selects a preset number of light emitters.

9. The controller of claim 1, wherein the circuitry selects a varying number of light emitters, wherein a moving average of the selected numbers converges towards a preset number.

10. The controller of claim 1, wherein the circuitry is further configured to adapt a number of selected light emitters based on the confidence values of the current depth accumulation buffer data.

11. The controller of claim 1, wherein the circuitry is further configured to select all of the plurality of light emitters for the first time-of-flight measurement.

12. The controller of claim 1, wherein the circuitry is further configured to select periodically all of the plurality of light emitters or a preset subset of light emitters.

13. The controller of claim 1, wherein the circuitry is further configured to obtain the current depth accumulation buffer data based on previous depth accumulation buffer data and current depth buffer data.

14. The controller of claim 13, wherein the circuitry is further configured to: obtain a pose change estimation of the time-of-flight device; reproject the previous depth accumulation buffer data according to the obtained pose change estimation; and obtain the current depth accumulation buffer data by merging the reprojected previous depth accumulation buffer data and the current depth buffer data.

15. The controller of claim 14, wherein the merging includes computing a weighted average of depth values of the reprojected previous depth accumulation buffer data and depth values of the current depth buffer data for overlapping pixel coordinates to obtain the depth values of the current depth accumulation buffer data, wherein the weights are based on the respective confidence values.

16. The controller of claim 14, wherein the circuitry is further configured to obtain current time-of-flight data and to compute confidence values in the current depth buffer data in accordance with a confidence metric, a confidence metric threshold and a distance to a detected spot pixel coordinate before the merging.

17. The controller of claim 14, wherein the circuitry is further configured to: obtain current time-of-flight data; detect, based on a confidence metric, spots at spot pixel coordinates in the current time- of-flight data; and interpolate depth values of the detected spots to obtain the depth values in the current depth buffer data.

18. A time-of-flight device, comprising: a transmitter, wherein the transmitter includes a plurality of light emitters, and wherein each light emitter is configured to emit a spot of temporally modulated light to a scene; a receiver, wherein the receiver includes a plurality of light detection pixels, and wherein each light detection pixel is configured to detect temporally modulated light reflected in the scene for generating time-of-flight data based on which depth accumulation buffer data are to be updated;a controller, wherein the controller includes circuitry configured to select, based on confidence values of current depth accumulation buffer data, a subset of light emitters from the plurality of light emitters of a transmitter of the time-of-flight device for activation in a next time-of-flight measurement.

19. A mobile electronic device, comprising a time-of-flight device, wherein the time-of-flight device includes: a transmitter, wherein the transmitter includes a plurality of light emitters, and wherein each light emitter is configured to emit a spot of temporally modulated light to a scene; a receiver, wherein the receiver includes a plurality of light detection pixels, and wherein each light detection pixel is configured to detect temporally modulated light reflected in the scene for generating time-of-flight data based on which depth accumulation buffer data are to be updated; a controller, wherein the controller includes circuitry configured to select, based on confidence values of current depth accumulation buffer data, a subset of light emitters from the plurality of light emitters of a transmitter of the time-of-flight device for activation in a next time-of-flight measurement.

20. A control method for a time-of-flight device, wherein the method comprises selecting, based on confidence values of current depth accumulation buffer data, a subset of light emitters from a plurality of light emitters of a transmitter of the time-of-flight device for activation in a next time-of-flight measurement.

Images