Circuitry and method

Abstract

The disclosure pertains to circuitry that is configured to: obtain a sequence of image data samples captured by a single photon imaging sensor; determine, based on the sequence of image data samples, a flux frequency component of a light source; and determine a segment in an image based on the determined flux frequency component of the light source.

Description

[0001] CIRCUITRY AND METHOD

[0002] TECHNICAL FIELD

[0003] The present disclosure generally pertains to a circuitry and a method.

[0004] TECHNICAL BACKGROUND

[0005] It is generally known to acquire image data samples (e.g., image data frames) by a single-photon imaging sensor, e.g., by a quanta image sensor (QIS), and to perform processing of the acquired image data samples.

[0006] Although there exist techniques for acquiring and processing image data samples, it is generally desirable to provide an improved circuitry and method.

[0007] SUMMARY

[0008] According to a first aspect, the disclosure provides circuitry that is configured to: obtain a sequence of image data samples captured by a single photon imaging sensor; determine, based on the sequence of image data samples, a flux frequency component of a light source; and determine a segment in an image based on the determined flux frequency component of the light source.

[0009] According to a second aspect, the disclosure provides a method that includes: obtaining a sequence of image data samples captured by a single photon imaging sensor; determining, based on the sequence of image data samples, a flux frequency component of a light source; and determining a segment in an image based on the determined flux frequency component of the light source.

[0010] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0011] BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0013] Fig. 1 illustrates an embodiment of a circuitry;

[0014] Fig. 2 illustrates an embodiment of a method for determining a segment in an image;

[0015] Fig. 3 illustrates a transform from a time domain to a frequency domain according to an embodiment;

[0016] Fig. 4 illustrates a flux spectrum block according to an embodiment; Fig. 5 illustrates a difference between a color balance based on conventional statistics and a color balance based on QIS-based segmentation according to an embodiment

[0017] Fig. 6 illustrates an AWB pipeline according to an embodiment;

[0018] Fig. 7 illustrates a first embodiment of an AWB;

[0019] Fig. 8 illustrates a second embodiment of an AWB;

[0020] Fig. 9 illustrates a flow for a color balance according to an embodiment;

[0021] Fig. 10 illustrates a LED flickering problem;

[0022] Fig. 11 illustrates an example of a multi-photodiode pixel;

[0023] Fig. 12 illustrates a pipeline of a QIS-based LED flickering solution according to an embodiment;

[0024] Fig. 13 illustrates a ToF system according to an embodiment; and

[0025] Fig. 14 illustrates an embodiment of a general -purpose computer.

[0026] DETAILED DESCRIPTION OF EMBODIMENTS

[0027] Before a detailed description of the embodiments under reference of Fig. 1 is given, general explanations are made.

[0028] As mentioned in the outset, image data samples (e.g., image data frames) may be acquired by a single-photon imaging sensor, e g., by a quanta image sensor (QIS) and / or by a photon counting (PC) sensor, and processing of the acquired image data samples may be performed. The image data samples may be acquired at high sample rates, e g., 103, 104, 105, 106, 107,or even more image data samples per second (without limiting the disclosure to these values or to this range).

[0029] It has been recognized that such high sample rates may allow a detection of light flux variations caused by light sources. Frequency components of the light flux variations may be determined, e.g., if the light flux variations satisfy the Nyquist-Shannon sampling theorem.

[0030] For example, artificial light sources may have a non-fixed light flux, e.g., the light flux may have small, periodical variations, e.g., due to variations in their power source.

[0031] For example, an electricity source (e.g., mains power) may have 50 Hz, 60 Hz and / or 120 Hz fluctuations. Thus, light sources that are directly driven from such an electricity source may exhibit a corresponding flickering.

[0032] For example, light-emitting diode (LED) bulbs may be fed by pulse-width modulation (PWM) drivers and, thus, may exhibit periodical fluctuations in orders of MHz. For example, monitors and / or video projectors may have a refresh rate of 50 Hz, 60 Hz, 120 Hz, 144 Hz, or the like.

[0033] A single photon imaging sensor (e.g., quanta image sensor (QIS), photon counting (PC) sensor etc.) may acquire image data samples at a sample rate far beyond a Nyquist frequency associated with a flickering of the light sources described above. Thus, image data samples from the single photon imaging sensor may be used to estimate a flux spectrum of light sources using signal processing techniques (e.g., Fourier transform).

[0034] This concept may be utilized for a single photon imager with flux spectrum estimation, and for applications that may use this estimation for enhancing an image quality for viewing and sensing applications.

[0035] Consequently, some embodiments of the present disclosure pertain to circuitry that is configured to: obtain a sequence of image data samples captured by a single photon imaging sensor; determine, based on the sequence of image data samples, a flux frequency component of a light source; and determine a segment in an image based on the determined flux frequency component of the light source.

[0036] The circuitry may include a processing unit, for example, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmed microprocessor or the like. The processing unit may control an overall function of the circuitry and / or may perform the determining of the flux frequency component and / or the determining of the segment in the image.

[0037] The circuitry may also include a memory, for example, dynamic random-access memory (DRAM), static random-access memory (SRAM), resistive random-access memory (RRAM) or the like. The memory may store instructions (e.g., software, firmware) that may control a processing of the processing unit. The memory may store the image data samples captured by the single photon imaging sensor.

[0038] The circuitry may further include a communication unit for communicating with an external device external to the circuitry, for example, via a protocol provided by the Mobile Industry Processor Interface (MIPI) Alliance, e.g., Camera Serial Interface (CSI), via Peripheral Component Interconnect (PCI), via serial interface (RS-232), via parallel interface (IEEE 1284), via Universal Serial Bus (USB), or the like.

[0039] Some embodiments pertain to a method that includes: obtaining a sequence of image data samples captured by a single photon imaging sensor; determining, based on the sequence of image data samples, a flux frequency component of a light source; and determining a segment in an image based on the determined flux frequency component of the light source.

[0040] The method may be performed by the circuitry and may be configured similar to the circuitry. Any feature described herein with respect to the circuitry may correspond to a feature of the method.

[0041] The circuitry includes the single photon imaging sensor in some embodiments. Thus, the determining of the flux frequency component and of the segment of the image may be performed on-chip, and the communication unit may transmit to the external device image data samples of the sequence of image data samples, an image generated based on the sequence of image data samples, an indication of the determined segment in an image, and / or further data generated by the circuitry (e.g., an image, a color balance parameter, a control command, as described herein). In some cases, the processing unit and the memory may be combined as a compute-in-memory (CIM) module.

[0042] The circuitry is provided separately from the single photon imaging sensor in some embodiments. Thus, the communication unit may receive the sequence of image data samples from the single photon imaging sensor, the processing unit may perform the determining of the flux frequency component and of the segment of the image, and the communication unit may output data generated by the circuitry (e.g., the determined flux frequency component(s), the determined image segment(s), an image, a color balance parameter, a control command (e.g., for the single photon imaging sensor), as described herein). For example, the circuitry may include a general-purpose computer, as described with respect to Fig. 14.

[0043] The single photon imaging sensor may be configured as a quanta imaging sensor (QIS), as a photon counting (PC) sensor, as direct time-of-flight (dToF) sensor (which may pe part of a dToF system) or the like. The single photon imaging sensor may include a plurality of single photon detection elements (e g., pixels, jots) that may be based on a complementary metal-oxide- semiconductor (CMOS) imaging sensor (CIS) and / or on a single-photon avalanche diode (SPAD). For example, the single photon imaging sensor may include a SPAD. The single photon detection elements may be arranged in a one-dimensional (ID) or two-dimensional (2D) array, and may be configured to generate a photon detection signal when detecting a photon.

[0044] For capturing the image data samples, the single photon imaging sensor may acquire photon detection signals from the plurality of single photon detection elements and may generate the image data samples based on the photon detection signals. For example, the image data samples may be configured as image data frames, and each image data frame of the sequence of image data frames may be associated with a time interval and may include a plurality of pixels. To each pixel of the image data frame, the single photon imaging sensor may assign a value that indicates whether (according to the photon detection signals) a photon has been detected by one or more single photon detection element(s) associated with the pixel within the time interval associated with the image data frame. An image data frame whose pixels merely indicate whether or not a photon has been detected (e.g., a binary frame) may be referred to as a quanta frame. In some embodiments, however, the single photon imaging sensor assigns a photon count to the pixels, such that the pixels indicate how many photons have been detected by the single photon detection element(s) associated with the pixel within the time interval associated with the image data frame.

[0045] The image data samples may, instead of a full image frame, only indicate the detected photons. For example, the image data samples may be configured as sparse frames or as lists of photon detection events, and may provide an interface (x, y, t), where x and y may indicate locations (e g., pixel column and pixel row of the single photon imaging sensor) at which a photon is detected and t may indicate a time of the detected photon. Thus, the image data may not always be outputted as complete image frames that indicate a value (e.g., photon count) for all photosensitive elements (e.g., pixels) of the single photon imaging sensor, but rather the single photon imaging sensor may output, as sparse frames, a metric (e.g., sum) of photons received during a spatio-temporal cubic (x, y, t). If no photons (or fewer photons than a predefined threshold) are detected during this spatio-temporal cubic, no output may be generated for the corresponding time t. The sequence of image data samples may be configured as a stream of photons (or of photon detection events, with time stamps of detection) generated for each photosensitive element (e.g., pixel) of the single photon imaging sensor.

[0046] A sample rate at which the single photon imaging sensor may capture the sequence of image data samples may correspond to 103, 104, 105, 106, 107, or even more image data samples per second (without limiting the disclosure to these values or to this range). For the higher sample rates, it may be advantageous to include the single photon imaging sensor in the circuitry in order to reduce bandwidth requirements of the communication unit, to reduce an energy consumption caused by a transmission of the sequence of image data samples, and / or to allow a higher sample rate of the sequence of image data samples.

[0047] The circuitry may determine one or more flux frequency components of a light source or of several light sources, as described below in more detail. The flux may correspond to, e.g., a luminous flux, a radiant flux and / or a photon flux measured by the single photon image sensor. The luminous flux may correspond to a perceived power of light. The radiant flux may correspond to a total power of electromagnetic radiation. The photon flux may correspond to a number of photons per time (e.g., a photon rate).

[0048] The circuitry may determine the flux frequency component(s) for each pixel or for each group of pixels in the sequence of image data samples. For example, the circuitry may apply a spatial filtering (e.g., summation, averaging, determining a median, applying an inclusive OR, etc.) to each group of pixels, and may determine the flux frequency component(s) for each group of pixels based on a result of the spatial filtering.

[0049] Based on the determined flux frequency component(s) of the (groups of) pixels, the circuitry may determine one or more segments in an image, as described below in more detail. The segment(s) may correspond to the pixel(s) or group(s) of pixels in which the flux frequency component(s) of the light source are detected.

[0050] The image may be based on the sequence of image data samples, for example, may be generated by merging the image data samples of the sequence of image data samples (e.g., summing, averaging, determining a median, applying an inclusive OR, etc. corresponding pixels of the sequence of image data samples), e.g., as may be known in the field of quanta burst imaging (QBI).

[0051] Alternatively, the image may be based on another imaging sensor (e.g., a red-green-blue (RGB) color image sensor or a monochrome image sensor), which may be based on CMOS and / or on charge-coupled device (CCD) technology. The circuitry may obtain a coordinate mapping between pixel coordinates in the image data samples from the single photon imaging sensor and pixel coordinates in the image from the other imaging sensor, and may determine, as the segment(s) in the image, one or more portions in the image that, according to the coordinate mapping, correspond to pixel(s) or group(s) of pixels in which the flux frequency component(s) of the light source are detected.

[0052] The coordinate mapping may be based on a predefined mounting position of the other imaging sensor relative to the single photon imaging sensor (e.g., if the single photon imaging sensor and the other imaging sensor are mounted on a same casing), or the coordinate mapping may be based on image registration (e.g., determining similar / corresponding features such as edges, comers, lines, gradients, wavelets etc. in the sequence of image data samples and in the image).

[0053] The circuitry may determine further segments in which flux frequency components of further light sources are detected. For example, the circuitry may determine which flux frequency components occur with a fixed (within a predefined tolerance) ratio in the (groups of) pixels, e.g., based on correlating frequency spectra of the (groups of) pixels with each other. For example, the circuitry may correlate a distribution of a contribution of a flux frequency component over the (groups of) pixels of the sequence of image data samples with distributions of contributions of other flux frequency components over the (groups of) pixels of the sequence of image data samples).

[0054] It is noted that a flux frequency component may be present in flux frequency spectra of several light sources, and that the circuitry may account for contributions from several light sources to a flux frequency component such that a ratio of the flux frequency component in a (group of) pixels may correspond to a contribution from several light sources to the flux frequency component.

[0055] The circuitry may determine that flux frequency components that occur with a fixed (within the tolerance) ratio in the (groups of) pixels belong to a same flux frequency spectrum and correspond to a same light source. Thus, a number of uncorrelated flux frequency spectra detected in the sequence of image data samples may correspond to a number of light sources. In this regard, shadow may be treated as a light source, e.g., with a zero light flux or with a reduced light flux.

[0056] For each determined light source, or for each light source whose contribution to a flux exceeds a predefined threshold, the circuitry may determine one or more segments in which the light source is detected or in which a contribution of the light source exceeds a predefined threshold. Segments associated with different light sources may overlap.

[0057] The segment(s) may include a plurality of (groups of) pixels in which the flux spectrum of the associated light source is detected, or each pixel or group of pixels in which the flux spectrum of the associated light source is detected may be treated as one segment. The circuitry may also determine (groups of) pixels in which an equal contribution of a flux spectrum of a light source is detected to belong to a same segment.

[0058] Thus, the circuitry may generate a map of segments that indicates which light source is detected in which portion (e.g., segment) of the image. In some embodiments, the map of segments also indicates how strong a contribution of the light source to the portion is. For example, the map of segments may indicate a relative contribution of a detected light source to the portion (e g., segment), e.g., a ratio or percentage of the contribution from the light source to a light flux of the portion (e.g., segment). For example (e.g., in a case where a direct current (DC) component of a light source is zero or negligibly small), the map of segments may indicate an absolute contribution of a detected light source, e.g., a value that indicates a measured light flux that is received from the portion (e.g., segment) and that corresponds to the light source.

[0059] Thus, the map of segment(s) in the image may indicate which portion of a scene represented by the image (e.g., shown by the image) is illuminated by which light source.

[0060] The light source may include any light source that exhibits light flux variations (e g., flickering) whose Nyquist frequency is below a sample rate at which the sequence of image data samples are captured. For example, the light source may include a light-emitting diode (LED) lamp, an incandescent light bulb, a fluorescent lamp, a gas discharge lamp, a compact fluorescent lamp (CFL), an arc lamp, a laser, a screen, a video projector, or the like. As mentioned, shadow that corresponds to light flux variations may be treated as a light source with zero light flux (or with a reduced flux). Thus, the processing with respect to light sources, as described herein, may as well be applied to shadows.

[0061] The circuitry may perform processing and / or generate data (e.g., an image, a parameter, a control command etc.) based on the segment(s) in the image and the determined contribution(s) of the light source(s), as described herein.

[0062] In some embodiments, the determining of the segment includes: determining spatial portions of the sequence of image data samples in which the flux frequency component of the light source is detected; and determining, as the segment in the image, a portion in the image that corresponds to the determined spatial portions of the sequence of image data samples.

[0063] As mentioned above, the spatial portions may correspond to pixels, pixel coordinates or to groups of pixels or of pixel coordinates in the sequence of image data samples. The circuitry may apply a spatial filtering to the groups of pixels, and the determining of the spatial portions and of the segment may be based on a result of the spatial filtering. The spatial portions may correspond to regions in a pixel coordinate system of the sequence of image data samples.

[0064] The circuitry may determine, as the spatial portions in which the flux frequency component of the light source is detected, spatial portions in which a similar flux frequency (e.g., a flux frequency that differs from the detected flux frequency component of the light source by less than a predefined tolerance or threshold) is detected.

[0065] The circuitry may determine, as spatial portions of the sequence of image data samples in which the flux frequency component of the light source is detected, one of more (groups of) pixels in which a contribution of the flux frequency component exceeds a threshold (e.g., satisfies a predefined minimum relative contribution as compared to other flux frequency components, satisfies at least a predefined percentile, satisfies a predefined minimum absolute light flux, or the like).

[0066] For example, the circuitry may determine (groups of) pixels in the sequence of image data samples in which a ratio between detected flux frequency components of a flux frequency spectrum associated with the light source is a fixed (e.g., within a predefined tolerance), and may determine such (groups of) pixels to correspond to the spatial portions of the sequence of image data samples in which the flux frequency component of the light source is detected, based on the fixed ratio of flux frequency components.

[0067] The circuitry may determine a portion (e.g., a region in a pixel coordinate system) in the image that corresponds to the determined spatial portions of the sequence of image data samples. The determining of the portion in the image may be based on a predefined coordinate mapping between a pixel coordinate system of the sequence of image data samples and a pixel coordinate system of the image, or may be based on image registration of the image in the sequence of image data samples.

[0068] Accordingly, the segment in the image may correspond to a portion of the image (e.g., a pixel range) in which the flux frequency component (or the flux frequency spectrum) of the light source is detected. Thus, for a scene represented by the image (e.g., shown in the image), the segment may indicate a portion of the scene that is illuminated by the light source.

[0069] As mentioned, the circuitry may be configured to determine a plurality of segments in the image, which may correspond to different light sources, wherein segments of different light sources may overlap. Thus, based on different determined flux frequency components in the sequence of image data samples, the circuitry may determine which portion in the scene represented by the image is illuminated by which light source.

[0070] In some embodiments, the determining of the segment is further based on a feature detected in the image. For example, the image may be configured as an RGB (or monochrome) image, and RGB (or monochrome) information from the image may be combined (e.g., augmented) with the flux frequency component from the image data samples (e.g., before segmentation). The spatial portions of the sequence of image data samples and / or the image segment may then be determined based on the flux frequency component and on the RGB (or monochrome) image. As an example, if an algorithm, e g., based on an artificial neural network, identifies an area in the picture which is sky, it may assist the segmentation process of light sources.

[0071] The RGB image or monochrome image may be captured by the single photon imaging sensor or by a different sensor than the single photon imaging sensor. In some embodiments, the determining of the flux frequency component of the light source includes: detecting a flux variation in the sequence of image data samples; and determining a frequency component of the detected flux variation.

[0072] The circuitry may detect a flux variation for each pixel or group of pixels, e.g., for pixels at a same pixel coordinate in the image data samples of the sequence of image data samples, for groups of neighboring pixels (pixel coordinates) in the image data samples (e.g., in a case where the detecting of a flux variation is based on a sum of neighboring pixels for increasing a signal- to-noise-ratio (SNR)), or for corresponding (groups of) pixels of the image data samples, wherein the circuitry may determine corresponding (groups of) pixels in the image data samples, e.g., based on image registration.

[0073] The detecting of a flux variation in the sequence of image data samples may include determining a light flux in (groups of) pixels of the sequence of image data samples at each point in time at which an image data sample of the sequence of image data samples has been captured, or determining a light flux in (groups of) pixels of a predefined subset of the sequence of image data samples (e.g., every n-th image data sample; and omitting interjacent image data samples), or may include merging predetermined numbers of subsequent image data samples (e.g., based on summation, averaging, determining a median, applying a inclusive logical OR, or the like) and determining a light flux in (groups of) pixels of each of the merged image data samples.

[0074] Determining the light flux based on a subsampled or merged sequence of image data samples may allow a faster processing and / or a reduced energy consumption because fewer data samples may have to be processed. Determining the light flux based on a merged sequence of image data samples may increase a signal-to-noise ratio (SNR) of the light flux variation.

[0075] However, if the circuitry determines the light flux based on a subset of the sequence of image data samples or based on merged image data samples, the circuitry may omit or merge at most so many image data samples that a sample rate of the sub sampled or merged sequence of image data samples exceeds a Nyquist frequency of a highest frequency component that should be detected in the sequence of image data samples.

[0076] In the determined light flux of each (group of) pixels in the sequence of image data samples, the circuitry may determine a flux frequency spectrum, e.g., by applying a Fourier transform (e.g., discrete Fourier transform (DFT), fast Fourier transform (FFT)) and / or a selected-frequencies estimation (e.g., Goertzel algorithm). Accordingly, in some embodiments, the determining of the flux frequency component of the light source is based on digital Fourier transform. The circuitry may filter the determined flux frequency spectrum (e.g., remove noise, remove spectrum components that do not satisfy a predefined threshold, or the like), may determine one or more flux frequency components of the determined flux frequency spectrum that satisfy a predefined threshold and / or a threshold determined based on a noise level of the determined flux frequency spectrum, may determine a flux frequency spectrum of a light source by correlating the flux frequency spectra of the (groups of) pixels with each other, or the like.

[0077] Thus, the circuitry may associate each detected flux frequency component and / or each detected flux frequency spectrum of correlated flux frequency components with another light source. Accordingly, a number of light sources that illuminate a scene represented by the image may be determined based on the flux variation in the sequence of image data samples.

[0078] The detecting of the flux variation and the determining of the frequency component(s) of the detected flux variation may be performed in a same step or in subsequent steps.

[0079] In some embodiments, the flux variation corresponds to a difference in a number of detected photons between image data samples of the sequence of image data samples.

[0080] One or more light sources that illuminate(s) a scene represented by the sequence of image data samples and by the image may exhibit a flickering (e.g., due to voltage and / or current variations in a power supply of the light source). The flickering may correspond to different energies emitted from the light source(s) in time intervals that correspond to different image data samples. Thus, the single photon imaging sensor may detect a different number of photons from the light source(s) in time intervals that correspond to different image data samples. For example, in a case of a periodic flickering (according to a flux frequency component) of a light source, the single photon imaging sensor may detect different numbers of photons from the light source at different phases of a flickering period of the light source.

[0081] In some embodiments, the image data samples of the sequence of image data samples associate numbers of photons detected by the single photon imaging sensor with spatial portions of the image data samples.

[0082] The spatial portions of the image data samples may correspond to pixels and / or groups of pixels. The single photon detection elements of the single photon imaging sensor may be associated with pixels of the image data samples, and each pixel in an image data sample may be assigned a value that corresponds to a number of photons detected by its associated single photon detection element(s) during the time interval that corresponds to the image data sample. When performing a spatial filtering of the image data samples, the circuitry may associate each group of pixels with a value that corresponds to a number of photons detected by the grouped pixels according to a filtering result (e.g., a sum, average, median, logical disjunction).

[0083] In some embodiments, the determining of the flux frequency component is based on a predefined wavelength range of light emitted by the light source.

[0084] For example, the predefined wavelength range may be characteristic for the light source. For example, one or more other light sources in the scene may emit no or significantly less light in the predefined wavelength range than the light source, such that the light source may be easily distinguished from the one or more other light sources based on light detected in the predefined wavelength range. For example, the predefined wavelength range may correspond to an emission line of the light source or to a predefined interval around the emission line. The predefined wavelength range may be contiguous or may be non-contiguous, e g., may include two or more disjoint wavelength intervals.

[0085] The single photon imaging sensor may be configured to detect and / or count photons in the predefined wavelength range. For example, the single photon imaging sensor may be configured to receive light of the predefined wavelength range at one or more photosensitive elements associated with the predefined wavelength range. The sequence of image data samples may indicate, for each pixel, for each detected photon and / or for each indicated number of detected photons, an associated wavelength range, and the flux frequency component may be determined based on a predefined wavelength range of the associated wavelength ranges of the pixels, photons or numbers of photons.

[0086] For example, the single photon imaging sensor may include a color filter (e.g., longpass filter, shortpass filter, bandpass filter; RGB filters, infrared filter, UV filter etc.), and the predefined wavelength range may correspond to a wavelength range that is transmitted by the color filter. Photosensitive elements of the single photon imaging sensor may be coupled with color filters, as known from an RGB imager. For example, the single photon imaging sensor may be configured as an RGB color image sensor in which color (RGB) filters are arranged in a Bayer pattern, and each photosensitive element of the single photon imaging sensor may detect photons transmitted by a color filter associated with (e g., arranged in front of) the photosensitive element. Thus, the single photon imaging sensor may perform wavelength filtering as in an RGB sensor.

[0087] For example, the single photon imaging sensor may receive light from a prism, from a diffracting grating, from a Fabry -Perot interferometer and / or from any other optical element that allows to distinguish between wavelengths, such that the single photon imaging sensor may receive light of one specific wavelength range, and / or may receive light of a plurality of wavelength ranges at different pixel coordinates and / or at different photosensitive elements.

[0088] The circuitry may determine one or more flux frequency components of the light source for one or more predefined wavelength ranges. The circuitry may further determine a segment in the image for the determined flux frequency component s) of the predefined wavelength or for each of the one or more predefined wavelength ranges.

[0089] For example, the circuitry may distinguish between different flux frequency components of a same color (e g., of the predefined wavelength range), and may, for example, easier distinguish between light sources of a same color based on the determined flux frequency component of the predefined wavelength range.

[0090] In such cases, an additional dimension may be opened up: Flux frequency components may be estimated and segmented for different colors (on top of a spatial domain). For example, two different red light sources (which may exhibit different flicker frequencies) may be distinguished more easily from each other. In general, the additional dimension of a wavelength range may assist in an estimation and segmentation process of flux frequency components (as it provides additional information).

[0091] In some cases, such color filtering may also pose challenges, as, in such a case, each pixel (e.g., photosensitive element of the single photon imaging sensor) may see only a specific color corresponding to the predefined wavelength range. However, these challenges may be solved, e.g., by using de-mosaic algorithms running in an image signal processor (ISP), which may be used also in this case.

[0092] A flux frequency component segmentation based on the predefined wavelength range (e.g., a segmentation by wavelength) may be performed similar to the processing described herein, for example by the circuitry or by an external ISP (when the single photon imaging sensor only outputs the flux frequency components, and the segment(s) in the image are determined by the ISP).

[0093] In some embodiments, the circuitry is further configured to output an indication of the determined flux frequency component of the light source. In some embodiments, the circuitry is further configured to output an indication of the determined segment in the image. For example, the circuitry may output an indication of one or more determined flux frequency component(s) or one or more light sources and / or a determined image segment (or a map of image segments) via its communication unit, e.g., to an external device. Fig. 1 illustrates an embodiment of a circuitry 1. The circuitry includes an array 2 of pixels, wherein each pixel includes a single photon detection element based on a SPAD. The circuitry further includes a read-out circuitry 3 that controls a read out of photon detection signals from the array 2 of pixels and generates a sequence of image data frames (an example of image data samples) based on the read out photon detection signals, an image processing circuitry 4 that processes the sequence of image data frames, a flux analysis circuitry 5 that analyzes characteristics of a light flux in the sequence of image data frames, a memory 6 that stores data generated by the read-out circuitry 3, by the image processing circuitry 4 or by the flux analysis circuitry 5, and a communication unit 7 that outputs data to an external device that is external to the circuitry 1.

[0094] The circuitry 1 is configured as a quanta imaging sensor (QIS; an example of a single photon imaging sensor) that captures the sequence of image data frames. The sequence of image data frames indicates detected photons per horizontal and vertical pixel coordinate ( , y) in the array 2 and time t (wherein the time t corresponds to an image data frame). The image processing circuitry 4 and the flux analysis circuitry 5 are examples of a processing unit.

[0095] The circuitry 1 is configured to generate a map 8 of image segments, and to output the map 8 of image segments and a sequence 9 of output image frames that are based on the sequence of image data frames. The circuitry 1 is configured to output the map 8 of image segments and the sequence 9 of output image frames via the communication unit 7 and to transmit them to an external device.

[0096] The map 8 of image segments indicates which flux frequency component is detected in which portion (segment) of the sequence of image data frames. The map 8 of image data frames indicates for each segment a frequency, an amplitude and a phase of a flux frequency component detected in the sequence. For example, as shown in Fig. 1, a first segment (vertically hatched) may be associated with a frequency f , with an amplitude and with a phase , a second segment (diagonally hatched from left-top to right-bottom) may be associated with a frequency f2, with an amplitude A2and with a phase <f)2, etc. It is noted that a segment may be associated with more than one flux frequency component (e g., with a flux frequency spectrum of a light source, and / or with flux frequency components from different light sources).

[0097] The sequence 9 of output image frames is based on the sequence of image data frames captured by the circuitry 1. In the embodiment of Fig. 1, each output image frame of the sequence 9 of output image frames is based on a sum of a predefined plurality of image data frames of the sequence of image data frames, and the pixels in the output image frame indicate a number of photons detected by associated single photon detection elements of the array 2 of pixels during time intervals that correspond to the respective image data frames. It is noted that the output image frames may be a summation not only in time domain, e.g., multiple image data frames, but also spatial filtering, such that the output image frames may be a downsampled version of the image data frames (where each output pixel may be a summation of NxN pixels of the image data frames). It is also noted that the spatial / time domain summation may be different over different potions of the image data frames (e.g. at some areas more spatial binning may be applied, while in other areas less spatial binning may be applied), similar to quanta burst imaging. In some embodiments, each output image frame of the sequence 9 of output image frames corresponds to a single image data frame of the sequence of image data frames. It is noted that the disclosure is not limited to outputting both the map 8 of image segments and the sequence 9 of output image frames, but that only one of the map 8 of image segments and the sequence 9 of output image frames is outputted in some embodiments.

[0098] The circuitry 1 is configured to perform any one of the methods of Fig. 2, 4, 5, 6, 7, 8, 9, 12, or 13.

[0099] Fig. 2 illustrates an embodiment of a method 10 for determining a segment in an image. The method 10 is an example of a method performed a circuitry according to the disclosure, e.g., by the circuitry 1 of Fig. 1 or by the general-purpose computer 250 of Fig. 14.

[0100] At 11, the circuitry obtains a sequence of image data frames (an example of image data samples) captured by a single photon imaging sensor (e.g., by the circuitry 1 of Fig. 1, or by an external QIS). The image data frames of the sequence of image data frames associate numbers of photons detected by the single photon imaging sensor with spatial portions of the image data frames.

[0101] At 12, the circuitry determines, based on the sequence of image data frames, a flux frequency component of a light source. The determining of the flux frequency component of the light source at 12 includes detecting, at 13, a flux variation in the sequence of image data frames, wherein the flux variation corresponds to a difference in a number of detected photons between image data frames of the sequence of image data frames, and determining, at 14, a frequency component of the detected flux variation.

[0102] In some embodiments, the image data frames indicate wavelength ranges associated with the detected photons, and the determining of the flux frequency component at 12 is based on a predefined wavelength range, of the associated wavelength ranges, of light emitted by the light source. However, the disclosure is not limited to determining the flux frequency component based on a predefined wavelength range, and the flux frequency component is determined at 12, in some embodiments, based on an accumulated light flux of all sensed wavelength ranges.

[0103] At 15, the circuitry determines a segment in an image based on the determined flux frequency component of the light source. The determining of the segment in the image includes determining, at 16, spatial portions of the sequence of image data frames in which the flux frequency component of the light source is detected, and determining, at 17, as the segment in the image, a portion in the image that corresponds to the determined spatial portions of the sequence of image data frames.

[0104] In some embodiments, the determining of the segment at 15 is further based on a feature detected in the image. For example, RGB (or monochrome) information from the image may be combined (e.g., augmented) with the flux frequency component from the image data samples (e.g., with ‘raw’ flux frequency components according to image data samples outputted by SPADs, e.g. before segmentation), and the spatial portions of the sequence of image data samples and / or the image segment may be determined based on the flux frequency component and on the RGB (or monochrome) image. As an example, if an algorithm, e.g., based on an artificial neural network, identifies an area in the picture which is sky, it may assist the segmentation process of light sources.

[0105] As indicated above, the method 10 is not limited to determining a single flux frequency component. The method 10 may determine, according to the processing at 12, a plurality of flux frequency components (e.g., a flux frequency spectrum) of the light source. The method 10 may also determine, according to the processing at 12, flux frequency components (and / or flux frequency spectra) of different light sources. The method 10 may determine, according to the processing at 15, a corresponding segment in the image for each determined flux frequency component and / or flux frequency spectrum.

[0106] Further, although the detecting of the flux variation at 13 and the determining of the frequency component at 14 are shown as separate blocks in Fig. 2, the detecting of the flux variation and the determining of the frequency component may be performed together in a same processing. For example, the detecting of the flux variation at 13 may be performed as part of the determining of the frequency component at 14 (e g., based on a strength of frequency components in a calculated frequency spectrum of the light flux). For example, a frequency spectrum may be calculated at 14 without prior analysis of a flux variation, and the detecting of a flux variation may correspond to determining one or more flux frequency components in the calculated spectrum and / or to determining that the flux frequency component(s) satisfy a predefined threshold.

[0107] Fig. 3 illustrates a transform from a time domain to a frequency domain according to an embodiment. The circuitry according to the disclosure (e.g., the circuitry 1 of Fig. 1 or the general -purpose computer 250 of Fig. 14) applies the transform to the light flux variations detected in the sequence of image data frames (an example of image data samples) in order to obtain flux power spectral components (an example of flux frequency components).

[0108] As mentioned, a luminous flux is a measure of a perceived power of light. A radiant flux is a measure of a total power of electromagnetic radiation. A photon flux is a measure of a number of photons per time (e.g., a photon rate). The luminous flux, the radiant flux and the photon flux are examples of the light flux.

[0109] As mentioned, the light flux may not be constant, depending on the light source. Thus, an instantaneous light flux < >(x,y,t) that corresponds to a flux received by a pixel (e.g., in the array 2 of Fig. 1) at a pixel coordinate (x, y) at time t may vary with time.

[0110] Graph 20 illustrates an exemplary light flux 21 <p(x,y,t) emitted by a light source in a time domain, wherein a horizontal direction in the graph 20 corresponds to time, and a vertical direction corresponds to an amount of light (in arbitrary units). The single photon imaging sensor detects more photons 22 when the light flux 21 is high, and fewer photons 22 when the light flux 21 is low.

[0111] A transform (e.g., Fourier transform, DFT, FFT, Goertzel algorithm) from the time domain to a frequency domain may allow to derive characteristics of the light flux <p(x,y, t), e.g., spectral components (flux frequency components) of the light flux at given pixel (x, y) over an observed time window t, from the photons 22 detected by the image sensor.

[0112] Graph 23 illustrates an exemplary flux frequency spectrum obtained by the transform from the time domain to the frequency domain. A horizontal direction in the graph 23 corresponds to a frequency (in arbitrary units), a vertical direction corresponds to a power (in arbitrary units), and each sample with a non-zero power corresponds to a detected flux frequency component.

[0113] Fig. 4 illustrates a flux spectrum block 30 according to an embodiment. The flux spectrum block 30 is an example of a processing performed by the flux analysis circuitry 5 of Fig. 1 and of the processing performed at 12 and 15 of the method 10 of Fig. 2.

[0114] The flux spectrum block 30 includes a spatio-temporal filtering & decimation 31, a spectrum components detection & filtering 32 and a spatial spectrum components segmentation 33. The flux spectrum block 30 receives algorithm parameters such as a list of frequencies (or range or frequencies) 34, a time scale for measurement 35, a spatial granularity 36, and additional algorithm parameters 37. The additional algorithm parameters 37 include, e.g., thresholds, a number of output components per group, etc.

[0115] The spatio-temporal filtering & decimation 31 includes a spatial filtering 41 (e.g., summation) of pixel groups and a time domain filtering and (optional) decimation 42.

[0116] The spatial filtering 41 (e.g., summation) of pixel groups receives an indication of a photon count per horizontal and vertical pixel coordinate and time (x, y, t) in the sequence of image data frames, adds up photon counts per pixel group, and outputs the photon count per horizontal and vertical coordinate of the pixel groups and time (%', y', t). Here, x' and y' refer to spatial indices in a horizontal and vertical direction, respectively, of the pixel groups after the spatial filtering 41. For example, in a case where the sequence of image data frames (an example of image data samples) have been captured by a 1 Megapixel (MP) sensor, the pixels of the sequence of image data frames may translate, after the spatial filtering 41, to 256x256 groups of pixels (without limiting the disclosure to these values).

[0117] The time domain filtering and (optional) decimation 42 receives the photon counts per pixel group and time (x',y'> t) and performs a time domain filtering (e.g., summation) of the pixel groups. For example, the time domain filtering reduces a number of image data frames by adding up corresponding pixel groups (i.e., pixel groups at corresponding pixel coordinates x' and y') of predefined numbers of subsequent image data frames. In some embodiments, a decimation is also performed, in which a predefined number of image data frames are dropped, whereas in some embodiments, the decimation is not performed. The time domain filtering and (optional) decimation 42 outputs pixel counts per horizontal and vertical coordinate of the pixel groups and per filtered time (x',y'> t'

[0118] The spectrum components detection & filtering 32 includes a frequency components estimation 43, a noise statistics determination 44 and a filtering 45. The processing at 43, 44 and 45 is performed per block of (x',y') group of filtered pixels.

[0119] The frequency components estimation 43 receives as input a range or selected list of frequencies and a time window, performs an algorithm such as DFT / FFT (for a range of frequencies) or a selected-frequencies estimation (e.g., Goertzel algorithm), and outputs, per frequency, an amplitude / energy and a phase of the frequency component. The noise statistics determination 44 estimates, based on a estimation result of the frequency components estimation 43 and on a further result of the time domain filtering and (optional) decimation 42, a noise level where no frequency component exists. The noise level determined at 44 does not necessarily correspond to noise, but is rather a statistical metric that is later used to calculate a threshold.

[0120] The filtering 45 filters frequency components determined at 43 according to the noise level determined at 44. For example, the filtering 45 selects only frequency components which exceed a threshold that is calculated according to the noise statistics.

[0121] The spatial spectrum components segmentation 33 includes further filtering and segmentation 46 that is performed based on the frequency components filtered at 45.

[0122] It is noted that other types of algorithms may be applied, e.g., at 43, and may include a combination of coherent and non-coherent integrations. Non coherent integrations may be based on an abs() or abs()2operation. For example, DFT operations of separate pixels may be performed, but the summation of them may be done non-coherently. In Fig. 4, however, coherent integration is shown.

[0123] Thus, a single photon imager may perform a flux spectral components estimation and segmentation according to the flux spectrum block 30.

[0124] Some embodiments are concerned with improving an image quality (e.g., for prettifying an image or for improving a perception) based on determining color balance parameter(s) (e.g., for an automated white balance (AWB)).

[0125] The single photon imaging sensor may be used to estimate a flux frequency spectrum per pixel or group of pixels in order to assist in estimating one or more light sources (illuminant(s)) that illuminate(s) a field-of-view (FoV) of the single photon imaging sensor (e g , a scene in the FoV).

[0126] There are two ways of utilizing the segmentation of illuminants (light sources) for a color balance.

[0127] As a first way, the spatial segmentation of illuminants (i.e., light sources) may be utilized for improving an AWB estimation and correction algorithm (e.g., a single AWB correction for the whole image) by determining a color balance parameter based on the determined segment(s).

[0128] This may be based on focusing on proper single illumination areas (represented by the segment(s) with relevant characteristics). By analyzing spectral components of the light flux received by the single photon imaging sensor, it may be possible to distinguish between light sources that illuminate the FoV, and to segment the FoV per light source accordingly.

[0129] This information may then be augmented into an image processing pipeline of an RGB sensor, and specifically, into a color consistency correction (e.g., auto-white-balance (AWB)). The RGB sensor may be included in the circuitry (e.g., the circuitry may include a further imaging sensor in addition to the single photon imaging sensor, or the RGB sensor may be included in the single photon imaging sensor), or the RGB sensor may be provided as a separate device external to the circuitry.

[0130] As a second way, different AWB corrections (with different color balance parameters) may be applied to different areas in the FoV. To improve a quality of an image, applying good corrections may be challenging. On the other hand, such capability may be valuable for sensing applications.

[0131] In some instances, a perceived light flux from a scene is based at least one of: on an illumination (e g., light from a light source that illuminates the scene), on a reflectance of objects in the scene, on scattered light scattered at objects in the scene, on cone sensitivities of a human eye, or on actual absorptions of the cones. To an image obtained by image capture, a white balance may be applied in order to compensate for one light source (e.g., a superposition of all light sources that affect a reference portion of the image). After the white balance, a color correction may be performed in order to account for the cone sensitivities. Finally, a gamma correction may be performed for accounting for cone absorptions.

[0132] AWB algorithms for conventional CIS RGB sensors may aim to compensate for a single illumination source. The AWB algorithms may be based on image statistics, e g., collected in a linear domain, and may aim to separate a spectral influence of the illumination source from scene objects reflectivity (transparency).

[0133] An RGB CIS sensor may collect corresponding (e.g., exactly the same) color samples from various combinations of an illumination spectrum and a spectral reflectance of an object.. Thus, in some instances, a main issue may pertain to the question how to correctly estimate a white balance in a multi-illumination environment.

[0134] As a result, in some instances, a conventional AWB may calculate some average illumination compensation, while neutral colors at any specific illumination may be not corrected properly. From an image visualization perspective, in an environment with multiple illumination sources, a preference may be to compensate for a single dominant illumination, e.g., in a case of a strong sunlight illumination incident through a window and an additional table lamp, compensation may be performed based only on the sunlight component and not on the table lamp component.

[0135] In some embodiments, the circuitry is further configured to determine a color balance parameter based on the determined segment in the image.

[0136] The image may be an RGB image, and colors in the image may be out of balance, e.g., such that neutral colors (such as white or gray) are not reproduced as neutral but exhibit a hue. Thus, a color balance (e g., automated white balance (AWB)) may be desired. The circuitry may determine one or more parameters (color balance parameter(s)) for the color balance to correct the colors in the image.

[0137] The hue, by which the colors may be out of balance, may be caused by a light source. For example, depending on a color temperature, a fluorophore emission line, a color rendering index (CRI), or the like of the light source, neutral colors may be skewed towards a hue.

[0138] The circuitry may determine, based on the segment(s) in the image in which the flux frequency component or the flux frequency spectrum of the light source is detected, a difference in a color balancing between the segment that corresponds to a portion of the scene that is illuminated by the light source and a portion of the image that corresponds to a portion of the scene that is not illuminated by the light source.

[0139] Based on this difference, the circuitry may determine one or more color balance parameters (e.g., scaling parameter of a base color) for transforming the colors in the image to a neutral regime in which neutral colors appear neutral. In some embodiments, instead of transforming the colors to a neutral regime, the color balance parameter may be determined such that an intentional cool or warm hue is added to the image.

[0140] Due to the segment(s) in the image, it may be possible to determine the color balance parameter(s) only based on the segment in which a contribution of the light source is present, and the whole image may be color balanced accordingly, irrespective of any further (e.g., smaller) light sources.

[0141] If several image sources are detected in the image, the circuitry may determine the color balance parameter based on a largest segment, based on a light source that provides a largest contribution to an illumination of the scene represented by the image, based on a segment (e.g., a light source that provides a largest contribution to the segment, or several detected light sources that contribute to the segment) in a focus region of the image (e.g., in a center of the image, in a reference area of an auto focus function, in a region of the image with a highest gradient, or the like). The color balance parameter may also be determined in combination with additional information from the RGB image.

[0142] The circuitry may apply a color balance to the image according to the determined color balance parameter(s) and, e.g., output the color balanced image via the communication unit. Alternatively, the circuitry may output the determined color balance parameter(s) via the communication unit to an external device that may apply a color balance to the image according to the determined color balance parameters).

[0143] The color balance parameter(s) may indicate (e.g., include, correspond to, etc.) the determined frequency component(s) and / or the determined image segment(s) such that the color balance may be applied based on the determined frequency component(s) and / or image segment(s). Accordingly, in some embodiments, the color balance parameter indicates the determined flux frequency component of the light source. For example, the circuitry may output, as the color balance parameter(s), (one or more of) the determined frequency component(s) and / or (one or more of) the determined image segment(s), and the color balance (e.g., AWB) may be performed outside the circuitry (e.g., by an external device) based on the outputted color balance parameter(s).

[0144] In some embodiments, the circuitry is further configured to determine a further color balance parameter based on a further segment in the image; apply a color balance based on the determined color balance parameter to the determined segment in the image; and apply a color balance based on the further color balance parameter to the further segment in the image.

[0145] For example, the further segment may correspond to a portion of the scene represented by the image that may be illuminated by another light source than the segment. The other light source may have different specifications than the light source and, thus, may cause a different hue (color skew) in the further segment than the hue caused by the light source in the segment.

[0146] The circuitry may determine the further color balance parameter(s) for the further segment similarly to the determining of the color balance parameter(s) for the segment. The circuitry may apply to each of the segments and the further segment a color balance according to the corresponding color balance parameter(s).

[0147] Thus, based on the determined segment(s) in which flux frequency components of different light sources are detected, the circuitry may perform a color balance for each detected light source separately and / or for each segment in the image separately according to contributions of the light sources to the segment.

[0148] The circuitry may apply a weighted color balance to segments that contain light flux contributions from several light sources according to contributions of the several light sources.

[0149] At a border of a segment, the circuitry may perform a smoothing, a fading-out and / or an interpolation between color balances with different color balance parameters in order to avoid or reduce an abrupt change (e g., a large gradient) of a hue (color skew) in the image.

[0150] Similar to the color balance parameter, the further color balance parameter may indicate a scaling parameter of a base color, and / or may indicate the further image segment and / or the determined frequency component(s) of the further image segment.

[0151] Fig. 5 illustrates a difference between a color balance based on conventional statistics and a color balance based on QIS-based segmentation according to an embodiment.

[0152] A graph 51 shows an AWB based on conventional statistics. A vertical direction indicates a normalized red value RN= - (wherein R, G and B represent red, green and blue portions, respectively) of a color in an AWB reference portion), and a horizontal direction indicates a normalized blue value BN— - of the color in the AWB reference portion. Stars indicate exemplary colors of AWB reference portions of an RGB image.

[0153] A goal of an AWB is transforming colors in the RGB image such that red, green and blue portions of neutral colors (e.g., white or gray) are equal, e.g., that a white point is converted to N=BN=3

[0154] In conventional AWB, a white balance (WB) correction vector 52 for converting the white point to RN= BN= is determined based on conventional statistics. It is noted that this is an example of an AWB calculation method, and that there may be alternative methods that may be used. In some embodiments, similar transformation is calculated based on other methods.

[0155] A graph 53 shows an AWB (an example of a color correction) based on QIS-based segmentation. A first WB correction vector 54 is determined based on an AWB reference portion that corresponds to a segment that corresponds to a first light source (indicated by white stars). A second WB correction vector 55 is determined based on an WB reference portion that corresponds to a segment that corresponds to a second light source (indicated by black stars).

[0156] Accordingly, a circuitry (e.g., the circuitry 1 of Fig. 1 or the general-purpose computer 250 of Fig. 14) may determine a color balance parameter for each light source based on the determined segments, and, thus, may use a light source segmentation according to the disclosure, e.g., for improving an AWB performance.

[0157] Based on QIS sensor information, it may be possible to separate WB calculations for each illumination (e.g., light source) independently. Instead of an average WB correction that may average over all light sources in an image, which may be wrong and not represent any one of several light sources, it may be possible to find a proper correction for each light source based on the segments of flux frequency spectra.

[0158] Using this information, a proper White Balance correction may be utilized in any one of the following two ways: A weighted (i.e., weighed between different light sources per image segment) correction for multi-illuminants may be performed per image region (which may be preferred for object detection, where proper color of each object may be critical), or a (single) weighted (i.e., weighted between different light sources in the image) correction of illumination sources may be applied to a full image (which may be preferred for proper visualization). For example, for ignoring a light source such that the WB correction is not based on the light source to be ignored, a weight of the light source to be ignored may be set to zero.

[0159] Fig. 6 illustrates an AWB pipeline 60 according to an embodiment. A scene 61 is illuminated by a plurality of light sources 61a, 61b and 61c. Each of the light sources 61a, 61b and 61c exhibits a flickering with a flux frequency spectrum different from the respective other light sources 61a, 61b and 61c.

[0160] A single photon imager 62 (an example of the single photon imaging sensor provided by the circuitry 1) captures a sequence of image data samples of the scene 61 based on QIS, and outputs the sequence of image data samples 63 (which include photon timestamps) to a flux spectral components estimation & segmentation 64. The flux spectral components estimation & segmentation 64 is an example of the processing at 12 and 15 of Fig. 2, and outputs a flux segmentation 65 (an example of the map 8 of image segments of Fig. 1).

[0161] In the flux segmentation 65, a first segment 65a corresponds to flux frequency component (e.g., flux frequency spectrum) of a first light source 61a in the scene 61 and indicates a portion of the scene 61 that is illuminated by the first light source 61. Likewise, a second segment 65b and a third segment 65c correspond to flux frequency components (e.g., flux frequency spectra) of a second light source 61b and a third light source 61c, respectively, in the scene 61, and indicate respective portions of the scene 61 that are illuminated by the second light source 61b and the third light source 61c, respectively. An RGB sensor 66 captures an RGB image of the scene 61 with an exposure that is so long that the flickering of the light sources 61a, 61b and 61c cannot be detected based on the RGB image. The RGB sensor 66 outputs the raw RGB image 67 to an image signal processing (ISP) 68.

[0162] The ISP 68 includes an automated white balance (AWB) 68a (an example of a color balance). The AWB 68a is performed based on the flux segmentation 65 from the flux spectral components estimation & segmentation 64. The ISP 68 then outputs an RGB image 69 to which the AWB 68a has been applied.

[0163] It is noted that, although Fig. 6 shows the flux spectral components estimation & segmentation 64 as a separate block than the single photon imager 62, flux spectral components estimation & segmentation 64 is in some embodiments included (e.g., performed by) the single photon imager 62.

[0164] It is further noted that, although Fig. 6 shows the single photon imager 62 and the RGB sensor 66 as separate blocks, the RGB sensor 66 is in some embodiments included in the single photon imager 62 (e.g., one sensor may provide both the single photon imager 62 and the RGB sensor 66).

[0165] Fig. 7 illustrates a first embodiment of an AWB 70. The AWB 70 is an example of the AWB 68a of Fig. 6. The AWB 70 includes an illuminant source estimation and segmentation 71 and an illuminant color estimation & correction 72.

[0166] The illuminant source estimation and segmentation 71 receives a flux segmentation (e.g., the flux segmentation 65 of Fig. 6), which is based on a sequence of image data samples from single photon imaging 73 (e g., the sequence of image data samples 63 (including photon counts) of the single photon imager 62 of Fig. 6), from flux spectral components estimation & segmentation 74 (e g., the flux spectral components estimation & segmentation 64 of Fig. 6). The illuminant source estimation and segmentation 71 further receives an RGB image (e.g., the raw RGB image 67 of Fig. 6) from RGB sensing 75 (e.g., by the RGB sensor 66 of Fig. 6). The illuminant source estimation and segmentation 71 determines a mapping of light sources to groups of pixels in the RGB image 67. It is noted that, in some embodiments, the illuminant source estimation and segmentation 71 does not receive the RGB image but determines the mapping of light sources to groups of pixels based on the flux segmentation (e.g., in a case where a transform from pixel coordinates of the flux segmentation to pixel coordinates of the RGB image is known).

[0167] The illuminant color estimation & correction 72 (e.g., AWB) is performed per group of pixels (e.g., the segments 65a, 65b and 65c of Fig. 6). After the illuminant color estimation & correction 72, the AWB 70 outputs a white balanced image 76, in which colors are corrected differently in different segments (e g., the segments 65a, 65b and 65c of Fig. 6) based light sources (e.g., the light sources 61a, 61b and 61c of Fig. 6) that illuminate the respective segments.

[0168] Accordingly, the AWB 70 determines a color balance parameter (e g., a parameter for the

[0169] AWB 70) based on a determined segment in the image, and applies a color balance based on the determined color balance parameter to the segment in the image. The AWB 70 also determines a further color balance parameter based on a further segment in the image, and applies a color balance based on the further color balance parameter to the further segment in the image.

[0170] Thus, as shown in Fig. 7, an image of the scene 61 may be corrected based on a multiple illuminants correction in the scene, where all light sources that illuminate the scene 61 may be corrected to a same “gray” illuminant.

[0171] It is noted that, in some embodiments, the AWB 70 determines a color balance parameter for the whole image (e.g., the RGB image 67) based on the various segments (e.g., the segments 65a, 65b and 65c) and applies the illuminant color estimation & correction 72 to the whole image based on the determined color balance parameter. In such a case, colors in the white balanced image 76 are corrected the same way in the whole image.

[0172] Fig. 8 illustrates a second embodiment of an AWB 80. The AWB 80 is a further example of the AWB 68a of Fig. 6. The AWB 80 applies a single illuminant correction to the RGB image 67 of the scene 61 based on weighting illuminants (light sources). For example, a WB correction and estimation may be focused based on a specific area.

[0173] The AWB 80 includes an illuminant source estimation & segmentation 81 (an example of the processing at 12 and 15 of Fig. 2).

[0174] The AWB 80 further includes an application 82 of a weighted average of pixels (which, for example, focuses on a specific area and determines weights of the weighted average such that the AWB 80 determines a color balance parameter for the AWB 80 based on an image segment determined at 81 that corresponds to the specific area).

[0175] The AWB 80 further includes an estimation 83 of colors of light sources in the focused area (i.e., in the determined segment), which is based on a result of the application 82 of the weighted average. The estimation 83 of light source colors weights colors of detected light sources according to a contribution of the light source to a light flux. The AWB 80 further includes an application 84 of a correction (color balance, AWB) to the whole image 67 based on a result of the estimation 83 of the light source colors, and outputs a white balanced image 85.

[0176] Fig. 9 illustrates a flow 90 for a color balance according to an embodiment.

[0177] At 91, a circuitry (e.g., the circuitry 1 of Fig. 1, or the general-purpose computer 250 of Fig. 14) detects photons and, at 92, collects photon timestamps per pixel (e.g., based on the array 2 of pixels).

[0178] Based on a list {T^7} of time stamps T / ‘’Jof arrivals of detected photons k for pixels with pixel coordinates i and j from the processing at 92, the circuitry performs an estimation of a flux frequency spectrum (and its flux frequency components) for every group of pixels (with {i 6 N, j 6 M}. For every group of pixels, a list 94 of detected spectral components is outputted.

[0179] At 95, further processing (e.g., clustering) is performed. It is noted that the further processing 95 is optional and is omitted in some embodiments.

[0180] At 96, a segmentation of illuminants per group of pixels is performed. The segmentation 96 is based on a result of the further processing 95 and / or of the estimation 93 of flux spectrum components, as well as on the RGB image 67 from the RGB sensor 66. The segmentation 96 outputs a list {s))7} of potential illuminants per group of pixels.

[0181] A list {s } of potential illuminants (light sources; identified according to their spectral components) is generated, and a weighting of the potential illuminants is determined for every group of pixels it is determined based on a estimated contribution of the respective potential illuminants to a light flux. There may be more than one illuminant (light source) per group of pixels.

[0182] The list {Sfc } of illuminants from 96 is an example of a color balance parameter, based on which the AWB 68a of the ISP pipeline 68 is based. The AWB 68a is further based on the raw image 67 of the RGB sensor 66 of Fig. 9.

[0183] A white balance algorithm 68a is applied based on the classification of illuminants (light sources) per group of pixels

[0184] The white balance algorithm 70 of Fig. 7 applies a different white balance correction per group of pixels. The white balance algorithm 80 of Fig. 8 applies one correction to all pixels, but this correction is selected according to a selected group of pixels that have a common light source. It is noted that shadow may also be classified as a light source (its flux spectrum may be very low over all frequencies).

[0185] In some embodiments, the circuitry is further configured to: estimate, based on the determined flux frequency component of the light source, a future time interval in which the light source is expected to be detectable in the determined segment.

[0186] In some embodiments, the circuitry is further configured to: control an overlap of an imaging exposure interval with the estimated future time interval. In some embodiments, the controlling of the overlap includes controlling the imaging exposure interval to overlap with the estimated future time interval. In some embodiments, the controlling of the overlap includes controlling the imaging exposure interval to not overlap with the estimated future time interval.

[0187] Thus, in case of a flickering light that should be visible in image data captured in the imaging exposure interval, the circuitry may control the imaging exposure interval to overlap with the estimated future time interval such that the flickering light may be visible in the image data. Vice versa, in case of an unwanted flickering light that should not be visible in image data captured in the imaging exposure interval, the circuitry may control the imaging exposure interval to not overlap with the estimated future time interval such that the flickering light may not be visible in the image data.

[0188] For example, the circuitry may mitigate a LED flickering problem. The LED flickering problem may be caused by LEDs that flicker (e.g., due to pulse-width modulation (PWM), due to a ripple in a power supply etc.) such that a conventional image sensor (e.g., CIS) that captures a sequence of images of the LED may capture the images in periods in which the LED is off. Consequently, the LED may not be visible (or may appear to be off) in the sequence of images.

[0189] Conventional image sensors (e.g., CIS) may utilize a short exposure time at daylight illumination, which may have a relatively low frame rate (e.g., 30 or 60 fps). For example, may be done to avoid image saturation, due to high photons flow at the (high) daylight illumination.

[0190] Due to this limitation, objects that utilize a high frequency LED, e.g., traffic lights, cars or road signs, may be missed in some or all frames. In some cases (e.g., automotive or assisted driving), missing an LED may be undesirable.

[0191] It may not be possible to detect with a conventional image data if a LED is missed in one or more frames, because of a low sampling rate of the conventional image sensor. For example, a flickering frequency of the LED may exceed 50 Hz. In some instances, a solution in conventional CIS sensors is to quantize an exposure time to a full cycle of the LED flicker. A difference to a desired sensitivity may be compensated by analog or digital gains.

[0192] Fig. 10 illustrates the LED flickering problem. A LED-based traffic light may exhibit a flicker with a flicker period ILED<11 ms (without limiting the disclosure to this value). A conventional camera may capture a sequence of image frames of the traffic light at daylight.

[0193] A graph 101 shows when the LED of the traffic light is on (high level), and a graph 102 shows when a shutter of the conventional camera causes the camera to capture an image frame (high level). Exposures are controlled to be short in order to avoid image saturation due to daylight conditions.

[0194] A first exposure overlaps with an emission period of the LED light, and an image frame 103 (that corresponds to frame N) captured at the first exposure shows a shining traffic light.

[0195] A second exposure does not overlap with an emission period of the LED light, such that the traffic light appears to be off in an image frame 104 (frame N+l) captured at the second exposure.

[0196] Likewise, a third exposer does not overlap with an emission period of the LED light, such that the traffic light appears to be off in an image frame 105 captured at the third exposure.

[0197] Fig. 11 illustrates an example of a multi -photodiode pixel. The multi -photodiode pixel includes a first photodiode SP1 (Dual Gain drive) and a second photodiode (Double Data Sampling (DDS) drive). The multi -photodiode pixel is used in some instances as a solution to the LED flickering problem in automotive sensors.

[0198] Modem automotive sensors include multi-photodiode pixels (as shown in Fig. 11), where one of the diodes (SP2) has very low sensitivity and can be configured for long exposure (“always open shutter”) without saturation.

[0199] The multi -photodiode pixel may have a significant ‘motion blur’ artifact, caused by long exposure, which may be a drawback.

[0200] This may be meaningful in automotive applications, where cameras and many objects may be in consistent motion.

[0201] Fig. 12 illustrates a pipeline 120 of a QIS-based LED flickering solution according to an embodiment. The pipeline 120 is performed by a circuitry (e.g., the circuitry 1 of Fig. 1, or the general -purpose computer 250 of Fig. 14). An RGB sensor 121 captures an RGB image of a LED that exhibits flickering, and provides the captured raw RGB image 122 to an image signal processor (ISP) 123. The ISP 123 generates, based on the raw RGB image 122, and outputs a (processed) RGB image 124. The ISP 123 further generates and outputs image brightness statistics 125 of the raw RGB image 122.

[0202] At 126, a single photon imager (e.g., the circuitry 1 of Fig. 1) captures a sequence of image data samples 127, which include photon timestamps, of the LED that exhibits flickering.

[0203] At 128, the circuitry performs a flux spectral components estimation and segmentation 128 (an example of the processing at 12 and 15 of Fig. 2) based on the captured sequence of image data samples 127.

[0204] At 129, the circuitry performs a LED flicker estimation 129, which yields flicker characteristics 130 of the LED. The flicker characteristics 130 include a flicker frequency (an example of a flux frequency component) and a time off set of the flicker.

[0205] At 131, an auto exposure is performed, which determines exposure settings 132 (e.g., including an analog gain). The auto exposure 131 is based on the flicker characteristics 130 and on the image brightness statistics 125.

[0206] The auto exposure 131 provides the determined exposure settings 132 to the RGB sensor 121. The RGB sensor 121 then captures further RGB image frames at an exposure timing based on the exposure settings 132 such that the LED appears to be on in the further RGB image frames.

[0207] The pipeline 120 is, for example, performed in a vehicle (or any suitable mobile platform) for assisted and / or autonomous driving, such that traffic signs, traffic lights, vehicle lights etc. may be sensed correctly by the RGB sensor 121.

[0208] Thus, in the pipeline 120, the circuitry estimates, based on a determined flux frequency component 130 of a light source (flickering LED), a future time interval in which the light source is expected to be detectable in a determined segment in the RGB image 122, and controls an imaging exposure interval of the RGB sensor 121 to overlap with the estimated future time interval. Accordingly, the circuitry may mitigate the LED flickering problem.

[0209] It is noted that, in some embodiments, the circuitry 1 of Fig. 1 or the general-purpose computer 250 of Fig. 14 includes the single photon imager 126 and the RGB sensor 121 (which may be included in a same sensor or may be provided as different sensors) as well as the ISP 123, and may perform the processing at 128, 129 and 131. In some embodiments, however, the ISP 123 is provided as separate device external to the circuitry 1 or general-purpose computer 250, respectively, while the RGB sensor 121 is included in the circuitry 1 or general- purpose computer 250. In some embodiments, the RGB sensor 121 and the ISP 123 are provided as separate devices external to the circuitry 1 or general-purpose computer 250. In some embodiments, the RGB sensor 121 and the ISP 123 are provided together, e.g., as a same device, as image sensors using a same aperture, as a same image sensor, or the like.

[0210] Accordingly, the LED flickering solution according to Fig. 12 is based on using a QIS. Due to the fact that QIS sensor may be not limited by a sample rate, there may be no Nyquist sampling frequency limitation to recognize a high LED flickering frequency (because the Nyquist sampling frequency that corresponds to the LED flickering frequency may still be significantly lower than a sample rate of the QIS).

[0211] Additionally, the LED flickering solution according to Fig. 12 may be performed for multiple segmented illumination sources (e g., traffic lights, traffic signs, vehicle lights etc.) separately.

[0212] The results 130 of a LED frequency detection (based on QIS) may be used as a source for the auto-exposure method 131, alongside image brightness statistics, to calculate appropriate RGB sensor settings 132.

[0213] An RGB sensor exposure time may be quantized to cycles of LED frequencies (as with conventional CIS). Remaining brightness component may be compensated by an analog gain (as with conventional CIS).

[0214] In some embodiments, compared to conventional CIS, the method according to Fig. 12 may be not limited by a sampling frequency (low CIS frame rate), and compared to a complex and expensive automotive sensors solution, the method according to Fig. 12 may minimize a motion blur to a minimum possible level.

[0215] In some embodiments, the circuitry is further configured to: detect, based on the determined flux frequency component of the light source, an ongoing time-of-flight (ToF) measurement in the segment of the image; and control a ToF sensor to avoid interference with the ongoing ToF measurement.

[0216] The ToF measurement may include a direct ToF (dToF) measurement and / or an indirect ToF (iToF) measurement. Since a ToF measurement may include determining a round-trip time of a light pulse, a result of the ToF measurement may be wrong when another light pulse (e g., from another ToF sensor that may be performing a ToF measurement) interferes with the ToF measurement.

[0217] When multiple ToF systems are illuminating a same scene at a same time, they may cause interference to each other. For example, light illuminated by a laser of one ToF system may be received by the other ToF system and may generate a wrong distance calculation for that ToF system. This phenomenon may increase with a number of active ToF systems in the same scene.

[0218] As an example, two dToF systems (dToF 1 and dToF 2) may be present in a same scene, and each one may illuminate short pulses at some periodicity. The two dToF systems may not be synchronized, and may not also not have a same periodicity. Pulses of dToF 1 may be received by dToF 2, and a histogram of dToF 2 may gave two peaks, one generated due to peaks of dToF 1, and the other one from dToF 2. Thus, the system dToF 2 may know which of the peaks is correct, and may calculate a distance wrongly based on the wrong peak caused by dToF 1.

[0219] The circuitry may be configured as a ToF (e.g., dToF and / or iToF) sensor. Before performing a ToF measurement, the circuitry may determine, based on the determined flux frequency component, whether another ToF sensor is performing a ToF measurement. If the circuitry detects an ongoing ToF measurement by another ToF sensor, the circuitry may avoid an interference with the ongoing ToF measurement.

[0220] The interference with the ongoing ToF measurement may include, e.g., waiting before starting a ToF measurement until the ongoing ToF measurement is finished, using another light pulse timing than the ongoing ToF measurement, using another light pulse wavelength than the ongoing ToF measurement, or the like.

[0221] To overcome the issue of interfering ToF measurements, a spectral component analysis block (e.g., the processing at 12 and 15 of Fig. 2) may be used, which may identify additional light sources (such as another ToF system that may be performing a ToF measurement). Once additional light sources (e g., laser sources) are identified, a dToF system may take multiple actions, for example, deactivating a dToF operation until a detected interferer is no longer active, or synchronizing a dToF operation with an interferer activity profile. The synchronizing with the interferer activity profile may include changing a Tx (and Rx) activation profile time (e.g., changing a frequency), and / or adapting a noise statistics block to neglect ‘colored’ noise that happens due to the interferer. Colored noise may be simply a peak generated by an interferer with a same frequency as the dToF system.

[0222] Fig. 13 illustrates a ToF system 140 according to an embodiment. The ToF system 140 is configured as a dToF system and includes a transmission (Tx) portion 141 (e.g., including a vertical -cavity surface-emitting laser (VCSEL)), a reception (Rx) portion 142 (e.g., including a SPAD) and a control section 143 that controls the Tx portion 141 and the Rx portion 142. The control portion 143 may also be included in the Rx portion 142.

[0223] The Rx portion 142 generates and outputs ToF data, photon counting (PC) data or the like. The Rx portion 142 includes a dToF sensor 150. The dToF sensor 150 includes a SPAD analog circuitry 151, a time-to-digital (TDC) conversion portion 152 that converts analog signals from the SPAD analog circuitry 151 into digital signals, a histogram builder 153 that generates a histogram based on the digital signals from the TDC conversion portion 152, a portion 154 for noise statistics and peak detection based on the histogram, and a readout 155 that generates, based on a result of the portion 154, and outputs distances 156.

[0224] The dToF sensor 150 further includes a flux spectral block 157 that receives an output of one or more of the TDC portion 152, the histogram builder 153 and the portion 154. The flux spectral block 157 is configured to perform the processing at 12 and 15 of Fig. 2 and to output a map 158 of image segments (an example of the map 8 of image segments of Fig. 1).

[0225] The dToF sensor 150 further includes a processor / logic block 159 that receives the map 158 of image segments. The processor / logic block 159 performs an activation (e.g. time / frequency) modification 161 of the control portion 143 and configures the portion 154 (noise & peak block).

[0226] The dToF sensor 150 is an example of the circuitry 1 of Fig. 1. Thus, the circuitry may mitigate an interference of dToF systems.

[0227] As mentioned, the disclosure proposes integrating a flux spectral components block (e.g., the processing at 12 and 15 of Fig. 2) into a single photon imager (e.g., the circuitry 1 of Fig. 1), using a single photon imager to estimate different light sources and segment a FoV accordingly, using a segmentation of light sources to improve an image quality (e.g., for controlling an AWB, and / or for mitigating a LED flickering problem), and / or mitigating an interference in ToF system performance. Thus, an image quality of an image captured by an image sensor may be enhanced based on a single photon image sensor.

[0228] Fig. 14 illustrates an embodiment of a general-purpose computer 250. The general-purpose computer 250 can be implemented such that it can basically function as any type of circuitry, for example, a smartphone, smart glasses, a head-mounted display, a smartwatch, a mobile phone, a mobile tablet, a notebook, a terminal device or the like. The general-purpose computer 250 is an example of circuitry that is configured to perform the method according to the present technology (e.g., the method of Fig. 2, 4, 5, 6, 7, 8, 9, 12, or 13). The computer has components 251 to 261, which can form a circuitry, such as any one of the unit 3, the unit 4, the unit 5, the unit 6, the unit 7, or the like, as described herein.

[0229] Embodiments which use software, firmware, programs or the like for performing the methods as described herein can be installed on computer 250, which is then configured to be suitable for the concrete embodiment. The computer 250 has a CPU 251 (Central Processing Unit), which can execute various types of procedures and methods as described herein, for example, in accordance with programs stored in a read-only memory (ROM) 252, stored in a storage 257 and loaded into a random-access memory (RAM) 253, stored on a medium 260 which can be inserted in a respective drive 259, etc.

[0230] Furthermore, the computer 250 includes an artificial intelligence (Al) processor 251a. The Al processor 251a may include a graphics processing unit (GPU) and / or a tensor processing unit (TPU). The Al processor 251a may be configured to execute an Al model (e.g., an artificial neural network).

[0231] The CPU 251, the ROM 252 and the RAM 253 are connected with a bus 261, which in turn is connected to an input / output interface 254. The number of CPUs, memories and storages is only exemplary, and the skilled person will appreciate that the computer 250 can be adapted and configured accordingly for meeting specific requirements which arise when it functions as an information processing apparatus according to the present technology.

[0232] At the input / output interface 254, several components are connected: an input 255, an output 256, the storage 257, a communication interface 258 and the drive 259, into which a medium 260 (compact disc (CD), digital video disc (DVD), universal serial bus (USB) flash drive, secure digital (SD) card, CompactFlash (CF) memory, or the like) can be inserted.

[0233] The input 255 can be a pointer device (mouse, graphic table, or the like), a keyboard, a microphone, a camera, a touchscreen, an eye-tracking unit etc.

[0234] The output 256 can have a display (liquid crystal display (LCD), cathode ray tube (CRT) display, light-emitting diode (LED) display, electronic paper, etc.; e.g., included in a touchscreen), loudspeakers, etc.

[0235] The storage 257 can have a hard disk drive (HDD), a solid-state drive (SSD), a flash drive and the like.

[0236] The communication interface 258 can be adapted to communicate, for example, via universal serial bus (USB), a serial port (RS-232), parallel port (IEEE 1284), a local area network (LAN; e.g., ethemet), wireless local area network (WLAN; e.g., Wi-Fi, IEEE 802.11), mobile telecommunications system (GSM, UMTS, LTE, NR etc ), Bluetooth, near-field communication (NFC), ZigBee, infrared, etc.

[0237] It should be noted that the description above only pertains to an example configuration of computer 250. Alternative configurations may be implemented with additional or other sensors, storage devices, interfaces or the like. For example, the communication interface 258 may support other radio access technologies than the mentioned UMTS, LTE and NR.

[0238] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0239] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding. Changes of the ordering of method steps may be apparent to the skilled person.

[0240] It is noted that image data frames are described in some embodiments as examples of image data samples for illustration purposes only, and that embodiments which use image data frames as examples of image data samples may be adapted to use sparse image data frames or any other type of sparse image data samples instead of full image data frames.

[0241] Please note that the division of the circuitry 1 into units 2 to 7 is only made for illustration purposes and that the present disclosure is not limited to any specific division of functions in specific units. For instance, the circuitry 1 could be implemented by a respective programmed processor, field programmable gate array (FPGA) and the like. For instance, one or more of the units 2 to 7 may be provided separately from the circuitry 1.

[0242] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0243] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0244] It is noted that the embodiments described herein may be combined in any suitable way.

[0245] Note that the present technology can also be configured as described below. (1) Circuitry, configured to: obtain a sequence of image data samples captured by a single photon imaging sensor; determine, based on the sequence of image data samples, a flux frequency component of a light source; and determine a segment in an image based on the determined flux frequency component of the light source.

[0246] (2) The circuitry of (1), wherein the determining of the segment includes: determining spatial portions of the sequence of image data samples in which the flux frequency component of the light source is detected; and determining, as the segment in the image, a portion in the image that corresponds to the determined spatial portions of the sequence of image data samples.

[0247] (3) The circuitry of (1) or (2), wherein the determining of the segment is further based on a feature detected in the image.

[0248] (4) The circuitry of any one of (1) to (3), wherein the determining of the flux frequency component of the light source includes: detecting a flux variation in the sequence of image data samples; and determining a frequency component of the detected flux variation.

[0249] (5) The circuitry of (4), wherein the flux variation corresponds to a difference in a number of detected photons between image data samples of the sequence of image data samples.

[0250] (6) The circuitry of any one of (1) to (5), wherein the image data samples of the sequence of image data samples associate numbers of photons detected by the single photon imaging sensor with spatial portions of the image data samples.

[0251] (7) The circuitry of any one of (1) to (6), wherein the determining of the flux frequency component is based on a predefined wavelength range of light emitted by the light source.

[0252] (8) The circuitry of any one of (1) to (7), further configured to: determine a color balance parameter based on the determined segment in the image. (9) The circuitry of (8), wherein the color balance parameter indicates the determined flux frequency component of the light source.

[0253] (10) The circuitry of (8) or (9), further configured to: apply a color balance based on the determined color balance parameter to the image.

[0254] (11) The circuitry of any one of (8) to (10), further configured to: determine a further color balance parameter based on a further segment in the image; apply a color balance based on the determined color balance parameter to the determined segment in the image; and apply a color balance based on the further color balance parameter to the further segment in the image.

[0255] (12) The circuitry of any one of (1) to (11), further configured to: estimate, based on the determined flux frequency component of the light source, a future time interval in which the light source is expected to be detectable in the determined segment.

[0256] (13) The circuitry of (12), further configured to: control an overlap of an imaging exposure interval with the estimated future time interval.

[0257] (14) The circuitry of (13), wherein the controlling of the overlap includes controlling the imaging exposure interval to overlap with the estimated future time interval.

[0258] (15) The circuitry of (13), wherein the controlling of the overlap includes controlling the imaging exposure interval to not overlap with the estimated future time interval.

[0259] (16) The circuitry of any one of (1) to (15), further configured to: detect, based on the determined flux frequency component of the light source, an ongoing time-of-flight measurement in the segment of the image; and control a time-of-flight sensor to avoid interference with the ongoing time-of-flight measurement.

[0260] (17) The circuitry of any one of (1) to (16), wherein the determining of the flux frequency component of the light source is based on digital Fourier transform.

[0261] (18) The circuitry of any one of (1) to (17), further configured to: output an indication of the determined flux frequency component of the light source. (19) The circuitry of any one of (1) to (18), further configured to: output an indication of the determined segment in the image.

[0262] (20) The circuitry of any one of (1) to (19), wherein the single photon imaging sensor is configured as a quanta imaging sensor.

[0263] (21) The circuitry of any one of (1) to (20), wherein the single photon imaging sensor includes a single-photon avalanche diode.

[0264] (22) The circuitry of any one of (1) to (21), wherein the single photon imaging sensor is configured as direct time-of-flight sensor.

[0265] (23) A method, comprising: obtaining a sequence of image data samples captured by a single photon imaging sensor; determining, based on the sequence of image data samples, a flux frequency component of a light source; and determining a segment in an image based on the determined flux frequency component of the light source.

[0266] (24) The method of (23), wherein the determining of the segment includes: determining spatial portions of the sequence of image data samples in which the flux frequency component of the light source is detected; and determining, as the segment in the image, a portion in the image that corresponds to the determined spatial portions of the sequence of image data samples.

[0267] (25) The method of (23) or (24), wherein the determining of the segment is further based on a feature detected in the image.

[0268] (26) The method of any one of (23) to (25), wherein the determining of the flux frequency component of the light source includes: detecting a flux variation in the sequence of image data samples; and determining a frequency component of the detected flux variation.

[0269] (27) The method of (26), wherein the flux variation corresponds to a difference in a number of detected photons between image data samples of the sequence of image data samples.

[0270] (28) The method of any one of (23) to (27), wherein the image data samples of the sequence of image data samples associate numbers of photons detected by the single photon imaging sensor with spatial portions of the image data samples.

[0271] (29) The method of any one of (23) to (28), wherein the determining of the flux frequency component is based on a predefined wavelength range of light emitted by the light source.

[0272] (30) The method of any one of (23) to (29), further comprising: determining a color balance parameter based on the determined segment in the image.

[0273] (31) The method of (30), wherein the color balance parameter indicates the determined flux frequency component of the light source.

[0274] (32) The method of (30) or (31), further comprising: applying a color balance based on the determined color balance parameter to the image.

[0275] (33) The method of any one of (30) to (32), further comprising: determining a further color balance parameter based on a further segment in the image; applying a color balance based on the determined color balance parameter to the determined segment in the image; and applying a color balance based on the further color balance parameter to the further segment in the image.

[0276] (34) The method of any one of (23) to (33), further comprising: estimating, based on the determined flux frequency component of the light source, a future time interval in which the light source is expected to be detectable in the determined segment.

[0277] (35) The method of (34), further comprising: controlling an overlap of an imaging exposure interval with the estimated future time interval.

[0278] (36) The method of (35), wherein the controlling of the overlap includes controlling the imaging exposure interval to overlap with the estimated future time interval.

[0279] (37) The method of (35), wherein the controlling of the overlap includes controlling the imaging exposure interval to not overlap with the estimated future time interval. (38) The method of any one of (23) to (37), further comprising: detecting, based on the determined flux frequency component of the light source, an ongoing time-of-flight measurement in the segment of the image; and controlling a time-of-flight sensor to avoid interference with the ongoing time-of-flight measurement.

[0280] (39) The method of any one of (23) to (38), wherein the determining of the flux frequency component of the light source is based on digital Fourier transform.

[0281] (40) The method of any one of (23) to (39), further comprising: outputting an indication of the determined flux frequency component of the light source.

[0282] (41) The method of any one of (23) to (40), further comprising: outputting an indication of the determined segment in the image.

[0283] (42) The method of any one of (23) to (41), wherein the single photon imaging sensor is configured as a quanta imaging sensor.

[0284] (43) The method of any one of (23) to (42), wherein the single photon imaging sensor includes a single-photon avalanche diode.

[0285] (44) The method of any one of (23) to (43), wherein the single photon imaging sensor is configured as direct time-of-flight sensor.

[0286] (45) A computer program comprising program code causing a computer to perform the method according to anyone of (23) to (44), when being carried out on a computer.

[0287] (46) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to anyone of (23) to (44) to be performed.

Claims

CLAIMS1. Circuitry, configured to: obtain a sequence of image data samples captured by a single photon imaging sensor; determine, based on the sequence of image data samples, a flux frequency component of a light source; and determine a segment in an image based on the determined flux frequency component of the light source.

2. The circuitry of claim 1, wherein the determining of the segment includes: determining spatial portions of the sequence of image data samples in which the flux frequency component of the light source is detected; and determining, as the segment in the image, a portion in the image that corresponds to the determined spatial portions of the sequence of image data samples.

3. The circuitry of claim 1 , wherein the determining of the flux frequency component of the light source includes: detecting a flux variation in the sequence of image data samples; and determining a frequency component of the detected flux variation.

4. The circuitry of claim 3, wherein the flux variation corresponds to a difference in a number of detected photons between image data samples of the sequence of image data samples.

5. The circuitry of claim 1, wherein the image data samples of the sequence of image data samples associate numbers of photons detected by the single photon imaging sensor with spatial portions of the image data samples.

6. The circuitry of claim 1, further configured to: determine a color balance parameter based on the determined segment in the image.

7. The circuitry of claim 6, further configured to: determine a further color balance parameter based on a further segment in the image; apply a color balance based on the determined color balance parameter to the determined segment in the image; and apply a color balance based on the further color balance parameter to the further segment in the image.

8. The circuitry of claim 1, further configured to: estimate, based on the determined flux frequency component of the light source, a future time interval in which the light source is expected to be detectable in the determined segment.

9. The circuitry of claim 8, further configured to: control an imaging exposure interval to overlap with the estimated future time interval.

10. The circuitry of claim 1, further configured to: detect, based on the determined flux frequency component of the light source, an ongoing time-of-flight measurement in the segment of the image; and control a time-of-flight sensor to avoid interference with the ongoing time-of-flight measurement.

11. A method, comprising: obtaining a sequence of image data samples captured by a single photon imaging sensor; determining, based on the sequence of image data samples, a flux frequency component of a light source; and determining a segment in an image based on the determined flux frequency component of the light source.

12. The method of claim 11, wherein the determining of the segment includes: determining spatial portions of the sequence of image data samples in which the flux frequency component of the light source is detected; and determining, as the segment in the image, a portion in the image that corresponds to the determined spatial portions of the sequence of image data samples.

13. The method of claim 11, wherein the determining of the flux frequency component of the light source includes: detecting a flux variation in the sequence of image data samples; and determining a frequency component of the detected flux variation.

14. The method of claim 13, wherein the flux variation corresponds to a difference in a number of detected photons between image data samples of the sequence of image data samples.

15. The method of claim 11, wherein the image data samples of the sequence of image data samples associate numbersof photons detected by the single photon imaging sensor with spatial portions of the image data samples.

16. The method of claim 11, further comprising: determining a color balance parameter based on the determined segment in the image.

17. The method of claim 16, further comprising: determining a further color balance parameter based on a further segment in the image; applying a color balance based on the determined color balance parameter to the determined segment in the image; and applying a color balance based on the further color balance parameter to the further segment in the image.

18. The method of claim 11, further comprising: estimating, based on the determined flux frequency component of the light source, a future time interval in which the light source is expected to be detectable in the determined segment.

19. The method of claim 18, further comprising: controlling an imaging exposure interval to overlap with the estimated future time interval.

20. The method of claim 11, further comprising: detecting, based on the determined flux frequency component of the light source, an ongoing time-of-flight measurement in the segment of the image; and controlling a time-of-flight sensor to avoid interference with the ongoing time-of-flight measurement.

Images