Anti-flicker filter for DTOF sensor

Abstract

Various embodiments of the present disclosure relate to anti-blooming filters for DTOF sensors. A method includes detecting, at respective successive time instants, successive sets of targets from a first histogram signal delivered by a sensor device; determining a current histogram output for a current set of currently detected targets, the current histogram output having, for each currently detected target of the current set, a set of current parameters including a confidence indicator stored in a memory; performing a matching operation between the current set of currently detected targets and a previous set of previously detected targets stored in the memory; and performing a filtering operation on at least one parameter of the set of current parameters of at least some of the currently detected targets of the current set based on a result of the matching operation, the filtering operation weighted based on at least the confidence indicators of the current and previous sets of detected targets.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to French application No. 2103003 filed on March 25, 2021, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates generally to time-of-flight (ToF), and in particular embodiments to anti-flickering filters used in ToF devices. BACKGROUND

[0004] Devices for determining the distance (or range) to an object or target are known. One such method is called “time-of-flight” (ToF), which comprises sending a light signal to an object and measuring the time taken for the signal to propagate to the object and back to the device.

[0005] Direct time-of-flight (dToF) devices directly measure the time taken for the signal to propagate to the object and back to the device. Indirect time-of-flight (iToF) devices calculate the time taken for the signal to propagate by measuring the phase shift between the signal from the light source and the signal reflected from the object and detected by a light sensor - knowing this phase shift and the speed of light enables the distance to the object to be determined.

[0006] Single-photon avalanche diodes (SPADs) can be used to detect reflected light. Typically, an array of SPADs is provided as a sensor for detecting reflected light pulses. Photons can generate charge carriers in a SPAD through the photoelectric effect. The photo-generated charge carriers can trigger an avalanche current in one or more SPADs in the array of SPADs. The avalanche current can signal an event, i.e. that a photon of light has been detected.

[0007] Time-of-flight histogram processing aims at detecting objects (targets) in the field of view of a device and estimating their distance to the sensor. This processing (Histogram Pipeline “HIP”) is embedded in the ToF sensor firmware (FW) and can be used by a smartphone to drive its camera auto-focus.

[0008] For example, the HIP uses a statistical hypothesis test to determine the presence of potential targets in the field of view: it is the role of a pulse segmenter (“PS”) or pulse detector. The PS finds statistically significant pulses in the histogram reported by the hardware (HW) of the ToF (e.g. dToF) sensor. Due to the statistical nature of this detection process and the photon noise, targets can “flicker” (i.e. they can be detected at a certain moment, then lost at the next moment, then come back again).

[0009] Flickering is inherent to this technology. Flickering is a temporal effect when the signal reflected by the target is close to the detectability threshold, because (i) the target is far away, (ii) the target reflectivity is low, or (iii) the target covers only a small fraction of the field of view (FoV) (of the sensor's area). This is called type 1 flickering.

[0010] Some other types of flickering can occur, for example "competition" of several targets with equivalent signals when only one needs to be reported (type 2), or successive reporting of close targets as separate or merged (type 3)

[0011] Several tracks are followed to reduce the flickering effect. The first one tries to avoid the problem by avoiding statistical detection when not needed. But this is not possible in all cases, because autofocus needs target distance identification. The second one tries to solve the problem using a hysteresis filter (dynamic adaptation of the PS threshold), which reduces the temporal effect but does not eliminate it. The "threshold curse" is still there, and it does not solve flickering types greater than 1. Moreover, a delay is introduced. The third one tries to solve the problem by using and exploiting spatial correlation. But this is not possible in all cases, for example for single-area sensors, and the (X,Y) resolution is further degraded. SUMMARY

[0012] According to embodiments, a flickering filtering method and system is proposed with the following advantages: (i) tight hardware coupling for target confidence computation (in firmware), (ii) anti-flickering filter (AFF) allowing to drastically reduce flickering of the histogram pipe output, (iii) keeping the delay at a reasonable level, (iv) adjustable trade-off between stability and delay, (v) handling multiple targets, (vi) detecting false long tail targets, (vii) not degrading spatial resolution (nor making assumptions on the scene).

[0013] According to embodiments, it is suggested to use a confidence measure on detected targets. This confidence measure is advantageously computed in FW using HW histogram counters inside HIP, by performing a temporal filtering with "short term" memory (direct part) and "long term" memory (feedback part) using weights of a filter with dynamic and double influence (for considering temporal effect of flickering), and valuing more recent past (temporal influence) and more confident data (confidence influence) than older ones, handling lost targets to "fill" holes in filter input, performing potential target database (memory) management taking into account previous confidence on targets, and using "affinity" criteria to handle multiple targets.

[0014] According to one aspect, a method comprises detecting, at a respective time instant, a successive set of targets from a first histogram signal (raw histogram) delivered by a sensor device; determining a current histogram output for a current set of detected targets, the current histogram output having, for each current detected target of the current set, a set of current parameters including a confidence indicator stored in a memory; performing a matching operation between the current set of detected targets and a previous set of detected targets stored in the memory; and based on a result of the matching operation, performing a temporal filtering operation on at least one parameter of the set of current parameters of at least some current detected targets of the current set, the temporal filtering operation being advantageously weighted based at least on the confidence indicators of the current and previous sets of detected targets.

[0015] The temporal filtering operation can be performed, for example but not limited to, by using an IIR filter (i.e. the parameters in the memory can be taken from previous input targets or from previous filtered output targets). Other types of filters can be used, for example FIR filters.

[0016] The confidence indicator of a current detected target can be based on a Z-score of a target in the first histogram signal (raw histogram).

[0017] According to one embodiment, the set of current parameters includes a speed, an ambient speed and a distance (range).

[0018] According to one embodiment, the matching operation includes using an affinity condition.

[0019] According to one embodiment, the affinity condition includes a condition based on the distance (range).

[0020] According to one embodiment, the affinity condition includes a condition based on the speed.

[0021] According to one embodiment, the affinity condition includes a condition based on the confidence indicator.

[0022] According to one embodiment, the affinity condition includes a distance-based affinity matrix.

[0023] According to one embodiment, a current detected target matches one previous target stored in the memory at the end of the matching operation.

[0024] According to one embodiment, a current detected target does not match any previous target stored in the memory at the end of the matching operation.

[0025] In this case, the current detected target can be added to the memory if an empty slot is available.

[0026] As a variant, a selective memory replacement can be performed based on the confidence indicator used to store the current detected target.

[0027] According to one embodiment, the previous target in the memory does not match in the current set of detected targets. A new target ("virtual" target) is added to the input of the filtering operation with a new confidence indicator.

[0028] The new confidence indicator can be very low if the corresponding previous target is very confident.

[0029] The new confidence indicator can be close to the previous confidence indicator if the previous confidence indicator is low.

[0030] According to one embodiment, the method comprises a double weighted filtering operation based on the target parameters and the confidence indicators.

[0031] The sensor device can comprise a ToF device, in particular a dToF device.

[0032] According to another aspect, a system is proposed, the system having: a sensor device configured to deliver a first histogram signal to allow detecting successive sets of targets at respective successive time instants; a first processor configured to determine a current histogram output for a current set of detected targets, the current histogram output storing, for each current detected target of the current set, a set of current parameters including a confidence indicator in a memory; a second processor configured to perform a matching operation between the current set of detected targets and a previous set of detected targets stored in the memory, and to perform a temporal filtering operation on at least one parameter of the set of current parameters of at least some current detected targets of the current set based on a result of the matching operation, the temporal filtering operation being advantageously weighted based at least on the confidence indicators of the current and previous sets of detected targets.

[0033] The different features pointed out above for the different embodiments of the method defined above can be incorporated into different embodiments of the system.

[0034] Moreover, according to a preferred embodiment, the first processor and the second processor can be implemented as firmware.

[0035] As a variant, the first processor comprises a first circuit configured to determine the confidence indicator and a second circuit configured to determine the other parameters. The first circuit and the second processor are implemented as software and the second circuit is implemented as firmware. BRIEF DESCRIPTION OF DRAWINGS

[0036] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings in which:

[0037] Figure 1 is a block diagram of an embodiment system;

[0038] Figure 2 is an embodiment histogram;

[0039] Figure 3 is a block diagram of circuitry to process histogram data;

[0040] Figure 4 is an embodiment target Z-score plot;

[0041] Figure 5 is a block diagram of an embodiment second processor of a system;

[0042] Figure 6 is a block diagram of embodiment circuitry of a second processor;

[0043] Figure 7 is an embodiment for affinity criteria requirements;

[0044] Figure 8 is an embodiment of a proposer preference matrix and an acceptor preference matrix;

[0045] Figure 9 is a flowchart of an embodiment method;

[0046] Figure 10 and Figure 11 is a schematic diagram of a mirror Z-score embodiment;

[0047] Figure 12 is a block diagram of an embodiment anti-flicker filter;

[0048] Figure 13 is a diagram of an embodiment weighted filtering operation;

[0049] Figure 14 is a diagram of an embodiment double weighted filtering operation;

[0050] Figure 15 is a block diagram of an embodiment system; and

[0051] Figure 16 illustrates embodiments of the invention. DETAILED DESCRIPTION

[0052] In various embodiments, a time-of-flight (ToF) device can refer to a direct time-of-flight (dToF) or an indirect time-of-flight (iToF) type of device, unless otherwise specified. One advantageous application, but not limited to such application, is for range estimation for autofocus assistance.

[0053] Figure 1A schematic of an example system is shown in Fig. 1. The system comprises a sensor device 101 configured to receive a light input 100 returned from an object after reflection of an incident signal onto the object. The sensor device 101 is configured to output a first histogram signal (i.e. a raw histogram 102) delivered to a first processor 103.

[0054] In an embodiment, the sensor device 101 is a multi-region sensor. The first histogram signal allows to detect successive sets of targets at respective successive time instants. The first processor 103 forms a histogram pipeline HIP and is configured to determine a current set of currently detected targets Target#i, Target#i', Target#i" and a current histogram output.

[0055] The current histogram output comprises, for each currently detected target of the current set (e.g. Target#i), a set of current parameters PRMi stored in a memory MM. In an embodiment, the set of parameters PRMi comprises a distance or range (d_i) representing a distance between the sensor device and an object within a "field of view" (FoV) of the sensor device, a velocity (r_i), an ambient velocity (ambient_rate_i) and a confidence indicator (confidence_i).

[0056] The first processor 103 comprises a first circuit 1031 for determining the confidence indicator in firmware (FW) advantageously using a hardware (HW) histogram counter. The first processor 103 further comprises a second circuit 1032 for determining other parameters sigma_d_i, sigma_r_i, sigma_d_i, sigma_r_i, respectively standard deviations of d_i and r_i.

[0057] The system further comprises a second processor AFF configured to perform a matching operation between the current set of currently detected targets and a previous set of detected targets stored in the memory MM. The second processor AFF is configured to perform a temporal filtering operation on at least one parameter of the set of current parameters of at least some of the currently detected targets of the current set based on a result of the matching operation. Advantageously, the temporal filtering operation is weighted based at least on the confidence indicator.

[0058] The second processor AFF delivers a filtered parameter PRMF and is configured to handle a flickering effect. In an embodiment, the second processor comprises a first circuit MD1 configured to perform the matching operation and a second circuit MD2 configured to perform the filtering operation. The second circuit can be considered as a filter, also called an anti-flickering filter.

[0059] In an embodiment, the system comprises a control device CTRLM, for example implemented by software, configured to control the first and second processors.

[0060] The sensor device 101 and the second circuit 1032 can be those described in US Patent Application No. US2018 / 0253404A1 (incorporated herein by reference in its entirety). Some sensor device 101 and second circuit features are now described in detail and one skilled in the art can refer to US Patent Application No. US2018 / 0253404A1 for all useful purposes.

[0061] The sensor device 101 can comprise a light source (for example, a vertical-cavity surface-emitting laser, VCSEL, or a light-emitting diode, LED), a suitable light source driver, an array of single-photon avalanche diode (SPAD) elements, and a time-to-digital converter (TDC) or a histogram generator configured to output a raw histogram signal or a raw histogram 102 representing a number of time-periods of photon event detection counts.

[0062] However, other configurations can be implemented, for example, a sensor device with a light source, a light source driver, and a fast photodiode detector with controlled charge, an analog-to-digital converter (ADC), etc.

[0063] In an embodiment, the sensor device 101 comprises a generator / driver configured to provide a periodic electrical signal (for example, the periodic signal can be a square signal, a sinusoidal signal, or any suitable periodic signal). The generator / driver can power the light source, thereby generating a modulated light output.

[0064] An example of a light source can be a light-emitting diode or any known illumination device, for example, a laser diode. The signal emitted from the light source is an incident signal that is transmitted to the object and reflected by the object.

[0065] The reflected light signal (i.e., Figure 1 The light input 100 is detected by a sensor of the sensor device 101, which in the following example comprises a SPAD array. A SPAD array can integrate several photodetection elements in the form of single-photon avalanche diodes or "SPADs" (also known as Geiger-mode avalanche photodiodes). These devices have a reverse-biased p-n junction in which a photo-generated carrier can trigger an avalanche current by a collision ionization mechanism. A SPAD can be designed to operate at a reverse-bias voltage much higher than the breakdown voltage.

[0066] The diode is biased at an initial time to a voltage value greater than its breakdown voltage. Receiving a photon in the diode junction region initiates an avalanche in the diode, resulting in an electrical pulse. The diode is then biased back to a voltage less than the breakdown voltage, so that the SPAD is again responsive to the reception of photons. The SPADs can currently be used in cycles with reactivation periods shorter than ten nanoseconds (ns). Thus, the SPADs can be used at high frequencies for detecting objects at relatively short distances from the measuring device, for example, distances from a few millimeters to a few meters. In different embodiments, different ranges can be supported.

[0067] Such a detection element can also be a fast charge-transfer photodiode. In embodiments, the signal output by the sensor is a signal phase-shifted by the generator / driver by an angle related to the distance to the object.

[0068] A time-to-digital converter (TDC) circuit can be configured to receive the signals generated by the generator / driver and the sensor and to calculate the phase shift (or time difference) between these signals to obtain the distance to the object. The detector (array of SPADs) is configured to generate many fast readings in a short period of time. Thus, the time-to-distance converter circuit can be configured to generate a histogram of detection events. For example, using a periodic pulse system, the arrival times of detection events can be quantified to build histogram data that can be post-processed to identify the positions / distances of multiple targets within the field of view.

[0069] Figure 2 An example of a first histogram generated (i.e., the raw histogram 102) is shown in FIG. 3. The raw histogram 102 is shown as a series of bars. Each bar represents the frequency or count of detected events for a particular time period or interval. As shown, two target reflections can be seen. The first target A 301 is shown as a peak above the ambient level, which occurs earlier than the second target B 303 peak.

[0070] In some embodiments, another reference detector within the sensor device is used. The reference detector is configured to receive light emitted by the light source and to be close to the light source. The distance to the object can be determined using the time difference between the reference detector receiving light from the light source and the sensor receiving light from the reflected object.

[0071] The histogram data-based distance determination analysis and filtering shown below is applicable to any time-of-flight ranging application, such as gesture analysis (where the range output can be used as an input to an algorithm to detect the motion of a hand or other object for a control interface) and autofocus assistance (where the range output can be used as a guide to an autofocus scheme to speed up the camera focusing time).

[0072] In some embodiments, the first processor 103 is physically integrated with the sensor device 101. In other words, the processing device is implemented in the same device as the sensor device, for example, by using register transfer level (RTL) synthesis, or by being implemented within an onboard microcontroller unit (MCU) on the sensor device.

[0073] In some embodiments, the first processor 103 is logically integrated with the sensor device 101, but is implemented on a separate dedicated processing device.

[0074] In some other embodiments, the first processor 103 is implemented as software on a host device configured to receive output from the sensor device 101.

[0075] like Figure 3 As shown, the second circuit 1032 of the first processor 103 includes various processing circuits capable of processing raw histogram data. In some embodiments, the second circuit 1032 includes some, but not necessarily, the processing circuits described below.

[0076] In one embodiment, the second circuit 1032 includes a stacking corrector circuit 201. The stacking corrector circuit 201 is configured to receive raw histogram data from the sensor device 101, process the data, and correct stacking errors in the histogram data.

[0077] In one embodiment, the corrected histogram data is passed to the crosstalk detector circuit 203. In another embodiment, the corrected histogram data is passed to the pulse detector circuit, the parameter extractor circuit, or both.

[0078] In one embodiment, the crosstalk detector circuit 203 is configured to receive the output of the stacking corrector circuit 201. In some embodiments where the stacking corrector circuit 201 is absent, the crosstalk detector circuit 203 is configured to directly receive the raw histogram data.

[0079] The crosstalk detector circuit 203 can therefore determine whether the histogram data includes crosstalk components and attempt to remove at least some of these components.

[0080] In an embodiment, the crosstalk detector circuit 203 outputs the corrected histogram data to the pulse detector circuit or the pulse segmenter (PS) circuit 205, the parameter extractor circuit 207, or both.

[0081] The pulse detector circuit or pulse segmenter (PS) circuit 205 can be configured to receive the output of the crosstalk detector circuit 203 (or the pile-up corrector circuit 201) and detect and filter (or isolate or blank) pulses in the histogram data. The operation of the pulse detector circuit will be described in more detail in a later application. In embodiments, the determined pulses are passed to the parameter extractor circuit 207.

[0082] In embodiments, the parameter extractor circuit 207 is configured to receive the output of the crosstalk detector circuit 203, the pulse segmenter (PS) circuit 205, or both, and extract suitable parameters, e.g. ranges, from the detected pulses in the histogram data. These parameters can be the output.

[0083] The first circuit 1031 is configured to determine a confidence indicator and is advantageously coupled to the output of the crosstalk detector circuit 203 or the parameter extractor circuit 207 and linked to the pulse segmenter (PS) circuit 205. The confidence indicator of a target can be based on the Z-score of that target in the original histogram 102.

[0084] Figure 4 The determination of the Z-score of a target (i.e. target Z-score) is explained in more detail in Figure 4 In the

[0085] In the Figure 4 In the

[0086] The number of time segments that constitute a detected pulse according to the pulse boundaries determined by the pulse segmenter is referred to herein as PulseWidth.

[0087] For the calculation of the target Z-score, the average of the event detection counts caused by ambient light photons in the time segments that constitute a detected pulse is considered. This average follows a probability law (e.g. the Gaussian law) whose average is Aestand whose standard deviation is A est divided by the square root of PulseWidth.

[0088] A est The estimated average of the event detection counts per time segment due to ambient light photons (referred to as “ambient per segment”) is Aest= Aest / PulseWidth. Figure 4 as shown in est equal to 1000 counts.

[0089] The target Z-score is the Z-score (also called standard score) of the PulseMean with respect to this probability law, where the PulseMean is the average of the event detection counts over the time period that constitutes a detected pulse.

[0090] PulseMean - A est is the average target signal (i.e. average useful signal). It is an estimate of the average of the event detection counts per time period due to the reflection of the photons emitted by the light source that forms part of the device on the target.

[0091] The target Z-score is therefore an indication of how much above the standard deviation of the noise (ambient light level) the average target signal is. The target Z-score is a good indicator of the ability of the pulse segmenter to detect the target, as the pulse segmenter threshold is the minimum Z-score that the target must have for it to be output by the pulse segmenter.

[0092] The confidence indicator is an estimate of the correctness of some of the parameters in the set of target parameters PRM, in particular the range. The confidence indicator of a target can be the value of the Z-score of the target. It can also be the estimated detection rate of the target calculated from the Gaussian error function (erf) with the Z-score of the target as input.

[0093] In one embodiment, the Z-score of a target is calculated as HIP post-processing in software form on the host device from the set of parameters PRM of the target, the crosstalk rate, the pulse size of the target, the effective SPAD count and some system timing parameters. In this embodiment, the confidence indicator is derived from the Z-score as HIP post-processing.

[0094] Embodiments and operation of the second processor AFF will now be described in more detail with reference to Figure 5 , Figure 6 .

[0095] As shown above and as shown in Figure 5 , the first circuit MD1 of the second processor AFF implemented in firmware FW here is configured to perform a matching operation 400 between the current set of detected targets Target#i, Target#i', Target#i" and the previous set of detected targets Target#j, Target#j', Target#j" stored in the memory MM for each region of the sensor device.

[0096] In the right part of Figure 5 , sd_j denotes sigma_d_j, sr_j denotes sigma_r_j, ar_j denotes ambient_rate_j, c_j denotes confidence_j.

[0097] As shown in Figure 6The second circuit MD2 of the second processor AFF implemented in firmware FW is configured to perform a temporal filtering operation 600 on at least one parameter of the set of current parameters of at least some of the current detection targets of the current set based on the result of the matching operation, the temporal filtering operation advantageously being weighted based at least on the confidence indicator, as illustrated schematically here.

[0098] In other words, to perform the temporal filtering operation, the second circuit MD2 implements an anti-flicker filter, and Figure 6 The double weighting operation performed by the anti-flicker filter MD2 of the second processor AFF is illustrated.

[0099] An example of target matching performed by the first circuit MD1 of the second processor AFF will now be described in more detail.

[0100] Matching the current set of targets with the set of targets in memory allows processing multiple targets without averaging all targets into a meaningless output. In general, the matching operation comprises matching the current target Target#i(T i ) with all past realizations (history) of targets stored in the memory MM:

[0101] where are previous targets Target#j stored in the memory MM at time instants t-1,..., t-m.

[0102] As Figure 5 illustrated, the matching operation 400 uses an affinity criterion or function AFCR. Figure 7 An embodiment of the affinity criterion requirement is illustrated, which comprises proximity in range - possibly considering a range sigma (standard deviation) (i.e. sigma_r_i), proximity in velocity - possibly considering a velocity sigma (standard deviation) (i.e. sigma_r_i), and possibly considering their respective confidence (confidence_i) values.

[0103] When one or more proximities exceed a selected one or more reasonable values, the affinity is not returned (an infinite affinity value). The person skilled in the art will be able to select these reasonable values. In an embodiment, the reasonable values are selected based on experiments in a controlled environment with multiple targets (e.g. a workbench scan in a "dark" room) to determine threshold values related to different target parameters and to distinguish these targets.

[0104] When the affinity function is based on range, a non-linear function can be used:

[0105] It corresponds to distance.

[0106] When p = 2 and the threshold Th is infinite, the distance is the square of the Euclidean distance.

[0107] Other embodiments are possible.

[0108] The threshold Th can be set based on pulse width, sigmas (standard deviation), or a combination thereof. The start and end of the pulse (in segments) detected by the pulse segmenter PS can be used to derive alternative overlap-based distances. Rate affinity can also be used to balance the proximity of range d and rate r via β.

[0109]

[0110] In the above formula, These are respectively related to the current target T i and previous goals Related range, These are respectively related to the current target T i and previous goals The associated ratio.

[0111] Distance matrix Aff based on affinity p,Th (t) Therefore, it can be used as the affinity criterion AFCR in the matching process:

[0112]

[0113] ,in:

[0114] I(t) is the value generated by the histogram pipeline at time t. Figure 5 The output target number J(t) is the number of valid targets stored in memory MM at time t. A suitable matching algorithm (Gale-Shapley algorithm) can be used to best match the current target with the targets in memory.

[0115] From the affinity matrix Aff p，Th In (t), the preference matrix PropPref(t) of the proposer and the preference matrix AccPref(t) of the acceptor are constructed, as follows: Figure 8 As shown. The "proposer" is the target output by the histogram pipeline at time t: {T1, ..., T2} I(t) The “receiver” is the valid target stored in memory MM at time t:

[0116] Construct the proposer's preference matrix PropPref(t), for example, to form T i The preference list, including by ascending pairs Sort them. Therefore, It is T i The first choice It was their last resort.

[0117] Construct the recipient's preference matrix AccPref(t), for example, to form The preference list, including by ascending pairs {T1, ..., T} I(t) Sort them. Therefore, yes The first choice This is its final choice. The matching algorithm provides the final set of matches from the proposer's preference matrix PropPref(t) and the acceptor's preference matrix AccPref(t).

[0118] In the case of the Gale-Shapley matching algorithm, the proposer advances along its corresponding row in PropPref(t). The acceptor checks its corresponding row in AccPref(t) to look for possible better matches in successive rounds that produce intermediate sets of matches, until the final set of matches is produced in the last round.

[0119] For targets that have already obtained final matches (some targets can remain single), the Gale-Shapley algorithm ensures that the obtained set of final matches is "stable" in the following sense:

[0120] No right and Where i≠j and k≠l, T i Preference arrive and Preference for T i To T j Or in other words,

[0121] and

[0122] An example of anti-flicker filtering performed by the second circuit MD2 of the second processor AFF will now be described in more detail.

[0123] Figure 9 The flowcharts illustrate three possible implementation scenarios. In the first scenario, at step 800, the current target matches the target in the memory MM. This corresponds to the nominal case of the anti-flicker filter MD2 of the second processor AFF.

[0124] In the second case, at step 801, the current target does not match any of the targets in the memory. This corresponds for example to an emerging target or to a glitch. At steps 803 and 804, if a slot is available, this emerging target can be added to the memory MM.

[0125] Otherwise, at step 805, a selective memory replacement of the stored targets by this emerging target can be performed based on the confidence indicators.

[0126] At step 802, a previous target in the memory does not match in the current set of detected targets. This corresponds to a lost target or to a virtual target. At step 806, this virtual target is added at the input of the filter to fill the gap for filtering.

[0127] In embodiments, the addition and replacement operations can be performed by the control device CTRLM.

[0128] In the case of a lost target (e.g. a virtual target), there are several possible reasons why a target in the memory is lost at some point. First, in order to achieve low latency, the target has actually disappeared from the scene. The filter MD2 of the second processor AFF advantageously reacts as quickly as possible and discards this target from the memory as soon as possible. Secondly, in order to achieve high stability, the target is weak and the pulse segmenter PS does not detect it from time to time. The anti-flickering filter MD2 advantageously keeps it. Thirdly, in order to achieve both objectives (i.e. low latency and stability), the past confidence of the target in the memory needs to be taken into account. More precisely, if the virtual target is very confident about its past realizations, the virtual target is added to the filter input with a very low confidence (even lower than the PS threshold - even negative) if it is very low (close to the PS level), close to the previous confidence level in the memory. An additional benefit is that the filtering process is simplified and does not require handling branches or special cases.

[0129] As Figure 10 illustrated, one embodiment includes a mirror Z-score (with or without offset). More precisely, the previous value Z-score of the previous target corresponding to the lost target or to the virtual target (as Figure 4 illustrated) is higher than the pulse segmenter threshold, the difference being equal to Δ.

[0130] Therefore, the Z-score given to the lost target (virtual target) is equal to the pulse segmenter threshold minus Δ. In this case, the virtual target is added to the filter input with the lost target Z-score and therefore with a very low confidence, since one is very confident about its past realizations (previous Z-score much higher than the PS threshold, step 900).

[0131] For example, the previous Z-score could be equal to 1000 (i.e., a very high confidence level), which results in a standard PS threshold of 6.1, a Δ of 993.9 (1000 - 6.1), and a lost target Z-score of -987.8 (6.1 - 993.9). Therefore, when added to the filter input, the virtual target exhibits a very low Z-score, which significantly impacts the filtered confidence level, causing the target to disappear rapidly.

[0132] Figure 11 Another embodiment of mirrored Z-score (with or without offset) is shown, where the Δ value is less than Figure 10 The Δ value. In this case, the virtual target is added to the filter input with the Z score of the lost target and therefore with a confidence level close to the previous confidence level in memory, because the confidence level is low (close to the PS threshold, step 1000).

[0133] For example, the previous Z score could be 6.5 (i.e., low confidence, close to the PS threshold), which results in a normal PS threshold of 6.1, Δ of 0.4 (6.5-6.1), and a Z score of 5.7 (6.1-0.4) for the lost target. In this case, added to the filter input, the virtual target exhibits a Z score close to the memory MM, thus smoothing the target output (i.e., fixing flicker).

[0134] like Figure 12 As shown in the diagram, the anti-flicker filter MD2 may include an IIR (Infinite Impulse Response) filter MD20 and an FIR (Finite Impulse Response) filter MD21. In an embodiment, the IIR filter MD20 is disabled in the event of a lost target.

[0135] Figure 13 The use of the confidence indicator c is shown. j The weighted filter MD2 is a weighted average of the weighted averages. The confidence level may include mirror confidence (for dummy targets). Filter MD2 includes an IIR filter MD20, a recursive filter, an FIR filter MD21, and a direct filter. The filtered confidence level... Given by the following formula:

[0136]

[0137] Where m is the number of filter coefficients, w k It's the weight.

[0138] A direct filter provides a direct time average, with its weights decreasing over time—for example, w1=4, w2=2, and w3=w4=1.

[0139] Figure 14The double-weighted filtering operation 1300 performed by filter MD2 using a double-weighted average on the target property: range is shown. The confidence can include a mirror confidence (for virtual targets).

[0140] Filter MD2 includes an IIR filter MD20, a recursive filter and a FIR filter MD21, a direct filter. The confidence direct average is here based on c j (t-k) and includes potential virtual target(s). The filtered range Given by the following equation, where only past real targets are considered:

[0141]

[0142] The direct filter provides a direct time average whose weights decrease over time - e.g. w1=4, w2=2 and w3=w4=1.

[0143] Embodiments of the disclosure notably allow to achieve the following results and advantages, i.e. a significant reduction of flickering at the output of the histogram pipeline, e.g. by about 90%, a reasonable level of latency, e.g. 2 frames latency in the dark (for appearance / disappearance), 2 frames latency at appearance at 2kLux, 4 frames latency at worst at disappearance, a tunable trade-off between latency and stability, a smoothing of multiple targets, a help for detection of false long tail targets, and no reduction of spatial reduction.

[0144] As shown above and as Figure 15 shown, according to an advantageous implementation, the first circuit 1031 of the first processor 103 and the second processor AFF can be integrated into the firmware FW.

[0145] According to the variant shown in Figure 16 , the second circuit 1032 is integrated into the FW, but the first circuit 1031 and the second processor AFF are integrated into the software SW, which allows backward compatibility since the histogram pipeline HIP is not modified.

[0146] Although the description has been described in detail, it should be understood that various changes, substitutions and alterations can be made hereto without departing from the spirit and scope of the disclosure as defined by the appended claims. In the various drawings, like elements are designated with like reference numerals. Furthermore, the scope of the disclosure is not intended to be limited to the particular embodiments described in the specification, because the disclosure of two processes, machines, manufacture, compositions of matter, means, methods, or steps can be performed in an manner differing from the order of steps described in the specification, and the specification should be construed to cover any and all novel processes, machines, manufacture, compositions of matter, means, methods, or steps performed in a manner differing from the order of steps described in the specification.

[0147] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense, and it is intended that the application be understood broadly to encompass any and all modifications, changes, combinations or equivalents that fall within the scope of the disclosure.

Claims

1. A method for filtering, comprising: detecting successive sets of targets from a first histogram signal at respective successive time instants, the first histogram signal being delivered by a sensor device; determining a histogram output for a current set of the successive sets of targets, the histogram output having, for each current set, a set of parameters stored in a memory, each set of parameters having a confidence indicator; performing a matching operation between the current set of the successive sets of targets and a previous set of the successive sets of targets stored in the memory; and based on a result of the matching operation, performing a filtering operation on at least one parameter of the set of parameters of the current set, the filtering operation being a doubly weighted filtering operation based on a time weight having a decreasing importance over time, and the confidence indicator of the current set and the confidence indicator of the previous set of the successive sets of targets being weighted.

2. The method of claim 1, wherein a set of parameters for the current set comprises a velocity parameter, an ambient velocity parameter, and a distance parameter.

3. The method of claim 2, wherein the matching operation comprises: using an affinity criterion having a condition based on the velocity parameter, the distance parameter, or the confidence indicator.

4. The method of claim 3, wherein the affinity criterion comprises a distance-based affinity matrix.

5. The method of claim 1, wherein at the end of the matching operation, the current set of the successive targets matches the previous set of the successive targets stored in the memory.

6. The method of claim 1, wherein at the end of the matching operation, the current set of the successive targets does not match the previous set of the successive targets stored in the memory.

7. The method of claim 6, wherein the current set of the successive targets is added to the memory based on availability of a slot in the memory.

8. The method of claim 1, wherein the method further comprises performing a selective memory replacement based on a confidence indicator for storing the current set of the successive targets.

9. The method of claim 1, wherein the previous set of the successive targets in the memory does not match the current set of the successive targets, and based thereon, a virtual target is added to an input of the filtering operation with a new confidence indicator.

10. The method of claim 9, wherein the new confidence indicator is very low in response to a high confidence of a corresponding previous target.

11. The method of claim 9, wherein the new confidence indicator is close to a confidence indicator of the previous set of the successive targets in response to the confidence indicator of the previous set of the successive targets being lower.

12. The method of claim 1, wherein the sensor device comprises a direct time-of-flight (ToF) device.

13. A system for filtering, comprising: a sensor device configured to transmit a first histogram signal to allow detection of successive sets of targets at respective successive time instants; a first processor configured to determine a histogram output for a current set of the successive sets of targets, the histogram output having, for each current set, a set of parameters stored in a memory, each set of parameters having a confidence indicator; and a second processor configured to: perform a matching operation between the current set of the successive sets of targets and a previous set of the successive sets of targets stored in the memory, and based on a result of the matching operation, perform a filtering operation on at least one parameter of the set of parameters of the current set, the filtering operation being a double weighted filtering operation based on a temporal weight having a decreasing importance over time, and the confidence indicator of the current set of the successive sets of targets and the confidence indicator of the previous set of the successive sets of targets being weighted, the confidence indicator of the current set being based on a target z-score in the first histogram signal.

14. The system of claim 13, wherein the set of parameters for the current set comprises a velocity parameter, an environmental velocity parameter, and a distance parameter.

15. The system of claim 14, wherein the matching operation comprises using an affinity criterion having a condition based on the velocity parameter, the distance parameter, or the confidence indicator.

16. The system of claim 15, wherein the affinity criterion comprises a distance-based affinity matrix.

17. The system of claim 13, wherein at the end of the matching operation, the current set of the successive targets matches the previous set of the successive targets stored in the memory.

18. The system of claim 13, wherein at the end of the matching operation, the current set of the successive targets does not match the previous set of the successive targets stored in the memory.

19. The system of claim 18, wherein the current set of the successive targets is added to the memory based on availability of a slot in the memory.

20. The system of claim 13, wherein the second processor is further configured to perform a selective memory replacement based on the confidence indicator for storing the current set of the successive targets.

21. The system of claim 13, wherein the previous set of the successive targets in the memory does not match the current set of the successive targets, and based thereon, a virtual target is added to an input of the filtering operation with a new confidence indicator.

22. The system of claim 21, wherein the new confidence indicator is very low in response to a high confidence in a corresponding previous target.

23. The system of claim 21, wherein the new confidence indicator is close to a confidence indicator of the previous set of the successive targets in response to the confidence indicator of the previous set of the successive targets being lower.

24. The system of claim 13, wherein the sensor device comprises a direct time-of-flight (ToF) device.

25. The system of claim 13, wherein the first processor and the second processor are implemented as firmware.

26. The system of claim 13, wherein the first processor comprises a first circuit configured to determine the confidence indicator and a second circuit configured to determine other parameters, the first circuit and the second processor are implemented as software and the second circuit is implemented as firmware.

27. A device for filtering, comprising: a sensor device configured to transmit a first histogram signal to allow detection of a set of successive targets at respective consecutive time instants; a non-transitory memory storage comprising instructions; and a first processor in communication with the non-transitory memory storage and the sensor device, wherein the instructions, when executed by the first processor, cause the first processor to determine a histogram output for a current set of the set of successive targets, the histogram output having, for each current set, a set of parameters stored in memory, each set of parameters having a confidence indicator; and a second processor in communication with the first processor, the non-transitory memory storage, and the sensor device, wherein the instructions, when executed by the second processor, cause the second processor to: perform a matching operation between the current set of the set of successive targets and a previous set of the set of successive targets stored in the memory, and perform a filtering operation on at least one parameter of the set of parameters of the current set based on a result of the matching operation, the filtering operation being a doubly weighted filtering operation based on a time weight having a decreasing importance over time and the confidence indicators of the current set and the previous set of the set of successive targets being weighted, the confidence indicator of the current set being based on a target z-score in the first histogram signal.

Images