Resolving distance measurement ambiguity using coded modulated phase image frames

Abstract

Distance ambiguity caused by indirect time of flight (ToF) measurements is resolved by using additional information from two or more coded modulation measurements. Indirect ToF measurements are performed for pixels of an image processor to obtain values ​​indicating a line of sight to an imaged object or scene. First and second coded modulation measurements are also performed using corresponding combinations of modulation codes and reference signals so that correlation peaks corresponding to these measurements overlap and cover corresponding first and second adjacent ranges of distance to the imaged object. First and second mask values ​​are determined based on correlation values ​​obtained from the coded modulation measurements, and the first and second mask values ​​are used to determine whether the value indicating the line of sight indicates an actual distance within a first distance range or within a second distance range.

Description

Technical Field

[0001] The present disclosure relates generally to image processing, and more particularly to processing image data using information generated from coded modulation time-of-flight measurements. Background Art

[0002] In optical sensing applications, depth measurements (i.e., measurements of the distance to various features of one or more objects in the field of view of an image sensor) can be performed as indirect time-of-flight (ToF) measurements, which are distance measurements determined using the speed of light and an image / pixel sensor, and which determine the distance to an imaged object or scene by determining the phase difference between the modulated light illuminating the object or scene and the reflected light received at the imaging sensor. The distance to the object of interest, which is a direct function of this phase, is typically calculated once per pixel, and once calculated, can be used for depth detection, gesture recognition, object detection, etc. These distances can be combined to create a depth map and / or a three-dimensional (3D) point cloud, for example, for 3D graphics rendering.

[0003] Many methods of indirect TOF measurement use so-called continuous wave (CW) ToF measurement, which typically requires multiple sequential exposures called subframes, raw frames, phase images or CW phase measurements. For each of these exposures, the light illuminating the object of interest is intensity modulated by a periodic waveform (such as a sine or pulse waveform), where each pixel of the image sensor measures the correlation of the light reflected from the object of interest with a reference signal that is a copy of the modulated waveform. For each exposure, the phase difference between the reference signal and the waveform of the modulated light changes. For example, one method requires four separate exposures (such as 0°, 90°, 180° and 270°). The measurement information from the four exposures is collected and can be used to determine a depth map.

[0004] A well-known problem with CW ToF measurements is that the correlation function for the modulating waveform and the corresponding reference signal is itself a periodic waveform. This creates ambiguity in the resulting CW phase measurements, since these phase measurements themselves do not provide any indication of which period of the periodic correlation waveform they are from. This phenomenon (often referred to as "phase wrapping") means that objects in the background of one or more objects of interest can reflect light that appears as an unwanted response in the image, especially if one or more of the background objects is highly reflective.

[0005] A conventional solution to this problem is to make an additional set of phase measurements using a reference signal with a different frequency, i.e., to perform an additional exposure. Since a properly chosen frequency for this second reference signal will result in a different blur distance, the measurements performed with the two different reference signal frequencies can be combined to resolve the blur. Of course, this approach increases the number of measurements required, and thus increases the data storage and processing power requirements of the imaging system.

[0006] Other methods for indirect ToF measurement include the use of modulation coding and may be referred to as coded ToF measurement or coded modulation ToF measurement. With some of these methods, the waveform used to modulate the intensity of the emitted light and the reference signal used to demodulate the reflected light received at the pixels of the image sensor are selected to have good autocorrelation characteristics, so that a correlation response with a significant amplitude only occurs when the offset range between the modulation signal and the reference signal is very narrow relative to the period of the waveform. At other offsets, the amplitude of the correlation response is very low, or zero. Modulation and reference signals with these characteristics can be formed by well-known sequences such as m-sequences, Barker codes, or other pseudo-random sequences.

[0007] Compared to conventional CW ToF measurements, coded modulation ToF measurements can be used to extend the ambiguity range. However, in order to generate enough reflected energy to be reliably detected at the imaging sensor, coded modulation ToF measurements typically use a repetitive modulation code so that the signal at the imaging sensor's pixel can be integrated over several cycles of the modulation waveform. Therefore, like CW ToF measurements, coded modulation ToF measurements produce a periodic correlated response even if the ambiguity range is extended, and therefore also suffer from the problem of phase wrapping and range ambiguity. Summary of the invention

[0008] Disclosed are time-of-flight (TOF) systems and techniques that address these needs, whereby range ambiguities caused by indirect ToF measurements (such as CW ToF measurements and other ToF measurements with periodically correlated responses) are resolved by using additional information from two or more coded modulation measurements that use a combination of appropriately selected modulation coding and reference signals. Using these techniques, range measurement ambiguities can be resolved without performing, for example, an additional set of CW phase measurements.

[0009] An example method for performing depth measurement using an image sensor according to the currently disclosed technology includes several steps performed for each pixel of at least one pixel. The method includes the following steps: performing indirect time-of-flight measurements using the pixels to obtain a value indicating the apparent distance to the imaged object or scene. The method further includes the following steps: performing a first coded modulation measurement for the pixel using a first combination of a modulation code and a reference signal selected to produce a first correlation response to obtain a first correlation value, the first correlation response having a peak extending over a first distance range to the imaged object. The first distance range can cover, for example, a distance range extending to a maximum distance that can be explicitly measured using a CW ToF measurement.

[0010] The method further comprises: performing a second coded modulation measurement for the pixel using a second combination of modulation code and reference signal selected to produce a second correlation response to obtain a second correlation value, the second correlation response having a peak extending over at least a second distance range to the imaged object. The second distance range is adjacent to the first distance range, and the correlation peak for the second combination of modulation code and reference signal overlaps with the correlation peak for the first code in an overlap region, the overlap region consisting of a portion of the first distance range and an adjacent portion of the second distance range. Thus, for example, with a correlation peak for the second combination of modulation code and reference signal covering the second range and overlapping a portion of the first range, the second distance range can extend from a maximum distance that can be unambiguously measured using CW phase measurement to twice that distance.

[0011] The method further includes determining a first mask value for the pixel by comparing the first correlation value to a first threshold, and determining a second mask value for the pixel by comparing the second correlation value to a second threshold. Finally, the method includes resolving distance ambiguity in the value indicating the viewing distance by determining whether the value indicating the viewing distance indicates an actual distance within a first distance range or within a second distance range based on the first mask value and the second mask value.

[0012] A corresponding image processing system is also disclosed. One such system includes: a sensor including a plurality of pixels, the plurality of pixels being configured to generate a corresponding plurality of pixel signal values ​​in response to received light, wherein each pixel is configured to obtain its corresponding pixel signal value by demodulating the received light using a reference signal. The system further includes: a reference signal generator configured to generate a reference signal and provide the reference signal to the plurality of pixels; and a control circuit configured to control the reference signal generator and the plurality of pixels to, for each of at least one of the pixels, perform an indirect time-of-flight measurement using the pixel to obtain a value indicating a range of view to an imaged object or scene, and to perform a first coded modulation measurement for the pixel using a first combination of a modulation code and a reference signal selected to produce a first correlation response to obtain a first correlation value, the first correlation response having a peak extending over a first range of distances to the imaged object. The control circuit is further configured to perform a second coded modulation measurement for the pixel using a second combination of a modulation code and a reference signal selected to produce a second correlation response to obtain a second correlation value, the second correlation response having a peak extending over at least a second range of distances to the imaged object, the second range of distances being adjacent to the first range of distances, wherein the correlation peak for the second modulation code overlaps the correlation peak for the first code in an overlap region, the overlap region being comprised of a portion of the first range of distances and an adjacent portion of the second range of distances. The control circuit is further configured to determine a first mask value for the pixel by comparing the first correlation value to a first threshold value, and to determine a second mask value for the pixel by comparing the second correlation value to a second threshold value, and further configured to resolve distance ambiguity in the value indicating the viewing distance by determining whether the value indicating the viewing distance corresponds to an actual distance within the first range of distances or within the second range of distances based on the first mask value and the second mask value.

[0013] The techniques and apparatus described herein can be used to efficiently obtain distance measurements, such as in depth maps, without ambiguity caused by phase wrapping. These techniques can also be used to selectively filter image data to limit the data to images of objects that fall within a predetermined distance range to the image sensor.

[0014] Variations and further advantages of the above-described techniques and apparatus are discussed in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a diagram illustrating the principle of time-of-flight measurement according to some embodiments described herein.

[0016] Figure 2 An example photonic mixing device (PMD) is shown.

[0017] Figure 3 is a diagram showing the principle of phase measurement according to the time-of-flight (TOF) technique.

[0018] Figure 4 The correlation function of the time-of-flight phase measurement is shown.

[0019] Figure 5 Phase unwrapping for phase measurement using two different modulation frequencies is shown.

[0020] Figure 6 A correlation function for an example coded modulation measurement is shown.

[0021] Figure 7 The correlation function of a pseudo-random binary sequence is shown.

[0022] Figure 8 The correlation response for a conventional 4-phase CW ToF phase measurement is shown superimposed with the correlation responses for two coded modulation measurements using two different combinations of modulation coding and reference signals.

[0023] Fig. 9 is a flow chart illustrating mapping of sight distances to first and second unambiguous distance ranges using an offset.

[0024] Fig.10 is a process flow diagram illustrating an example method according to some disclosed embodiments.

[0025] Fig.11 is a block diagram illustrating components of an example image processing system in accordance with some embodiments. DETAILED DESCRIPTION

[0026] The present invention will now be described with reference to the accompanying drawings, wherein like reference numerals are used to refer to like elements throughout, and the structures and devices shown therein are not necessarily drawn to scale. In the present disclosure, the terms "image" and "image sensor" are not limited to images or sensors involving visible light, but encompass the use of visible light and other electromagnetic radiation. Therefore, the term "light" as used herein is broadly intended and refers to visible light as well as infrared and ultraviolet radiation.

[0027] Figure 1The basic principle of continuous wave (CW) time-of-flight (TOF) phase measurement is shown, which is a well-known method for performing indirect ToF measurements. A light source 110 (such as a light emitting diode (LED) or a vertical cavity surface emitting laser (VCSEL)) is modulated with an electrical signal (e.g., a 300 MHz RF sine wave) so that the light source 110 transmits an amplitude modulated light signal to a target scene 120. The light signal travels at the speed of light c, reflects from one or more objects in the scene 120 and arrives back at the pixel array 135 in the TOF sensor 130, has a flight time to the target scene 120, and a phase shift is applied back on the light signal received at the pixel array 135 relative to the originally transmitted light signal

[0028] The modulation signal 137 used to modulate the emitted light, or a phase-shifted version thereof, is also provided as a reference signal to pixels in the pixel array 135 to correlate with the modulation signal superimposed on the reflected light signal - in effect, the reflected light signal is demodulated by each pixel in the pixel array 135.

[0029] Although the structure and design of the photosensitive pixels may vary, in some cases each pixel in pixel array 135 may be a photonic device, or PMD. Figure 2 The basic structure of an example PMD is shown, which includes readout diodes A and B and modulation gates A and B. A reference signal is differentially applied across the modulation gates A and B, thereby generating a potential gradient across the p-substrate while receiving incident light at the photogate / diode. A differential sensor signal is generated across the readout diodes A and B. The sensor signal from the pixel can be integrated for a period of time to determine phase measurement information.

[0030] The difference between the voltages at the Read-A and Read-B nodes of the PMD corresponds to the correlation between the modulated light signal detected by the photodiode structure in the device shown and a reference signal applied between nodes Mod-A and Mod-B of the device. Thus, as discussed in further detail below, the PMD (and other photosensitive pixel structures) demodulates the modulated light signal reflected from the target scene 120 to produce a pixel signal value (in this case, the difference between the voltages at Read-A and Read-B) that indicates the distance traveled by the reflected light signal.

[0031] Although the modulating signal can take any of a number of forms, it is easiest to see the principles behind this correlation / demodulation using a sinusoidal signal as the modulating signal. If the modulation amplitude is 'a' and the phase shift is The modulated signal g(t) and the received signal s(t) are given as:

[0032] m(t) = cos(ωt), and

[0033]

[0034] Then the correlation between the received signal and the reference signal is:

[0035]

[0036] It is the phase difference between the two signals It will be appreciated that for periodic modulated signals, such correlation may be performed over an extended period of time (eg, several periods of the modulated signal) to improve the signal-to-noise ratio of the resulting measurement.

[0037] The phase difference between the transmitted optical signal and the received reflection of that signal can be extracted by the N-phase shift technique, which is proportional to the distance traveled by the optical signal. This requires sampling the correlation function at N different points relative to the modulated signal g(t), for example by performing a correlation using N different phase shifts of a reference signal. At least two measurements are required to calculate the phase shift and thus determine the distance traveled. This is typically done using four different phase shifts (e.g., at 0, 90, 180, 270 degrees) as this simply eliminates systematic offsets in the correlation result. This is done in Figure 3 As can be seen in Figure 1, the figure shows how the correlations A0 and A1 at 0 and 90 degrees, respectively, correspond to a first phase vector having an "ideal" component corresponding to the actual difference in the travel of the optical signal and a systematic component reflecting the systematic errors in the measurement and readout. Similarly, the correlations A2 and A3 at 180 and 270 degrees, respectively, correspond to a second phase vector pointing in the opposite direction, having exactly opposite "ideal" components and the same systematic components. In the figure, the ideal component is indicated by the vector extending from the origin to the circle, while the systematic error component is indicated by the smaller vector. The actual phase can then be calculated as follows

[0038]

[0039] From this phase, the viewing distance or "depth" to the target scene 120 can be calculated as follows:

[0040]

[0041] Among them, f mod is the frequency of the modulating signal.

[0042] It should be immediately recognized that this distance calculation has an ambiguous result, making it impossible to accurately determine which of several possible distances may have produced the calculated phase from a single distance calculation. For this reason, this distance is referred to herein as the "line-of-sight distance." As the imaged object moves farther and farther from the light source, the measured phase increases from 0° to 360° and then repeats after a certain distance, such as Figure 4 In practice, there is an aliasing effect that may cause objects farther than this certain distance to be folded back into the first region (see Figure 2 ). In other words, because the correlated response has a strong periodic component, several different values ​​of D can produce the same phase measurement. This phenomenon is sometimes called "phase wrapping."

[0043] The distance from the optical source where this phase wrapping first begins to appear is called the blur distance, which is defined as:

[0044]

[0045] The ambiguity distance is half the free space wavelength of the modulated signal and is therefore shorter for higher modulation frequencies. Thus, for example, the ambiguity distance for an 80 MHz modulated signal is approximately 1.9 meters.

[0046] The importance of this phase wrapping can be seen in Figure 4 As can be seen in the figure, which shows an example of the phase measurement results varying as a function of distance for a CW TOF phase measurement. It can be seen from the figure that although the amplitude of the correlation decays with increasing distance, without other information it is impossible to tell the difference between the signal reflected from an object at distance x and the signal reflected from an object at distance x+d. amb In other words, the blur distance d amb is the width of the region in which each correlation value is unique - extending beyond this region results in duplicate correlation values.

[0047] Various techniques for resolving this ambiguity are known. One approach is to incorporate the amplitude information obtained from the reflected light signal into the distance determination. Another approach is to repeat the measurement using different modulation frequencies, with different modulation frequencies resulting in different ambiguous distances. The results of the second measurement can be combined with those of the first measurement to resolve the ambiguity, since these measurements should provide consistent results for only a single distance. This can be obtained from Figure 5As can be seen in the figure, two sets of phase measurements with different modulation frequencies are shown. In this example, one set of measurements uses a modulation frequency that produces an ambiguous distance of 3.75 meters, while the other set uses a modulation frequency that produces an ambiguous distance of 5.0 meters. The imaged object at 9 meters is farther than any of these ambiguous distances. As shown in the figure, the first set of measurements indicates a line of sight of 1.5 meters. If only the first set of measurements is considered, it is impossible to determine whether the imaged object is actually at 1.5 meters, 5.25 meters, 9.0 meters, or 12.75 meters. Similarly, if only the second set of measurements is considered, it is impossible to determine whether the imaged object is at 4.0 meters (the line of sight indicated by the second set of measurements), 9.0 meters, or 13.0 meters. However, the first possible distance shared by these two measurements is 9.0 meters-therefore, the possibilities of 1.5 meters, 4.0 meters, 5.25 meters, and 13.0 meters can be ruled out.

[0048] Thus, it can be seen that using two sets of phase measurements with different modulation frequencies can increase the blur distance to far more than is applicable when using either set of phase measurements alone. However, this approach increases power consumption by a factor of 1 compared to a single set of exposures, introduces motion artifacts due to the additional time required to complete the measurement, and requires significantly more computation.

[0049] Sensing distance using coded modulation is another method that can be used to reduce the distance ambiguity problem by extending the ambiguity distance. With modulation coding, the modulation waveform applied to the emitted light and the pixel reference signal (the pixel reference signal is used by the TOF pixel to measure the correlation with the received signal reflected from the object of interest) is adjusted in such a way that the autocorrelation function cuts off at a certain distance. Figure 6 An example of this is depicted in . In this figure, the bold line 620 indicates the correlation measurement at the ToF pixel as a function of distance, for an example coded modulation measurement. It can be seen from the figure that the function has a single peak over the distance range shown, and only produces a significant amplitude (i.e., exceeding the threshold 610) for a limited range of distances over this total range. For example, this result can be achieved using a modulation signal (and corresponding reference signal) that amplitude modulates the optical signal using an m-sequence, but other coded modulation signals are known and have been discussed. Note that although Figure 6 The correlation response is shown to have a positive value for the correlation at the peak, but some implementations and / or combinations of modulation signals and reference signals may produce a correlation response with a negative peak. Figure 2 When the PMD is shown, the peak value can be arbitrarily selected to be negative or positive simply by choosing to subtract Read-A from Read-B or Read-B from Read-A.

[0050] Using only coded modulation measurements to calculate the range image has some disadvantages. These measurements may require significantly more energy per measurement compared to CW ToF measurements and may result in poorer depth resolution. If the same signal strength as with regular continuous wave ToF is required, the exposure time needs to be increased. However, this may not be appropriate in some cases due to eye safety regulations.

[0051] The advantage of coded modulation is that it has a correlation peak that extends over a limited range of distances while providing very low correlation values ​​over extended distances. Figure 7 This can be clearly seen in the figure, which shows the ideal correlation response of a pseudo-random binary sequence with a bit (or "chip") length of Tc and a total length of L bits. As shown, the peak has a maximum amplitude proportional to L and is strictly limited to a range of width 2*Tc. Between peaks, the magnitude of the correlation value is proportional to 1 / L.

[0052] exist Figure 7 In , the correlation response has peaks at distances 0 and L*Tc. By cyclically shifting the reference signal relative to the modulation waveform, the peaks can be arbitrarily positioned at any point between these distances. Note that the correlation response is periodic, but because the peaks are widely spaced compared to, for example, the periodicity of the correlation response of conventional CW ToF flight measurements, the ambiguity distance can be quite long. Thus, for example, if a Barker-11 code is used for coded modulation measurements, the ambiguity distance is 20 meters at a bit rate of 80 MHz. Note that the Barker code is an example of a pseudo-random sequence that can be used for coded modulation measurements. The same Barker code or a shifted version of it used to modulate the emitted light can be used for the reference signal applied to the pixel to demodulate the light reflected from the target or scene of interest. Other sequences, such as m-sequences, and other combinations of modulation signals and reference signals may also be used.

[0053] Although coded modulation measurements can have long ambiguity distances compared to other indirect ToF measurements (including measurements based on CW ToF phase measurements as described above), the performance of coded modulation measurements may not be as good as the performance of measurements using conventional CW ToF phase methods, especially in terms of the transmitted optical power. However, coded modulation measurements can be advantageously used in combination with CW ToF phase measurements or other indirect ToF measurements to improve the ambiguity distances that can typically be obtained from CW ToF phase measurements or other indirect ToF measurements without (for example) using two different sets of CW phase measurements with different modulation frequencies. Detailed techniques according to this general approach are described below with reference to CW ToF measurements. However, it should be understood that these techniques (by virtue of combining coded modulation measurements with indirect ToF measurements to resolve distance ambiguities) can be applied to indirect ToF measurements other than CW ToF measurements.

[0054] Figure 8 The correlation response for a conventional 4-phase CW ToF phase measurement is shown superimposed with the correlation responses for two coded modulation measurements using two different combinations of modulation coding and reference signals. In this example, the correlation response for the coded modulation measurement has negative peaks and is indicated in the figure as corresponding to "Code 1" and "Code 2".

[0055] These first and second codes, or more precisely, the first and second combinations of the modulation waveform and the reference signal, are selected to provide correlation peaks covering different but overlapping distance ranges. It should be understood that in various embodiments, these may use different codes, for example, different pseudo-random sequences, or cyclically shifted versions of the same code.

[0056] As shown, "Code 1" produces a correlation peak extending from approximately 2.0 meters to 4.0 meters. As shown, the useful portion of this correlation peak can be limited by filter threshold 810. On the other hand, "Code 2" has a correlation peak extending from approximately 3.3 meters to approximately 5.6 meters. The useful portion of this second correlation peak can also be limited by filter threshold 810; in practice, different thresholds can be used.

[0057] In the example shown, the CW ToF measurement produces a periodic correlation response that produces an ambiguous distance of approximately 1.875 meters (corresponding to a modulation frequency of 80 MHz). Therefore, it is impossible to distinguish between a distance between 1.875 and 3.75 meters and a distance between 0 and 1.875 meters or a distance between 3.75 meters and 5.625 meters. Each of these ranges can be considered a different "clear range". The figure shows portions of these ranges, labeled "clear range 1" and "clear range 2", using a dividing line indicating a flip point of the phase measurement at approximately 3.75 meters. Code 1 and Code 2 are selected so that each of their corresponding correlation peaks extends over most, if not all, of a corresponding one of these clear ranges. As explained in further detail below, these coded modulation correlation responses can be used to determine whether the results of the CW ToF measurement indicate the presence of an object within a first clear range or a second clear range.

[0058] An example of how to perform this blur removal is as follows. First, an indirect ToF measurement is performed for at least one pixel (e.g., for each pixel of an image sensor) to obtain a value indicating the line of sight to the imaged object or scene for the pixel. For example, as described above, a conventional four-phase depth measurement can be performed. Although these measurements can be used to calculate the line of sight (or depth map) to the imaged object or scene, the line of sight is susceptible to blur. For example, it may not be possible to know from these measurements alone whether the imaged object is 2.4 meters or 4.275 meters from the imaging device, because the phase measurements for both will be similar.

[0059] Next, two coded modulation measurements are performed using two different combinations of modulation signal and reference signal, whose corresponding correlation peaks cover two different but overlapping ranges, such as Figure 8 Note that although these measurements are described herein as being performed after the indirect ToF measurement that produces a value indicative of line of sight, these measurements may be performed in any order.

[0060] As described above, depth or line of sight can be calculated from conventional four-phase measurements. Note that line of sight need not necessarily be calculated - some other value may indicate the line of sight. For example, given a certain blur distance, the phase value (ranging from zero to 360 degrees) obtained by combining the results of multiple CW phase measurements indicates the line of sight, where 90 degrees indicates a line of sight of one quarter of the blur distance, 180 degrees indicates a line of sight of half the blur distance, and so on.

[0061] For phase unwrapping, i.e., to resolve ambiguity in the calculated line-of-sight or some other value indicative of line-of-sight, a filter mask is applied to each of the coded modulation measurements (alternatively referred to as a "coded modulation frame" when a pixel array is used). This can be done by simply comparing the correlation value obtained from each measurement to a threshold. For a vector with Figure 8 Coded modulation measurement of a negative-going peak is shown, comparing the correlation value to a threshold with a negative threshold - i.e., values ​​below the threshold are set to 1, and values ​​above the threshold are set to 0:

[0062]

[0063] This is performed using the same or different thresholds for the two coded modulation measurements. Thus, for each pixel, the first mask value F1 indicates whether the correlation with code 1 falls within the peak of the correlation response for code 1, and the second mask value F2 indicates whether the correlation with code 2 falls within the peak of the correlation response for code 2. These two mask values ​​can be used directly to determine whether the value of the line-of-sight indication obtained by the indirect ToF measurement indicates that Figure 8Notice that Region 1 and Region 2 extend across part (but not all) of Explicit Range 1 and the adjacent Explicit Range 2, respectively; Region 2 lies between Region 1 and Region 3 and includes part of Explicit Range 1 and Explicit Range 2. The boundaries of Region 2 are defined by the points where the correlation peaks for Code 1 and Code 2 cross the filter threshold.

[0064] It is possible to determine whether the value indicating line of sight should be mapped to region 1 or region 2 using only the mask values ​​discussed above. In short, if F1=1, and F2=0, then the line of sight indicated by the indirect ToF measurement must fall into region 1. Similarly, if F1=0 and F2=1, then the line of sight indicated by the CW phase measurement must fall into region 3. If either of these two conditions is met, the value indicating line of sight will be ambiguous, because the former case maps the line of sight to clear range 1, while the latter maps the line of sight to clear range 2. This can be expressed as:

[0065]

[0066] That is to say, when the first mask value indicates that the first correlation value is below the first threshold and the second mask value indicates that the second correlation value is above the second threshold, the line of sight obtained from the indirect ToF measurement is mapped to the first distance range (i.e., clear area 1), and when the first mask value indicates that the first correlation value is above the first threshold and the second mask value indicates that the second correlation value is below the second threshold, the line of sight is mapped to the second distance range (i.e., clear area 2).

[0067] Again, the term "line-of-sight" is used here to indicate that the distance calculated by indirect ToF measurement (e.g., using the CW phase measurement described above) is ambiguous because it is affected by the phase wrapping distance phenomenon described above. The line-of-sight will always indicate a distance within the first unambiguous range - therefore, it may or may not be the same as the actual distance to the imaged object or scene.

[0068] It should also be noted that the formula given above assumes a negative correlation peak. If the opposite is true, i.e. if the correlation peak (and the corresponding threshold) is positive, then the value indicating the line of sight obtained from the indirect ToF measurement is mapped to the first distance range (i.e., clear zone 1) when the first mask value indicates that the first correlation value is above the first threshold and the second mask value indicates that the second correlation value is below the second threshold, and the value indicating the line of sight is mapped to the second distance range (i.e., clear zone 2) when the first mask value indicates that the first correlation value is below the first threshold and the second mask value indicates that the second correlation value is above the second threshold.

[0069] Determining how the CW phase measurement should be mapped to the first and second unambiguous ranges when the results of the coded modulation measurement fall into Region 2 requires additional evaluation. Region 2 defines the range in which the magnitude of the correlation values ​​from two coded modulation measurements exceeds the magnitude of their respective thresholds; therefore, any mapping just described above does not apply. Two different approaches are possible, which can also be combined.

[0070] The first method is based on the magnitude of the coded modulation measurements. Assuming that the magnitude of the correlation values ​​of both coded modulation measurements exceeds the magnitude of their respective thresholds, if the correlation value from the first coded modulation measurement is lower than the correlation value from the second coded modulation measurement, the value indicating the line of sight is mapped to the first clear range. Otherwise, that is, if the correlation value from the first coded modulation measurement is greater than the correlation value from the second coded modulation measurement, the value indicating the line of sight is mapped to the second clear range. This can be expressed as:

[0071]

[0072] Again, it should be noted that this formula assumes a negative value of the correlation peak. If the correlation peak is instead a positive value, then the inequality in the previous formula should be reversed. More specifically, it should be understood that, assuming that the magnitude of the correlation values ​​of both coded modulation measurements exceeds the magnitude of their respective thresholds, then if the magnitude of the correlation value from the first coded modulation measurement is greater than the magnitude of the correlation value from the second coded modulation measurement, then the value indicating the line of sight is mapped to the first well-defined range. Otherwise, that is, if the magnitude of the correlation value from the first coded modulation measurement is greater than the magnitude of the correlation value from the second coded modulation measurement, then the value indicating the line of sight is mapped to the second well-defined range.

[0073] In other words, according to the first method, determining whether the value indicating the line of sight calculated from the indirect ToF measurement indicates an actual distance within the first distance range or within the second distance range includes, in the case where the first and second mask values ​​indicate that both the first and second correlation values ​​have amplitudes above the respective amplitudes of the first and second thresholds: assigning the value indicating the line of sight to the first range of actual distances in the case where the amplitude of the first correlation value is further greater than the amplitude of the second correlation value; otherwise, assigning the value indicating the line of sight to the second range of actual distances in the case where the amplitude of the second correlation value is further greater than the amplitude of the first correlation value. Note that if the correlation values ​​are equal (and the amplitudes of both values ​​are greater than the amplitudes of their corresponding thresholds), the value indicating the line of sight can be mapped to the first or second range of explicit ranges, since this case indicates that the actual distance is at the point where the two ranges intersect.

[0074] Another way to handle the mapping of line-of-sight to unambiguous ranges when the coded modulation measurements fall within region 2 is to use the phase values ​​obtained from the CW ToF measurements. Because region 2 covers a portion of the rightmost end of the first unambiguous range, the actual distance obtained from the CW ToF measurements falls within region 2 and the phase in the first unambiguous range should be between 180° and 360°. Conversely, because region 2 covers a portion of the leftmost end of the second unambiguous range, the actual distance obtained from the CW ToF measurements falls within region 2 and the phase in the second unambiguous range should be between 0° and 180°. As described above, this method can be combined with the above method according to the following:

[0075]

[0076] Note that if the phase measurement has a value of 180° (and the magnitudes of both coded modulation measurements are greater than the magnitudes of their corresponding thresholds), then the value indicating line of sight can be mapped to the first or second unambiguous range, since this situation indicates that the measured distance is at the point where the two ranges intersect.

[0077] Fig. 9 is a flow chart showing how the mapping from regions 1, 2, and 3 is applied to clear ranges 1 and 2. If the above test indicates that the line of sight indicated by the indirect ToF measurement falls in region 1, the distance is mapped to the first clear range. This means that the line of sight calculated from the indirect ToF measurement can be used directly, i.e., the offset is zero, as shown in the figure. However, if the test indicates that the line of sight falls in region 2, other tests are required. In the example given in the figure, the example test is a phase-based test - if the phase is between 180 degrees and 360 degrees (i.e., less than zero), the line of sight is mapped to the first clear range, again indicating that the calculated line of sight with an offset of zero can be used as the actual distance.

[0078] However, if the phase-based test indicates that the line of sight falls in the other half of region 2, or if the above test indicates that the line of sight obtained from the CW phase measurement falls in region 3, then the line of sight is mapped to the second unambiguous region. In this case, an offset equal to the ambiguous distance of the indirect ToF measurement can be applied to the line of sight calculated from the indirect ToF measurement to effectively "unfold" the ambiguous measurement.

[0079] If only two coded modulation measurements are used (as described above and as Figure 8 and Fig. 9 ), then CW phase measurements that map to actual distances farther than region 3 are filtered. This prevents these measurements from producing erroneous depths in the depth map, for example. If only normal CW phase measurements were used, these measurements (and any measurements for objects that fall within the second unambiguous range) would have been collapsed into the first unambiguous range.

[0080] It will be appreciated that additional coded measurements may also be used to extend the effective range of ambiguity removal. Thus, for example, a third coded modulation measurement may be performed having an associated response spanning a third unambiguous range adjacent to the second unambiguous range, thereby allowing the mapping of the view range to any of the first, second and third unambiguous regions using a direct extension of the above techniques.

[0081] More specifically, if the above tests indicate that the line of sight should be mapped to the first unambiguous region, no further testing is required. If these tests indicate that the line of sight is neither within the first unambiguous range nor within the second unambiguous range, additional tests similar to those described above may be performed for the second and third coded modulation measurements to determine whether the line of sight should be mapped to a third unambiguous range adjacent to the second unambiguous range. If the line of sight should be mapped to the third unambiguous range, an offset equal to twice the ambiguity distance of the CW phase measurement should be added to the line of sight.

[0082] As shown, if the test indicates that the sight distance does not fall within any of Region 1, Region 2, or Region 3, then that distance measurement may be filtered out (eg, by giving it a value of zero).

[0083] In view of the above description and detailed examples, it will be understood that Fig.10 It is a process flow chart showing an example method for performing depth measurement using an image sensor. As shown, the method is applicable to a single pixel, such as a PMD similar to the above-mentioned PMD. However, it will be appreciated that the same technology can be applied to each pixel in several (or multiple) pixels in an image sensor, and / or to different types of pixels.

[0084] As shown in block 1010, the method includes the steps of performing an indirect ToF measurement for a pixel to obtain a value indicative of the line of sight (which may be the line of sight itself, or some other value indicative of the line of sight). As described above, in some embodiments, this may include a CW phase measurement, and may more specifically include performing four continuous wave phase measurements for a pixel using a reference signal for the four continuous wave phase measurements, the continuous wave phase measurements having different phase differences from one another (relative to the signal modulating light emitted toward the imaged object or scene).

[0085] As shown in boxes 1020 and 1030, the method further includes: performing a first coded modulation measurement for the first pixel using a first combination of modulation code and reference signal selected to produce a first correlation response to obtain a first correlation value, the first correlation response having a peak extending over a first distance range to the imaged object; and performing a second coded modulation measurement for the first pixel using a second combination of modulation code and reference signal selected to produce a second correlation response to obtain a second correlation value, the second correlation response having a peak extending over at least a second distance range to the imaged object. The first and second combinations of modulation code and reference signal are selected so that the second distance range is adjacent to the first distance range, so that the correlation peak for the second modulation code overlaps with the correlation peak for the first code in an overlapping area, and the overlapping area is composed of a portion of the first distance range and an adjacent portion of the second distance range. These combinations of modulation codes can be based on pseudo-random sequences, such as Barker codes or m-sequences. In some embodiments, they can be based on different codes or sequences, or in other embodiments, they can be based on cyclically shifted versions of the same code or sequence. Note that although Fig.10 It is recommended that the first and second coded modulation measurements shown at blocks 1020 and 1030 be performed after the indirect ToF measurement shown at block 1030, but these measurements may be performed in any order. In fact, in some embodiments, coded modulation measurements may be interleaved with CW phase measurements used to obtain indirect ToF measurements.

[0086] As shown in block 1040, Fig.10 The method further includes: determining a first mask value for the first pixel by comparing the first correlation value with a first threshold, and determining a second mask value for the first pixel by comparing the second correlation value with a second threshold. In various embodiments, the first threshold and the second threshold may be the same or different.

[0087] As indicated at block 1050 , the method further includes the step of resolving distance ambiguity in the value indicating the viewing range by determining whether the value indicating the viewing range corresponds to an actual distance within the first distance range or within the second distance range based on the first mask value and the second mask value.

[0088] The details of determining whether the line of sight is mapped to an actual distance within the first distance range or the second distance range may vary depending on whether the correlation responses corresponding to the first and second coded modulation measurements have positive or negative peaks (note that either coded modulation measurement can be obtained using the PMD as shown, depending on which direction is used to obtain the differential measurement from readout diodes A and B). When the peaks of the first and second correlation responses have negative maximum values, determining whether the value indicating the line of sight corresponds to an actual distance within the first distance range or the second distance range may include: mapping the value indicating the line of sight to the first distance range when the first mask value indicates that the first correlation value is below a first threshold and the second mask value indicates that the second correlation value is above a second threshold, and mapping the value indicating the line of sight to the second distance range when the first mask value indicates that the first correlation value is above the first threshold and the second mask value indicates that the second correlation value is below the second threshold. On the other hand, when the peaks of the first and second correlation responses have negative maximum values, determining whether the value indicating the line of sight corresponds to an actual distance within the first distance range or within the second distance range may include: mapping the value indicating the line of sight to the first distance range when the first mask value indicates that the first correlation value is above a first threshold and the second mask value indicates that the second correlation value is below a second threshold, and mapping the value indicating the line of sight to the second distance range when the first mask value indicates that the first correlation value is below the first threshold and the second mask value indicates that the second correlation value is above the second threshold.

[0089] In either case, determining whether the value indicating the line of sight corresponds to an actual distance within the first distance range or within the second distance range may include, where the first and second mask values ​​indicate that both the first and second correlation values ​​have amplitudes above respective amplitudes of first and second thresholds: where the amplitude of the first correlation value is further greater than the amplitude of the second correlation value, assigning the value indicating the line of sight to the first range of actual distances; otherwise, where the amplitude of the second correlation value is further greater than the amplitude of the first correlation value, assigning the value indicating the line of sight to the second range of actual distances.

[0090] Alternatively, determining whether the value indicating the line-of-sight corresponds to an actual distance within the first range of distances or within the second range of distances may include, where the first and second mask values ​​indicate that both the first and second correlation values ​​have amplitudes above respective amplitudes of first and second thresholds: in a further case where the phase value obtained from the one or more continuous wave phase measurements is between 180 degrees and 360 degrees, assigning the value indicating the line-of-sight to the first range of actual distances, otherwise assigning the value indicating the distance to the second range of actual distances. In some embodiments, the phase information may be combined with amplitude information from the first and second correlation values ​​to determine how to map the line-of-sight to the first range or the second range of actual distances where both the first and second correlation values ​​have amplitudes above respective amplitudes of the first and second thresholds.

[0091] As discussed in the detailed examples above, in some embodiments, performing an indirect time-of-flight measurement for a pixel may include: for each of one or more continuous wave phase measurements, using a reference signal having a different phase difference relative to the waveform of the emitted light modulating the illumination of the object of interest, and for each of the one or more continuous wave phase measurements, measuring the correlation between the light received at the pixel and the reference signal. In some embodiments, performing an indirect time-of-flight measurement includes: performing four continuous wave phase measurements for the pixel, the reference signals for the four continuous wave phase measurements having a phase difference (relative to the corresponding modulation signal) that differs from each other by, for example, 90 degrees. Variations of this approach may be used, for example, using a non-continuous modulation signal for each of the phase measurements.

[0092] Likewise, in some embodiments, performing a first or second coded modulation measurement (or both) for a pixel may include measuring a correlation between light received at the pixel and the first reference signal using a first reference signal having a cross-correlation function relative to a waveform of the emitted light modulating the object of interest, the cross-correlation function having only a single peak with an amplitude exceeding a first threshold over a distance range substantially greater than an ambiguous distance for indirect ToF measurements. In some embodiments, the reference signal and / or the modulation signal may be or be based on a Barker code or an m-sequence, although other encodings / waveforms may be used. In some embodiments, the second coded modulation measurement uses a reference signal that is a cyclically rotated version of the reference signal used for the first coded modulation measurement.

[0093] Fig.10Other variations of the illustrated techniques are also possible. For example, in some embodiments, a threshold for obtaining a mask value may be set dynamically. For example, in some embodiments, one or both of the first threshold and the second threshold may be based on a reflectance parameter or an intensity parameter of the imaged object or scene. The parameter may be derived from a previous measurement made using a pixel or group of pixels of interest, and / or may be derived from a measurement performed to obtain an indirect ToF measurement.

[0094] In view of the detailed examples provided above, it will be appreciated that the presently disclosed techniques can be applied to several image processing applications to obtain one or more advantages. For example, for applications requiring close-range time-of-flight sensing, the techniques disclosed herein can be used to remove depth measurements that are not within the intended measurement range. For example, these techniques can be employed in front-facing smartphone cameras or gesture recognition systems.

[0095] Other applications include privacy imaging. The techniques employed are similar to those used to remove range blur, but the resulting depth image / 3D point cloud is used to remove information from a color camera stream, where the color camera is mounted next to the ToF sensor. For example, this approach can be used in a video chat application for a smartphone, where the ToF sensor is mounted next to the color camera facing forward. The background can then be removed using a mask generated according to the presently disclosed techniques to protect the user's privacy by not showing details of the user's surroundings.

[0096] Of course, the techniques described can also be used to eliminate erroneous measurements caused by phase wrapping discussed above. This is particularly useful for medium to long range applications such as 3D scanning. By using the mask value obtained from the coded modulation measurement as a guide, it can be determined to which period the sensed depth value belongs. In some embodiments, the shape and / or position of the peak of one or both of the coded modulation measurements can be adapted to the scene in order to provide optimal phase unwrapping performance.

[0097] Other applications of the presently disclosed technology are also possible.

[0098] Fig.11 Correspondingly, an example image processing system 1100 is shown in accordance with several embodiments of the presently disclosed apparatus and systems. The system 1100 can be used to detect objects (eg, objects shown in target scene 1102), and to determine distances to detected objects.

[0099] The illustrated system 1100 includes a light source 1124 configured to amplitude modulate a light beam and emit amplitude modulated light toward a scene 1102. The amplitude modulation can be based on a reference signal generated by a reference signal generator 1108. For continuous wave time of flight measurements or other indirect ToF measurements, the reference signal can be a radio frequency (RF) signal, for example in the MHz range, but other modulation frequencies can be used. For coded modulation measurements, the reference signal can be a coded sequence, such as an m-sequence. The emitted light can include light with a varying range of wavelengths, such as sunlight and infrared. The emitted light is reflected from one or more objects in the scene and then returned to the sensor 1104.

[0100] Fig.11 The image processing system 1100 shown in FIG. 1 further includes a sensor 1104 including a plurality of pixels configured to generate a corresponding plurality of pixel signal values ​​in response to received light 1114, wherein each pixel is configured to obtain its corresponding pixel signal value by demodulating the received light using a reference signal 1122. Fig.11 As shown, received light 1102 may be reflected from target scene 1102. As described above, while several suitable pixel configurations are possible, one suitable pixel design is the PMD described above.

[0101] The number of pixels, rows, and columns may vary from one embodiment to another and may be selected based on factors including desired resolution, intensity, etc. In one example, these sensor characteristics are selected based on the object to be detected and the expected distance to the object. Thus, for example, the pixel resolution of the pixels in the sensor 1104 may vary from one embodiment to another. Small objects require higher resolution for detection. For example, finger detection requires a resolution of <5mm per pixel within a distance or range of approximately 0.5 meters. Medium-sized objects (such as hand detection) require a resolution of <20mm per pixel within a range of approximately 1.5 meters. Large-sized objects (such as the human body) require a resolution of <60mm per pixel within a range of approximately 2.5 meters. It should be understood that the above examples are provided for illustrative purposes only, and variations may occur including other objects, resolutions, and distances for detection. Some examples of suitable resolutions include VGA-640x400 pixels, CIF-352x288 pixels, QQ-VGA-160x120 pixels, etc.

[0102] The image processing system 1100 further includes a reference signal generator 1108, which can be configured, for example, to generate a reference signal 1122 having a selectable phase relative to the phase of the modulation signal applied to the light transmitted toward the target scene 1102, and provide the reference signal 1122 to a plurality of pixels in the sensor 1104. The reference signal generator 1108 can be further configured to generate a coded modulation reference signal, which can take the form of, for example, an m-sequence or a Barker code, and provide the reference signal 1122 to a plurality of pixels in the sensor 1104 for coded modulation measurement. The image processing system 1100 also further includes an analog-to-digital converter (ADC) circuit 1106, which can include one or more ADCs operably coupled to a plurality of pixels in the sensor 1104.

[0103] The image processing system 1100 shown further includes a control circuit 1112, which may include, for example, a processor, a controller, etc., and / or other digital logic. As shown, the control circuit 1112 can control the operation of the reference signal generator 1108, the sensor 1104, and the light source 1124. The control circuit 1112 is further configured to control the generation and use of outputs (such as the depth map 1110, the 3D point cloud, etc.). In some embodiments, the control circuit 1112 is configured to cause the image processing system 1100 to perform operations similar to those described above in conjunction with Fig.10 Any of those methods described.

[0104] Thus, for example, in some embodiments, the control circuit 1112 can be configured to control the reference signal generator 1108 and a plurality of pixels in the sensor 1104 to perform an indirect ToF measurement using the pixel to obtain a value indicating a line of sight to an imaged object or scene, and to perform first and second coded modulation measurements for the pixel using respective first and second combinations of modulation coding and reference signals to obtain first and second correlation values, respectively. The first and second combinations of modulation coding and reference signals can be selected to produce a first correlation response having a peak extending over a first range of distances to the imaged object and a second correlation response having a peak extending over a second range of distances to the imaged object, wherein the first and second ranges are adjacent to each other and the first and second correlation peaks overlap in an overlap region consisting of a portion of the first range and an adjacent portion of the second range. The control circuit 1112 in these embodiments can be further configured to determine a first mask value for the pixel by comparing the first correlation value to a first threshold value, and to determine a second mask value for the pixel by comparing the second correlation value to a second threshold value. Finally, the control circuit 1112 can be configured to resolve the distance ambiguity in the value indicating the viewing distance by determining whether the value indicating the viewing distance corresponds to an actual distance within the first distance range or within the second distance range based on the first and second mask values. Fig.10 Any variation of the method discussed can be applied to Fig.11 device.

[0105] In view of the detailed discussion above, it should be understood that the claimed subject matter can be implemented as a method, device, or article of manufacture using standard programming and / or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. The term "article of manufacture" as used herein is intended to encompass a computer program accessible from any computer-readable device, carrier, or medium. Of course, those skilled in the art will recognize that many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter.

[0106] In particular, with respect to the various functions performed by the above-described components or structures (assemblies, devices, circuits, systems, etc.), unless otherwise noted, the terms describing these components (including references to "means") are intended to correspond to any component or structure that performs the specified function of the described (e.g., functionally equivalent) components, even if they are not structurally equivalent to the disclosed structures that perform the functions in the exemplary implementations of the invention herein. In addition, although a particular feature of the invention may have been disclosed for one of several implementations, such feature may be combined with one or more other features of other implementations as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "includes," "comprising," "having," "containing," "with," or variations thereof are used in the detailed description and claims, such terms are intended to be inclusive in a manner similar to the term "comprising."

Claims

1. A method for performing depth measurement using an image sensor, the method comprising, for each pixel in at least one pixel: performing an indirect time-of-flight measurement using the pixels to obtain a value indicative of a viewing distance to an imaged object or scene; performing a first coded modulation measurement for the pixel to obtain a first correlation value using a first combination of a modulation code and a reference signal selected to produce a first correlation response having a peak extending over a first range of distances to an imaged object; performing a second coded modulation measurement for the pixel using a second combination of a modulation code and a reference signal selected to produce a second correlation response to obtain a second correlation value, the second correlation response having a peak extending over at least a second range of distances to the imaged object, the second range of distances being contiguous to the first range of distances, wherein a correlation peak for the second modulation code overlaps a correlation peak for the first code in an overlap region consisting of a portion of the first range of distances and a contiguous portion of the second range of distances; determining a first mask value for the pixel by comparing the first correlation value to a first threshold, and determining a second mask value for the pixel by comparing the second correlation value to a second threshold; as well as A distance ambiguity in the value indicating the viewing range is resolved by determining whether the value indicating the viewing range corresponds to an actual distance within the first distance range or within the second distance range based on the first mask value and the second mask value.

2. The method of claim 1 , wherein the peaks of the first correlation response and the second correlation response have negative maximum values, and wherein determining whether the value indicating the line-of-sight distance corresponds to an actual distance within the first distance range or within the second distance range comprises: mapping the value indicative of the line-of-sight distance to the first distance range if the first mask value indicates that the first correlation value is below the first threshold and the second mask value indicates that the second correlation value is above the second threshold; as well as In case the first mask value indicates that the first correlation value is above the first threshold and the second mask value indicates that the second correlation value is below the second threshold, mapping the value indicative of the line of sight to the second distance range.

3. The method of claim 2, wherein determining whether the value indicating the line-of-sight distance corresponds to an actual distance within the first distance range or within the second distance range further comprises, if the first mask value and the second mask value indicate that both the first correlation value and the second correlation value have magnitudes above the respective magnitudes of the first threshold and the second threshold: In a further condition that the magnitude of the first correlation value is greater than the magnitude of the second correlation value, assigning the value indicative of the line of sight to the first distance range; and Otherwise, in a further case that the magnitude of the second correlation value is greater than the magnitude of the first correlation value, the value indicative of the line of sight is assigned to the second distance range.

4. The method of claim 2, wherein the indirect time-of-flight measurement is a continuous wave time-of-flight measurement, and wherein determining whether the value indicative of the line-of-sight range corresponds to an actual distance within the first distance range or within the second distance range further comprises, if the first mask value and the second mask value indicate that both the first correlation value and the second correlation value have magnitudes above the respective magnitudes of the first threshold and the second threshold: In the further case that the phase value obtained from the continuous wave time-of-flight measurement is between 180 degrees and 360 degrees, the value indicative of the line-of-sight is assigned to the first distance range, and otherwise the value indicative of the distance is assigned to the second distance range.

5. The method of claim 1 , wherein the peaks of the first correlation response and the second correlation response have negative maximum values, and wherein determining whether the value indicating the line-of-sight distance corresponds to an actual distance within the first distance range or within the second distance range comprises: mapping the value indicative of the line-of-sight distance to the first distance range if the first mask value indicates that the first correlation value is above the first threshold and the second mask value indicates that the second correlation value is below the second threshold; as well as In case the first mask value indicates that the first correlation value is below the first threshold and the second mask value indicates that the second correlation value is above the second threshold, mapping the value indicative of the line of sight to the second distance range.

6. The method of claim 5, wherein determining whether the value indicating the line-of-sight distance corresponds to an actual distance within the first distance range or within the second distance range further comprises, if the first mask value and the second mask value indicate that both the first correlation value and the second correlation value have magnitudes above the respective magnitudes of the first threshold and the second threshold: In a further condition that the magnitude of the first correlation value is greater than the magnitude of the second correlation value, assigning the value indicative of the line of sight to the first distance range; and Otherwise, in a further case that the magnitude of the second correlation value is greater than the magnitude of the first correlation value, the value indicative of the line of sight is assigned to the second distance range.

7. The method of claim 5, wherein the indirect time-of-flight measurement is a continuous wave time-of-flight measurement, and wherein determining whether the value indicative of the line-of-sight range corresponds to an actual distance within the first distance range or within the second distance range further comprises, if the first mask value and the second mask value indicate that both the first correlation value and the second correlation value have magnitudes above the respective magnitudes of the first threshold and the second threshold: In the further case that the phase value obtained from the continuous wave time-of-flight measurement is between 180 degrees and 360 degrees, the value indicative of the line-of-sight is assigned to the first distance range, and otherwise the value indicative of the distance is assigned to the second distance range.

8. The method of claim 1 , wherein the indirect time-of-flight measurement is a continuous wave time-of-flight measurement including one or more continuous wave phase measurements, and wherein performing the continuous wave time-of-flight measurement for the pixel comprises: For each of the one or more continuous wave phase measurements, a reference signal having a different phase difference relative to the waveform of the emitted light modulating the illumination of the object of interest is used, and for each of the one or more continuous wave phase measurements, a correlation between the light received at the pixel and the reference signal is measured. 9 . The method of claim 8 , wherein performing the continuous wave time-of-flight measurement comprises performing four continuous wave phase measurements for the pixel, the reference signals used for the four continuous wave phase measurements having different phase differences from one another.

10. The method of claim 1, wherein performing the first coded modulation measurement for the pixel comprises: The correlation between the light received at the pixel and the first reference signal is measured using a first reference signal having a cross-correlation function relative to the waveform of the emitted light modulating and illuminating the object of interest, wherein the cross-correlation function has only a single peak with an amplitude exceeding the first threshold over a distance range greater than the blur distance measured for the indirect time-of-flight measurement. The method according to claim 10 , wherein the first reference signal is a Barker code or an m-sequence.

12. The method of claim 11, wherein the second coded modulation measurement uses a second reference signal, the second reference signal being a cyclically rotated version of the first reference signal.

13. The method of claim 1, wherein one or both of the first threshold and the second threshold are based on a reflectivity parameter of the imaged object or scene.

14. An image processing system, comprising: a sensor comprising a plurality of pixels configured to generate a corresponding plurality of pixel signal values ​​in response to received light, wherein each pixel is configured to obtain its corresponding pixel signal value by demodulating the received light using a reference signal; a reference signal generator configured to generate a reference signal and provide the reference signal to the plurality of pixels, and A control circuit is configured to control the reference signal generator and the plurality of pixels to: performing an indirect time-of-flight measurement using the pixels to obtain a value indicative of a viewing distance to an imaged object or scene; performing a first coded modulation measurement for the pixel to obtain a first correlation value using a first combination of a modulation code and a reference signal selected to produce a first correlation response having a peak extending over a first range of distances to an imaged object; performing a second coded modulation measurement for the pixel using a second combination of a modulation code and a reference signal selected to produce a second correlation response to obtain a second correlation value, the second correlation response having a peak extending over at least a second range of distances to the imaged object, the second range of distances being contiguous to the first range of distances, wherein a correlation peak for the second modulation code overlaps a correlation peak for the first code in an overlap region consisting of a portion of the first range of distances and a contiguous portion of the second range of distances; determining a first mask value for the pixel by comparing the first correlation value to a first threshold, and determining a second mask value for the pixel by comparing the second correlation value to a second threshold; as well as A distance ambiguity in the value indicating the viewing range is resolved by determining whether the value indicating the viewing range corresponds to an actual distance within the first distance range or within the second distance range based on the first mask value and the second mask value.

15. The image processing system of claim 14, wherein the peak values ​​of the first correlation response and the second correlation response have negative maximum values, and wherein the control circuit is configured to determine whether the value indicating the viewing distance corresponds to an actual distance within the first distance range or within the second distance range by: mapping the value indicative of the line-of-sight distance to the first distance range if the first mask value indicates that the first correlation value is below the first threshold and the second mask value indicates that the second correlation value is above the second threshold; and In case the first mask value indicates that the first correlation value is above the first threshold and the second mask value indicates that the second correlation value is below the second threshold, mapping the value indicative of the line of sight to the second distance range.

16. The image processing system of claim 15 , wherein the control circuit is configured to, if the first mask value and the second mask value indicate that both the first correlation value and the second correlation value have magnitudes above the respective magnitudes of the first threshold and the second threshold, determine whether the value indicating the viewing distance corresponds to an actual distance within the first distance range or within the second distance range by: In a further condition that the magnitude of the first correlation value is greater than the magnitude of the second correlation value, assigning the value indicative of the line of sight to the first distance range; and Otherwise, in a further case that the magnitude of the second correlation value is greater than the magnitude of the first correlation value, the value indicative of the line of sight is assigned to the second distance range.

17. The image processing system of claim 15 , wherein the indirect time-of-flight measurement is a continuous wave time-of-flight measurement, and wherein the control circuit is configured to, if the first mask value and the second mask value indicate that both the first correlation value and the second correlation value have magnitudes above the respective magnitudes of the first threshold and the second threshold, determine whether the value indicating the line-of-sight distance corresponds to an actual distance within the first distance range or within the second distance range by: In the further case that the phase value obtained from the continuous wave time-of-flight measurement is between 180 degrees and 360 degrees, the value indicative of the line-of-sight is assigned to the first distance range, and otherwise the value indicative of the distance is assigned to the second distance range.

18. The image processing system of claim 14, wherein the peak values ​​of the first correlation response and the second correlation response have negative maximum values, and wherein the control circuit is configured to determine whether the value indicating the viewing distance corresponds to an actual distance within the first distance range or within the second distance range by: mapping the value indicative of the line-of-sight distance to the first distance range if the first mask value indicates that the first correlation value is above the first threshold and the second mask value indicates that the second correlation value is below the second threshold; and In case the first mask value indicates that the first correlation value is below the first threshold and the second mask value indicates that the second correlation value is above the second threshold, mapping the value indicative of the line of sight to the second distance range.

19. The image processing system of claim 18, wherein the control circuit is configured to, if the first mask value and the second mask value indicate that both the first correlation value and the second correlation value have magnitudes above the respective magnitudes of the first threshold and the second threshold, determine whether the value indicating the viewing distance corresponds to an actual distance within the first distance range or within the second distance range by: In a further condition that the magnitude of the first correlation value is greater than the magnitude of the second correlation value, assigning the value indicative of the line of sight to the first distance range; and Otherwise, in a further case that the magnitude of the second correlation value is greater than the magnitude of the first correlation value, the value indicative of the line of sight is assigned to the second distance range.

20. The image processing system of claim 18, wherein the indirect time-of-flight measurement is a continuous wave time-of-flight measurement, and wherein the control circuit is configured to, if the first mask value and the second mask value indicate that both the first correlation value and the second correlation value have magnitudes above the respective magnitudes of the first threshold and the second threshold, determine whether the value indicating the line-of-sight distance corresponds to an actual distance within the first distance range or within the second distance range by: In the further case that the phase value obtained from the continuous wave time-of-flight measurement is between 180 degrees and 360 degrees, the value indicative of the line-of-sight is assigned to the first distance range, and otherwise the value indicative of the distance is assigned to the second distance range.

21. The image processing system of claim 14, wherein the indirect time-of-flight measurement is a continuous wave time-of-flight measurement, and the control circuit is configured to perform the continuous wave time-of-flight measurement for the pixel by: For each of the one or more continuous wave phase measurements, a reference signal having a different phase difference relative to the waveform of the emitted light modulating the illumination of the object of interest is used, and for each of the one or more continuous wave phase measurements, a correlation between the light received at the pixel and the reference signal is measured.

22. The image processing system of claim 21, wherein the control circuit is configured to perform the continuous wave time-of-flight measurement by performing four continuous wave phase measurements for the pixel, the reference signals used for the four continuous wave phase measurements having different phase differences from each other.

23. The image processing system of claim 14, wherein the control circuit is configured to perform the first coded modulation measurement for the pixel by: The correlation between the light received at the pixel and the first reference signal is measured using a first reference signal having a cross-correlation function relative to the waveform of the emitted light modulating and illuminating the object of interest, wherein the cross-correlation function has only a single peak with an amplitude exceeding the first threshold over a distance range greater than the blur distance measured for the indirect time-of-flight measurement.

24. The image processing system according to claim 23, wherein the first reference signal is a Barker code or an m-sequence.

25. The image processing system of claim 24, wherein the second coded modulation measurement uses a second reference signal, the second reference signal being a cyclically rotated version of the first reference signal.

26. The image processing system of claim 14, wherein the control circuit is configured to determine one or both of the first threshold and the second threshold based on a reflectivity parameter of the imaged object or scene.

Images