Distortion determining device and method of determining distortion

Abstract

A distortion determining apparatus and a method of determining distortion are disclosed. A distortion determining apparatus (100) comprises a light source configured to emit light and a photoelectric device (102) configured to receive an electromagnetic signal and convert the signal into a plurality of electrical output signals corresponding to a plurality of predetermined phase shift values according to an indirect time-of-flight measurement technique. Signal processing circuitry (110, 116, 126, 132, 136) of the apparatus (100) is configured to process the electrical output signals to calculate a plurality of measurement vectors derived from the plurality of electrical signals. The vectors are with respect to a plurality of frequencies and comprise a first measurement vector for a fundamental frequency and a second measurement vector for a non-fundamental frequency. The circuitry (110, 116, 126, 132, 136) is configured to calculate a scalar relating a first amplitude of the first vector to a second amplitude of the second vector and use the scalar to identify a distortion with respect to the photoelectric device (102).

Description

Technical Field

[0001] This invention relates to a distortion determination apparatus, such as one employing an indirect time-of-flight measurement technique. The invention also relates to a method for determining distortion, which belongs to the type employing, for example, an indirect time-of-flight measurement technique. Background Technology

[0002] In so-called time-of-flight sensing systems and other systems (such as game console vision systems), it is known to employ an illumination source to illuminate the surrounding environment (sometimes referred to as the "scene") within the field of view of the illumination source, and to process the light reflected by features of that scene. Such so-called LiDAR (Light Detection and Ranging) systems utilize light generated by an illumination source to illuminate the scene, and use detection devices (such as photodiode arrays, some optical elements, and a processing unit) to detect light reflected from objects in the scene. The light reflected from objects in the scene is received by the detection device and converted into an electrical signal, which is then processed by the processing unit by applying time-of-flight (ToF) calculations to determine the distance of the object from the detection device. Although different types of LiDAR systems are known to be based on different operating principles, these systems essentially all illuminate the scene and detect reflected light.

[0003] In this regard, so-called "flash LiDAR" technology (a direct Time-of-Flight ranging technology) uses a light source that emits light pulses, which are then reflected by features in the scene and detected by a detector device. In this type of technology, the distance to the reflecting feature is directly calculated using the round-trip time of the light pulse to the reflecting feature and back to the detector device. The light pulses incident on the detector device are sampled at a very high sampling rate in the time domain. Therefore, the signal path in the processing circuitry used to implement this technology requires high bandwidth and large silicon space; that is, this implementation requires a relatively large area on the silicon wafer, which in turn limits the number of channels that can be supported on the integrated circuit. Therefore, the actual number of channels that such flash LiDAR sensors can support is typically less than 100. To overcome this limitation, moving parts are needed to implement a mechanical scanning system.

[0004] Another known LiDAR system employs a technique called "indirect time-of-flight" (iTOF). An iTOF system emits a continuous-wave light signal, and the reflections of this signal are received and analyzed by a detector device. Multiple samples (e.g., four samples) of light reflected from scene features are acquired within one frame period, with each sample having a phase step, for example, 90°. Using this illumination and sampling method, the phase angle between illumination and reflection can be determined, and the determined phase angle can be used to determine the distance to the reflective features of the scene.

[0005] In iToF systems, high-frequency signal processing (demodulation) occurs at the pixel level, resulting in lower post-pixel bandwidth required to integrate a large number of pixels on the same chip. Consequently, iToF systems can support a greater number of channels compared to direct ToF systems, and therefore support higher spatial resolution measurements.

[0006] However, iToF systems are highly sensitive to the motion of objects in a scene within the measurement frame period, which can lead to so-called "motion artifacts" in the captured image. For an iToF system using a four-phase scheme and a 20MHz modulation frequency at a rate of 50 frames per second, the result is 3.75 ms. -1 Moving objects are budgeted to 1% error for explicit distance calculations. When an object moves laterally relative to the scene, the measured time of flight and consequently distance can vary due to movement occurring between two consecutive phase shifts within a measurement frame. Therefore, the aforementioned iToF system can measure for 200ms when the object is 1 meter in front of the scene background. -1 The object's velocity. This velocity produces motion artifacts, especially at the edges of objects moving laterally to the iToF camera.

[0007] While motion detection can be performed by comparing the phase angle or calculated distance of each pixel in one frame period with the phase or calculated distance of the previous frame period, a frame buffer is required. Furthermore, this process introduces latency because it requires comparing measurements from two consecutive frame periods.

[0008] U.S. Patent No. 9,785,824 describes an iToF (In-Time of Flight) system that uses an illumination signal with an even duty cycle and identifies motion by analyzing the even harmonics of the received signal. At least one even harmonic frequency element is compared to a threshold to detect motion artifacts. However, when a duty cycle other than 0.5 is used, the level of the even harmonic elements increases, regardless of whether the level change is caused by motion artifacts. In this respect, limiting the use of pulse-width symmetrical waveforms is undesirable because some applications require pulse-width asymmetrical waveforms. For example, U.S. Patent Publication No. 2017 / 205497 describes a technique for suppressing odd harmonics (such as the fifth harmonic) in an iToF system by using pulse-width asymmetrical waveforms to reduce the occurrence of circular errors. However, this technique introduces additional power into the even harmonics. Therefore, when using pulse-width asymmetrical waveforms, the technique of US 9,785,824 cannot easily determine motion artifacts. US 2012 / 013887 A1 relates to an iToF system that uses an odd number of phase offset values ​​to mitigate the so-called wobbling effect.

[0009] Another form of distortion suffered by iToF systems is saturation, which occurs when the intensity of the reflected light is at the upper limit of the available dynamic range of the iToF system. Harmonics of the strong received signal aliased back to the desired signal (corresponding to the fundamental frequency), and the increased power leads to errors in the calculation of the extracted phase angle, thus causing errors in the calculated range. Summary of the Invention

[0010] According to a first aspect of the invention, a motion or saturation determination apparatus is provided, comprising: a light source configured to emit light according to an indirect time-of-flight measurement technique; an optoelectronic device configured to receive electromagnetic signals and convert the electromagnetic signals into a plurality of electrical output signals according to the indirect time-of-flight measurement technique, the plurality of electrical output signals respectively corresponding to a plurality of predetermined phase offset values ​​applied within a frame period, the optoelectronic device further configured to store each of the plurality of electromagnetic signals; and a signal processing circuit configured to process the plurality of electrical output signals according to the indirect time-of-flight measurement technique to calculate a plurality of substantially parallel measurement vectors derived from the plurality of electrical output signals generated within the frame period, and the plurality of measurement vectors respectively relating to a plurality of frequencies, including a first measurement vector relating to a fundamental frequency and a second measurement vector relating to a non-fundamental frequency; wherein the signal processing circuit is further configured to calculate a scalar relating a first amplitude of the first measurement vector to a second amplitude of the second measurement vector, and use the scalar to identify motion or saturation of the optoelectronic device within the frame period.

[0011] Distortion can be identified by determining a threshold based on the distortion.

[0012] The signal processing circuit can be configured to calculate a compensated second amplitude of a second measurement vector; the calculation of the compensated second amplitude may include applying a scalar to a first amplitude of a first measurement vector.

[0013] The signal processing circuit can be further configured to subtract the scaled first amplitude from the second amplitude.

[0014] The signal processing circuit can be configured to calculate a compensated second amplitude of a second measurement vector; the calculation of the compensated second amplitude may include applying a scalar to the second amplitude of the second measurement vector and performing a subtraction; and the signal processing circuit may be further configured to subtract a first amplitude from the scaled second amplitude.

[0015] The signal processing circuit can be configured to estimate the true received strength with respect to the optoelectronic device; the estimation of the true received strength may include applying the reciprocal of a scalar to the second amplitude of the second measurement vector.

[0016] According to a second aspect of the present invention, a motion detection device is provided, comprising: a motion or saturation determination device as described above with respect to the first aspect of the present invention; wherein the non-fundamental harmonic frequency is an even harmonic frequency; and a signal processing circuit is configured to calculate the absolute value of the subtraction result and compare the absolute value with a distortion determination threshold.

[0017] The distortion determination threshold can be a predetermined motion threshold.

[0018] The motion detection device may further include signal processing circuitry configured to generate a detected motion mask.

[0019] According to a third aspect of the present invention, a saturation detection device is provided, comprising: the motion or saturation determination device as described above with respect to the first aspect of the present invention; wherein a signal processing circuit is configured to compare the result of subtraction with a distortion determination threshold.

[0020] The distortion determination threshold can be a predetermined saturation threshold.

[0021] The motion detection device may also include signal processing circuitry configured to generate a saturation mask for the detected motion.

[0022] The optoelectronic device may have an associated saturation limit; and the signal processing circuit may be configured to calculate the saturation threshold constituting the distortion determination threshold by calculating the available clearance between the second amplitude and the saturation limit of the optoelectronic device.

[0023] According to a fourth aspect of the invention, a saturation detection device is provided, comprising: a motion or saturation determination device as described above with respect to the first aspect of the invention; wherein the photoelectric device has an associated saturation limit; and a signal processing circuit is configured to calculate the difference between the saturation limit and the calculated true received intensity.

[0024] The distortion determination threshold can be a predetermined saturation threshold; and the signal processing circuit can be configured to compare the calculated difference with the predetermined saturation threshold.

[0025] According to a fifth aspect of the invention, a saturation detection device is provided, comprising: the motion or saturation determination device as described above with respect to the first aspect of the invention; wherein a signal processing circuit is configured to estimate a true received intensity with respect to an optoelectronic device, the estimation of the true received intensity comprising applying the reciprocal of a scalar to a second amplitude of a second measurement vector; the optoelectronic device having an associated saturation limit; and the signal processing circuit is configured to calculate the difference between a first amplitude of a first measurement vector and the calculated true received intensity, thereby providing a measurement of saturation.

[0026] The device may further include a predetermined saturation threshold for identifying saturation; and the signal processing circuitry may be configured to compare the calculated difference with the predetermined saturation threshold.

[0027] The saturation detection device may further include signal processing circuitry configured to generate the detected saturation mask.

[0028] According to a sixth aspect of the present invention, an imaging system is provided, comprising: an array of optoelectronic devices including optoelectronic devices; a saturation detection device as described above regarding a fourth aspect of the present invention, the saturation detection device being configured to detect saturation with respect to each optoelectronic device in the array of optoelectronic devices; wherein a signal processing circuit is configured to perform a first image capture using the array of optoelectronic devices; the signal processing circuit is configured to statistically analyze a difference calculated between a saturation threshold limit and the true received intensity with respect to the array of optoelectronic devices, and to identify regions in the array of optoelectronic devices that experience saturation with respect to the first image capture. The signal processing circuit is configured to modify performance parameters associated with the first image capture and perform a second image capture implementing the modified performance parameters.

[0029] The signal processing circuit can be configured to replace a first plurality of measurements from a first image capture relating to a region in the photoelectric array that has experienced saturation with a second plurality of measurements from a second image capture relating to an identified region in the photoelectric array.

[0030] The device may further include a signal generator configured to generate a carrier signal with an asymmetric pulse width duty cycle. The light source may be configured to emit light using the carrier signal. Signal processing circuitry may be configured to process multiple electrical output signals according to an indirect time-of-flight measurement technique using the carrier signal.

[0031] According to a seventh aspect of the invention, a ranging system is provided, comprising motion or saturation determination means as described above with respect to the first aspect of the invention, wherein the signal processing circuitry is further configured to measure the phase angle of the second measurement vector and calculate the distance using the measured phase angle.

[0032] According to an eighth aspect of the present invention, a method is provided for determining motion or saturation of an optoelectronic device for an indirect time-of-flight measurement apparatus, the method comprising: emitting light according to an indirect time-of-flight measurement technique; generating and storing a plurality of electrical output signals in response to received optical signals and corresponding respectively to a plurality of predetermined phase offset values ​​applied within a frame period according to the indirect time-of-flight measurement technique; processing the plurality of electrical output signals according to the indirect time-of-flight measurement technique to calculate a plurality of measurement vectors substantially parallel to each other and derived from the plurality of electrical output signals generated within the frame period, the plurality of measurement vectors being respectively associated with a plurality of frequencies and including a first measurement vector associated with a fundamental frequency and a second measurement vector associated with a non-fundamental frequency; calculating a scalar relating a first amplitude of the first measurement vector to a second amplitude of the second measurement vector; and using the scalar to identify motion or saturation of the optoelectronic device within the frame period.

[0033] Therefore, an apparatus and method can be provided that offers improved motion and / or saturation detection, regardless of the transient duty cycle of the illumination signal. This apparatus and method detect motion at an improved speed and avoids the use of a frame buffer. Thus, motion artifacts can be detected within a single frame period, regardless of the presence of even harmonics in the measurement being performed. In the case of saturation, this can therefore improve the measurement of dynamic distance and enhance image quality. Attached Figure Description

[0034] Referring to the accompanying drawings, at least one embodiment of the invention will now be described by way of example only, in which:

[0035] Figure 1 This is a schematic diagram of an indirect flight distance time calculation device that constitutes the distortion determination device in an embodiment of the present invention;

[0036] Figure 2 yes Figure 1 A schematic diagram of the motion detection unit of the device;

[0037] Figure 3 It is a flowchart of a distance calculation method that includes detecting distortions constituting another embodiment of the present invention;

[0038] Figure 4 yes Figure 3 The motion detection method of the flowchart and Figure 2 The flowchart shows how the motion detection unit executes this method.

[0039] Figures 5(a) to 5(c) are related to Figure 3 and Figure 4 The diagram shows usage examples related to the method;

[0040] Figure 6 It includes the moved objects and is included. Figure 1The imaging system of the device captures sample images of the fundamental frequency signal;

[0041] Figure 7 yes Figure 6 The moved object is Figure 1 Another sample image captured by the imaging system regarding the second harmonic frequency signal;

[0042] Figure 8 Is using from Figure 6 Sample images and Figure 7 The sample compensation image is generated from the image data of the sample image;

[0043] Figure 9 From Figure 8 The mask generated from the sample compensation image; and

[0044] Figure 10 This is a schematic diagram of another distortion determination unit constituting another embodiment of the present invention;

[0045] Figure 11 It is by Figure 10 The flowchart shows the method for detecting saturation executed by the distortion determination unit and the detection unit.

[0046] Figures 12(a) to 12(c) are related to Figure 11 The diagram shows usage examples related to the method;

[0047] Figure 13 It is a sample image including a first moving object and a second object causing saturation, the sample image being included. Figure 10 The imaging system of the device captures the fundamental frequency signal;

[0048] Figure 14 yes Figure 13 The moving and saturated objects are included Figure 10 The imaging system captures sample images of the second harmonic frequency;

[0049] Figure 15 Is using from Figure 13 Sample images and Figure 14 The sample compensation image is generated from the image data of the sample image;

[0050] Figure 16 From Figure 15 Motion masks generated from sample-compensated images;

[0051] Figure 17 From Figure 16 The saturation mask generated by the motion mask;

[0052] Figure 18 From Figure 17 Another motion mask is generated from the saturation mask;

[0053] Figure 19 It constitutes Figure 1 A schematic diagram of another saturation detection unit of the distortion determination device;

[0054] Figure 20 This is a flowchart of another saturation detection method constituting another embodiment of the present invention;

[0055] Figures 21(a) to 21(c) are about Figure 20 A diagram illustrating the usage of the method;

[0056] Figure 22 It constitutes Figure 1 A schematic diagram of another saturation detection unit of the distortion determination device;

[0057] Figure 23 This is a flowchart of another saturation detection method constituting another embodiment of the present invention;

[0058] Figures 24(a) to 24(c) are about Figure 23 A diagram illustrating the usage examples of the method; and

[0059] Figure 25 Is using Figure 23 The flowchart shows the method for generating depth maps. Detailed Implementation

[0060] Throughout the following description, the same reference numerals will be used to identify the same parts.

[0061] refer to Figure 1The indirect time-of-flight distance calculation system includes an electromagnetic radiation source (not shown), such as a laser diode (LD) or light-emitting diode (LED) constituting the light source. In this example, the electromagnetic radiation source is infrared light whose amplitude is modulated according to indirect time-of-flight measurement techniques to be emitted as a continuous-wave light signal. The distance calculation apparatus of the system includes a distortion determination device 100. The distortion determination device 100 includes a photodetector photonic mixer pixel device 102, which includes a photodiode 104 having an anode operatively coupled to ground potential and a cathode coupled to a first input of a photonic mixer 106, the output of which is coupled to an integrator 108. In this example, for the sake of brevity and clarity, a single photonic mixer pixel device 102 is described. However, those skilled in the art will understand that the distortion determination device 100 includes an array of photonic mixer pixel devices 102 of the above type. Furthermore, the use of the photonic mixer pixel device described herein is an example of a suitable device structure, and those skilled in the art will understand that any suitable optoelectronic device can be employed. In this regard, suitable optoelectronic devices can convert received optical signals from the optical domain to the electrical domain, select the charge generated during the electronically controlled time period (e.g., by mixing signals), and accumulate the selected charge. These operations can be performed by a single monolithic device or by separately connected devices.

[0062] Phase signal generator 112 is configured to generate a continuous wave electrical signal with a carrier signal having an asymmetric pulse width duty cycle. The phase offset of the continuous wave signal can be selected via control input 114, and the phase of the continuous wave signal can be selected from a set of m phase offsets: [θ0, θ1, …, θ…]. m-1 The output of the phase signal generator 112 is coupled to the second input of the photonic mixer 106.

[0063] The output of integrator 108 is coupled to the input of digital Fourier transform (DFT) unit 110. In this respect, phase angle measurements are transmitted serially to DFT unit 110, thereby reducing the memory requirements of distortion determination device 100—DFT unit 110 includes an internal buffer (not shown) to support the serial transmission of measurements from integrator 108. To support this arrangement, DFT unit 110 is operatively coupled to timing control unit 116 to maintain synchronization of data processing.

[0064] The timing control unit 116 has a synchronization output 118 that is operatively coupled to the timing input 120 of the DFT unit 110. The control output 122 of the timing control unit 116 is coupled to the control input 114 of the phase signal generator 112.

[0065] DFT unit 110 has multiple digital in-phase (I) / quadrature (Q) outputs 124. In this example, DFT unit 110 includes b pairs of digital I / Q outputs corresponding to different harmonics of the measured signal. Since the output of integrator 108 is accumulated charge, and therefore in this example, the output of integrator 108 needs to be converted to the digital domain in the analog domain. This can be achieved, for example, by using a photon counter instead of photon mixer device 102 and integrator 108, or by providing an analog-to-digital converter before DFT unit 110.

[0066] The first pair of I / Q outputs of the plurality of digital I / Q outputs 124, relating to the first harmonic or fundamental frequency of the received reflected light signal, is coupled to a phase angle calculation unit, such as a Cartesian-polar coordinate converter 126. In this example, the Cartesian-polar coordinate converter 126 includes an arctangent unit with respect to the phase angle and a first amplitude calculation unit, such as a Euclidean norm or so-called taxi norm calculation unit. The Cartesian-polar coordinate converter 126 includes a phase angle output 128 operably coupled to the arctangent unit and an amplitude output 130 operably coupled to the first amplitude calculation unit.

[0067] A second pair of I / Q outputs from multiple digital I / Q outputs 124, associated with higher-order even-order harmonic frequency signals (e.g., second harmonics), is coupled to the input of a second amplitude calculation unit 132, such as a Euclidean norm or so-called taxi norm calculation unit. The output of the second amplitude calculation unit 132 is coupled to a first input 134 of a distortion determination unit 136, and a second input 138 of the distortion determination unit 136 is coupled to the amplitude output 130 of a Cartesian-polar coordinate converter 126. A third input 140 of the distortion determination unit 136 is a motion threshold input configured to receive a distortion determination threshold; in this example, the distortion determination threshold is a motion threshold C. m The distortion threshold, in this example, is the motion threshold C. m This can be predetermined and applied to one or more photonic mixer pixel devices, such as all photonic mixer pixel devices. In some examples, the motion threshold C m The motion threshold C can be recalculated during system use, for example, through higher-level processing (not shown). In this regard, system parameters (e.g., integration time and illumination power) may change during system operation, thus requiring recalculation, which can be based on these parameters. Those skilled in the art will also understand that other factors influence the motion threshold C. m The calculation of this threshold C can affect its recalculation, such as dynamic distance and / or decision reliability; the latter may require adjustments to the motion threshold C. mEmpirical adjustments may be made. Recalculation may be necessary in response to modifications to one or more measurement parameters, such as integration time or illumination power. The distortion threshold may be stored in the digital memory (not shown) of device 100 and used when needed. The distortion determination unit 136 also includes a distortion detection output 142, such as a motion detection output.

[0068] In this example, the DFT unit 110, the Cartesian-polar coordinate converter 126, the second amplitude calculation unit 132, and the distortion determination unit 136 constitute the signal processing circuit.

[0069] Turning Figure 2 The distortion determination unit 136 constituting the motion artifact detection device includes a first summing unit 200, which has a first input 202 coupled to a first input 134 of the distortion determination unit 136. A second sign-inverting input 204 of the first summing unit 200 is coupled to the output of a scaling unit 206, and the input of the scaling unit 206 is coupled to a second input 138 of the distortion determination unit 136. The output 208 of the first summing unit 200 is coupled to the input of an absolute value calculator 210. The output of the absolute value calculator 210 is coupled to the first input of a first comparator 212, and the second input of the first comparator 212 is coupled to a third input 140 of the distortion determination unit 136. The output of the first comparator 212 is coupled to the distortion detection output 142 of the distortion determination unit 136.

[0070] In operation ( Figure 3 In step 300, the light source emits a continuous wave light signal that illuminates the scene. In this example, the light source uses a carrier signal generated by phase signal generator 112, such as light emitted by objects in the scene reflecting light. Phase signal generator 112 generates a continuous wave electrical signal, and timing control unit 116 controls a loop through a set of phase shifts of the electrical signal relative to the continuous wave light signal. A synchronization signal is also applied to DFT unit 110 via synchronization output 118.

[0071] To detect distortion with respect to the photonic mixer pixel device 102, the phase angle and amplitude are calculated by applying an electrical carrier signal generated by the phase signal generator 112 to the photonic mixer 106, and the phase offset of the electrical signal is cyclically passed through the set of phase offsets. A digital representation of the charge of each phase offset in the set of phase offsets, stored in the integrator 108, is measured (step 302) and received in series by the DFT unit 110 and converted (step 304) into a first pair of I / Q outputs constituting a first I / Q vector V1 in the complex domain and a second pair of I / Q outputs constituting a second I / Q vector V2 in the complex domain. In this respect, the integrator 108 provides multiple per-phase-off amplitude measurement outputs in series, representing various accumulated charge levels with respect to the applied phase offset values ​​of the photonic mixer pixel device 102. The DFT unit 110 calculates intermediate I and Q values ​​of the separately received phase-separated amplitude measurements for each frame period, which are accumulated within the frame period to generate the final I and Q value results. The operation of this arrangement involves iteratively computing a vector for each incoming phase angle measurement using DFT unit 110.

[0072] After converting the electrical measurement signal to the frequency domain, the DFT unit 110 provides the I and Q values ​​of the fundamental frequency and second harmonic frequency at its output. In this example, a synchronization signal ensures that the Cartesian-polar coordinate converter 126 and the second amplitude calculation unit 132 synchronously receive the fundamental frequency I / Q output and second harmonic I / Q output of the current measurement frame from the DFT unit 110. Then, the Cartesian-polar coordinate converter 126 calculates (step 306) the angle of the first vector V1 according to the indirect time-of-flight measurement technique, which constitutes the calculated phase angle extracted (measured) in the complex plane according to the fundamental frequency I and Q values. meas. The Cartesian-polar coordinate converter 126 also calculates (step 308) the amplitude of the first vector V1, which constitutes the calculated amplitude a1 extracted (measured) in the complex plane according to the fundamental frequency I and Q values.

[0073] At essentially simultaneously, the second amplitude calculation unit 132 calculates (step 310) the amplitude of the second vector V2, which constitutes the calculated amplitude a2 extracted (measured) in the complex plane based on the harmonic frequencies I and Q.

[0074] It has been found that, after compensation, the analysis of the amplitude of the output vector of the even harmonics of the electrical output signal originating from the integrator 108 in the complex domain provides an indication of distortion, such as motion or saturation, in scenarios independent of the duty cycle of the continuous wave optical signal emitted by the aforementioned electromagnetic radiation source. In fact, as will be explained in further detail below, for some applications, the use of harmonic frequency signals is not limited to even harmonic frequencies. It should also be understood that, in some examples, the 0th harmonic frequency signal can constitute an even harmonic frequency signal, and in some examples, it can be used.

[0075] Generally speaking, the Fourier expansion coefficients of the i-th harmonic of a rectangular pulse can be written as:

[0076] (1)

[0077] Using equation (1) above, the relationship between the pulse fundamental frequency amplitude and higher harmonics can be written as:

[0078] (2)

[0079] Considering the above relationship (2) regarding even harmonics, such as the second harmonic frequency, the relationship between the sine and cosine trigonometric functions can be used to rewrite the above expression (2) as follows:

[0080] (3)

[0081] Equation (1) shows that when the duty cycle D is 0.5, the amplitude of the even harmonic frequency is 0. However, this is not the case for duty cycles other than 0.5. In this regard, as shown above, the amplitude of the even harmonic frequency signal of the optical power received by the photonic mixer pixel device 102 is related to the factor β. i The received optical power is amplitude-dependent on the fundamental frequency signal. This is especially true for even-harmonic frequency signals, where the factor β... i Non-zero for duty cycles other than 0.5. However, this phenomenon is used to compensate for the calculated amplitude of even-harmonic frequency signals by scaling the amplitude of the fundamental frequency signal, in order to detect distortion. In this regard, the scaling factor β... i This can be used to identify distortions in the photonic mixer pixel device 102. Detection can be achieved by analyzing the compensated amplitude α of the even-order harmonic frequency signal relative to zero. i This is achieved because if no further distortion occurs in the scene or there are changes (e.g., external forces applied to some or all of the system's hardware and / or no distortion in signal processing), the compensated amplitude α of the even harmonic frequency signal is... i 'It is zero.' In this regard, when reflected light is received from an object that is suddenly moved into or out of the scene during the frame period, the resonant content of the power of the light signal received by the photonic mixer pixel device 102 changes, and therefore the compensated amplitude a i 'Becomes non-zero. After compensation, the amplitude a' i 'Use the following formula to calculate:'

[0082] (4)

[0083] In the formula, i is the order of the harmonic frequency and is even, while a iTherefore, it is the amplitude of the even-harmonic frequency signal. a1 is the amplitude of the fundamental frequency signal. In one example, during factory calibration using a calibration scenario, the scaling factor β can be predetermined. i This information is then stored in the digital memory of device 100. In other examples, the scaling factor β can be determined in the field during use (e.g., when device 100 is powered on). i .

[0084] refer to Figure 4 And additional reference Figure 2 and Figure 3 The distortion determination unit 136 receives (step 400) a first calculated amplitude a1 from the Cartesian-polar coordinate converter 126 at its second input 138, and receives (step 402) a second calculated amplitude a2 from the second amplitude calculation unit 132 at its first input 134. The scaling unit 206 receives the first calculated amplitude a1 via the second input 138 of the distortion determination unit 136 and calculates it using a scaling factor β. i Scaling (step 404) of the first calculated amplitude a1. In the current example, where the second harmonic frequency signal is used to detect distortion, the scaling factor is β2. Therefore, the first calculated amplitude a1 is scaled using the scaling factor β2, which relates the amplitude of the fundamental frequency signal to the amplitude of the second harmonic frequency signal. The scaled first calculated amplitude β2a1 is then received at the second sign inversion input 204 of the first summing unit 200, and the second calculated amplitude a2 is received by the first summing unit 200 via the first input 134 of the distortion determination unit 136.

[0085] The summing unit 200 then sums the second calculated amplitude a2 and the scaled first calculated amplitude β2a1 with its sign reversed (step 406). The result of the summation (actually subtraction), i.e., the output of the first summing unit 200, constitutes the calculation of the compensated amplitude a2′ (step 312) and is received by the absolute value calculator 210, which calculates the absolute value from the output of the first summing unit 200 (step 408). Then, the first comparator 212 receives the output of the absolute value calculator 210, which is the absolute value obtained by subtracting the scaled first calculated amplitude β2a1 from the second calculated amplitude a2. Then, the first comparator 212 compares the output of the absolute value received from the absolute value calculator 210 with the motion threshold C received by the distortion determination unit 136 via its third input 140. m Comparison ( Figure 3 Step 314). In such cases... Figure 4 In this example, shown in more detail, the first comparator 212 compares the received absolute value with the motion threshold C. m Compare (step 410), and if the absolute value equals the motion threshold C m Or greater than the motion threshold C mThe first comparator 212 outputs (step 412) a motion detection signal for the photon mixer pixel device 102, for example, a logic high signal; otherwise, the first comparator 212 outputs (step 414) a logic low signal as a motion detection signal.

[0086] Referring to Figure 5(a), an illumination source with a duty cycle D (not 0.5, e.g., 0.4) illuminates inanimate objects in the scene that reflect the illumination light. Phase signal generator 112 also uses the same duty cycle D. The reflected light is received by photon mixer pixel device 102, and a first amplitude a1 and a second amplitude a2 are generated by Cartesian-polarity converter 126 and second amplitude calculation unit 132, respectively, with respect to the fundamental frequency vector V1 and the second harmonic vector V2 output by DFT unit 110, as described above. Since the duty cycle D is not 0.5, the second harmonic signal output by DFT unit 110 is non-zero even when there is no motion to be detected in the scene. Using a previously calculated scaling factor, the absolute value a2' of the compensated second amplitude returns to zero from its uncompensated value to provide a value comparable to a threshold for motion detection (if present). Subtracting the scaled first calculated amplitude β2a1 from the second calculated amplitude a2 of the second harmonic frequency signal from the inanimate objects in the scene yields a zero or near-zero value, independent of the duty cycle D of the continuous wave light signal emitted by the aforementioned electromagnetic radiation source. In this example, the photonic mixer pixel device 102 is unaffected by motion, so the second calculated amplitude a2 remains at its nominal value. Therefore, the scaled first calculated amplitude β2a1 is the expected value of the second calculated amplitude a2, hence the subtraction result is zero. This is below the motion threshold C. m Therefore, this will result in the calculation of undetected motion.

[0087] Turning to Figure 5(b), where the reflected light received by the photon mixer pixel device 102 originates from a moving object in the scene, assuming the same lighting parameters, the second calculated amplitude a2 rises above the nominal level typically associated with no detected motion. Therefore, the compensated second amplitude a2', formed by the absolute value of the difference between the second calculated amplitude a2 of the second harmonic frequency signal and the scaled first calculated amplitude β2a1, is non-zero and above the motion threshold C. m This will lead to the discovery that the photon mixer pixel device 102 has been affected by motion.

[0088] Referring to Figure 5(c), assuming the same illumination parameters, the value of the second calculated amplitude a2 does not always increase, and in some instances, it may decrease when the photon mixer pixel device 102 is affected by motion. Therefore, subtracting the scaled first calculated amplitude β2a1 from the second calculated amplitude a2 may result in a negative value. However, performing an absolute value calculation to compensate for the second calculated amplitude a2 converts the negative result of the subtraction into a positive result. Since the photon mixer pixel device 102 is affected by motion, the compensated second amplitude a2' remains non-zero and exceeds the motion threshold C. m This will cause the photon mixer pixel device 102 to be affected by motion.

[0089] The system can use the output generated by the first comparator 212 to identify one or more regions of the depth map generated by the system, including motion artifacts. As described below, where feasible, these regions can be remeasured, for example, by modifying one or more parameters associated with the depth measurement.

[0090] Repeat the above steps (steps 300 to 314 and 400 to 414) (step 316) until depth maps are no longer needed.

[0091] refer to Figures 6 to 9 This allows for a better demonstration of the effects of the aforementioned processing. In this regard, it is assumed that the indirect time-of-flight distance calculation system, including the distortion determination device 100, includes the necessary functional elements used in conjunction with the aforementioned processing elements to generate a depth map of a scene, for example, a scene including a movie projector on a table, the projector having a film reel rotating when generating the depth map, the depth map being generated at a predetermined frame rate (e.g., 50 frames per second). Reference Figure 6 As the reel moves, an amplitude map of the fundamental frequency signal generated by the DFT unit 110 is captured at any given time point. Although invisible, the amplitude map contains information (not shown) that can be used to support the identification of motion artifacts in the depth map. In this example, such information can be used to detect motion artifacts associated with the reel's movement, which are of course moving as the depth map is generated. Therefore, the depth / distance measurement of the photonic mixer pixel device 102, by receiving reflected light from the rotating reel, is affected by the motion that causes a certain degree of distortion with respect to the reel.

[0092] Go to Figure 7 The recorded amplitude diagram is about the second-order even-order harmonic frequency signal generated by DFT unit 110, which is the corresponding output signal of the fundamental frequency signal generated by DFT unit 110. Although not easily visible to the naked eye, the amplitude diagram generated using the second-order harmonic signal ( Figure 7The second harmonic frequency signal has a higher amplitude associated with moving parts in the scene, such as the edges of scroll spokes. However, in order to eliminate the influence of illumination light with a non-0.5 duty cycle (i.e., asymmetric pulse width waveform) on the second harmonic frequency signal, the amplitude of the second harmonic frequency signal generated by the photonic mixer pixel device 102 needs to be compensated in the manner described above, thereby obtaining... Figure 8 The compensated amplitude diagram.

[0093] refer to Figure 9 The output of the first comparator 212 yields a graph, which can be arranged by the system's signal processing circuitry as a motion mask for identification. Figure 6 The depth map showing the location affected by motion can be used for subsequent processing as described above to improve... Figure 6 Distorted regions in the depth map. In this regard, the system can use a mask to identify erroneous depth information. Depending on the strategy applied by the system to process potentially erroneous depth pixels, the system can selectively assign low confidence levels to pixels determined to be affected by motion artifacts and / or remove or compensate for lost or unreliable depth information, for example, in a manner related to illumination power and / or integration time as described above.

[0094] refer to Figure 10 In another embodiment, the distortion determination unit 136 is configured to identify saturation in relation to the photon mixer pixel device 102, thus the device 100 constitutes a saturation detection device. In this regard, it should be understood that, depending on the circumstances and function, saturation is not considered a phenomenon occurring only with respect to the photon mixer pixel device 102 or a particular component thereof, and saturation can occur alone or in combination with other parts of the light receiving "chain," including the photon mixer pixel device 102, due to limitations imposed by other parts of the chain. The distortion determination unit 136 in this example relates to... Figure 2 The difference in the previously described example is the absence of an absolute value calculator 210. Therefore, the distortion determination unit 136 includes a first summing unit 200, whose first input 202 is coupled to a first input 134 of the distortion determination unit 136, and a second sign-inverted input 204 of the first summing unit 200 is coupled to the output of the scaling unit 206. The input of the scaling unit 206 is coupled to a second input 138 of the distortion determination unit 136. The output 208 of the first summing unit 200 is coupled to a first input of a first comparator 212, and the second input of the first comparator 212 is coupled to a third input 140 of the distortion determination unit 136. In this and further examples, the distortion threshold is a saturation threshold C. sat This can be predetermined and applied to one or more photon mixer pixel devices, such as all photon mixer pixel devices. For example, integration time and illumination power can be used to calculate the saturation threshold C. satHowever, those skilled in the art should also understand that other factors affect the saturation threshold C. sat The calculation of this threshold C can affect its recalculation, such as dynamic distance and / or decision reliability; the latter may require adjustments to the saturation threshold C. sat Adjustments are made based on experience. In some examples, the saturation threshold C... sat The distortion threshold can be recalculated during system use, for example, through higher-level processing (not shown). Recalculation may be necessary in response to modifications to one or more measurement parameters, such as integration time or illumination power. As described above, the distortion threshold can be stored in the digital memory (not shown) of device 100 and used when needed. In this example, the output of the first comparator 212 is coupled to a distortion detection output 142, which serves as a saturation detection output.

[0095] In operation ( Figure 3 In this process, the scene is illuminated, and reflected light is measured per pixel by each photon mixer pixel device 102 of the system according to an indirect time-of-flight measurement technique. The DFT unit 110 generates output vector components related to the fundamental frequency and higher harmonic frequencies in the manner described above with respect to the previous embodiment (steps 300 to 304). As described above, the phase angle and amplitude of the fundamental frequency signal are calculated by a Cartesian-polar coordinate converter 126, and the amplitude of the higher harmonic frequency signal is calculated by a second amplitude calculation unit 132. In this example, the Cartesian-polar coordinate converter 126 therefore calculates a first extracted (measured) calculated amplitude a1 in the complex plane based on the fundamental frequency signals I and Q, and the second amplitude calculation unit 132 calculates a second extracted (measured) calculated amplitude a2 in the complex plane based on the values ​​of the second harmonic frequencies I and Q (steps 306 to 310). Reference Figure 11 After receiving (steps 400 and 402) the first and second calculated amplitudes a1 and a2, the distortion determination unit 136 calculates (steps 404 and 406) a compensated second amplitude a2', which is formed by subtracting the scaled first calculated amplitude β2a1 from the second calculated amplitude a2 of the second harmonic frequency signal. Then, the compensated second amplitude a2' is compared with the saturation a value C applied via the third input 140 of the distortion determination unit 136 by the first comparator 212. sat A comparison (step 416) is made to determine whether the level of the compensated second amplitude a2' is equal to or exceeds the saturation threshold C. sat In this case, the compensated second amplitude a2' equals or exceeds the saturation threshold C. sat In the case of a certain condition, the first comparator 212 outputs (step 418) a saturation detection signal for the photon mixer pixel device 102, for example, a logic high, indicating that saturation has been detected. Otherwise, the first comparator 212 outputs (step 420) a logic low, for example, as a saturation detection signal.

[0096] Referring to Figure 12(a), a scene composed of objects with ordinary reflectivity is illuminated using a lighting source with a duty cycle D that is not 0.5, which reflects the illumination light. The phase signal generator 112 also uses the same duty cycle D. The reflected light is received by the photon mixer pixel device 102, and a first amplitude a1 and a second amplitude a2 are generated with respect to the fundamental frequency vector V1 and the second harmonic vector V2, as described above. These components are output by the DFT unit 110 and processed by the Cartesian-polar coordinate converter 126 and the second amplitude calculation unit 132, respectively. Because the duty cycle D is not 0.5, i.e., the waveform is pulse-width asymmetric, the second harmonic signal output by the DFT unit 110 is non-zero even when no saturation with respect to the scene occurs. Using a previously calculated scaling factor, the second amplitude a2 returns from its uncompensated value to zero to provide a value comparable to a threshold for detecting saturation (if present). The photonic mixer device 102 is not saturated by reflected light received from objects in the scene, i.e., the first amplitude a1 is below the saturation limit value L of the photonic mixer 102. sat In this case, the second calculated amplitude a2 of the second harmonic frequency signal minus the scaled first calculated amplitude β2a1 produces a zero value or a substantially zero saturation limit value L that is independent of the duty cycle of the continuous wave light signal emitted by the aforementioned electromagnetic radiation source. sat The margin provided by subtracting from the known limitations of the analog-to-digital conversion section of the photonic mixer device 102 and the signal processing circuitry can be considered. In this example, the photonic mixer pixel device 102 is not saturated, so the second calculated amplitude a2 remains at its nominal value. Therefore, the scaled first calculated amplitude β2a1 is the expected value of the second calculated amplitude a2, and thus the subtraction yields a result of zero. This is below the saturation threshold C. sat Therefore, it will result in the detection of saturation not being detected.

[0097] Turning to Figure 12(b), where the reflected light received by the photonic mixer pixel device 102 originates from a highly reflective object in the scene, the first calculated amplitude a1 reaches the saturation limit value L corresponding to the saturation of the photonic mixer pixel device 102. sat This means that the photon mixer pixel device 102 is in a saturated state. The second calculated amplitude a2 is therefore at a nominal level typically associated with no detected saturation. However, the compensated second amplitude a2', formed by subtracting the scaled first calculated amplitude β2a1 from the second calculated amplitude a2 of the second harmonic frequency signal, is reduced to zero or essentially zero by applying a compensation factor in the form of the scaled first calculated amplitude β2a1. At this level, the compensated second amplitude a2' is below the saturation threshold C. sat This will result in the discovery that the photon mixer pixel device 102 is not yet saturated, because the photon mixer pixel device 102 in saturation is within acceptable operating limits.

[0098] Referring to Figure 12(c), when a highly reflective object in the scene causes the reflected light to saturate the photon mixer pixel device 102, the compensated second amplitude a2' becomes non-zero and exceeds the saturation threshold C. sat This will lead to the discovery that the photon mixer pixel device 102 is already saturated.

[0099] Similar to the method described above for detecting motion artifacts, the system can use the output generated by the first comparator 212 to identify one or more regions of the depth map generated by the system that are affected by saturation. As described below, where feasible, these regions can be remeasured, for example, by modifying one or more parameters associated with the depth measurement.

[0100] refer to Figures 13 to 18 This better illustrates the effects of the aforementioned processing. In this regard, assume the system includes the necessary functional elements used in conjunction with the processing elements described above to generate a depth map of a scene, for example, a scene with a movie projector on a table with highly visible clothing—in this example, a jacket placed behind the projector. As described in the previous example, the projector has a film reel that rotates during depth map generation, and the depth map is generated at a predetermined frame rate (e.g., 50 frames per second). [Reference] Figure 13 As the reel moves, an amplitude map of the fundamental frequency signal generated by the DFT unit 110 is captured at any given time point. Although invisible, the amplitude map contains information that can be used to support the identification of motion artifacts in the depth map (not shown). In this example, such information can be used to enable the detection of motion artifacts associated with the reel's movement, which are of course moving as the depth map is generated. However, some pixels, particularly those related to the high-visibility jacket, are saturated. Therefore, depth / distance measurements performed by the photonic mixer pixel device 102, which receives reflected light from the rotating reel and the high-visibility jacket, are affected by the reel's movement and the jacket's high reflectivity, respectively, resulting in distortions related to the reel and the jacket.

[0101] Go to Figure 14 The recorded amplitude diagram is about the higher-order (e.g., second-order) harmonic frequency signal generated by DFT unit 110, which is the corresponding output signal of the fundamental frequency signal generated by DFT unit 110, as described above. Although not easily visible to the naked eye, the amplitude diagram generated using the second-order harmonic frequency signal ( Figure 14The amplitude level of the second harmonic frequency signal is higher or lower than the normal expected amplitude level associated with moving objects and highly reflective objects in the scene (such as the spoke edges of a scroll and a high-visibility jacket). However, in order to eliminate the influence of illumination light with a non-0.5 duty cycle (i.e., asymmetric pulse width waveform) on the second harmonic frequency signal, the amplitude of the second harmonic signals generated by the photonic mixer pixel device 102 needs to be compensated in the manner described above, thereby obtaining... Figure 15 The amplitude map shows that the compensated amplitude map regions associated with motion or reflection have larger amplitudes. In fact, the motion detection threshold C... m Combined with the first comparator 212, it can be used to generate a motion detection mask ( Figure 16 ).exist Figure 16 In this context, saturation is incorrectly detected as motion in the scene. However, this incorrect detection of saturated regions as motion artifacts is not problematic in situations where motion detection is required but motion correction is not necessary. Furthermore, using a saturation threshold C... sat The saturation mask and the output of the first comparator 212 can also be generated by the system's signal processing circuitry. Figure 17 ).

[0102] refer to Figure 18 Since the threshold for saturation detection is greater than the threshold for motion detection, i.e., C sat >C m Therefore, it is possible Figure 16 In motion detection masks, saturated regions and motion regions are distinguished. In this regard, a saturated mask ( Figure 17 ), can be used for motion masks ( Figure 16 Filtering (e.g., correction) is performed to exclude saturated regions from the motion mask. An example filtering technique employs a Boolean combination of the motion mask and the saturated mask described herein, e.g., a motion mask and (unsaturated mask), which can be applied to the depth map. This motion mask and / or saturated mask can be used for subsequent processing as described above to improve... Figure 13 Distortions in the depth map. In this regard, the system can use a mask to identify erroneous depth information. Depending on the strategy applied by the system for handling potentially erroneous depth pixels, the system can optionally assign low confidence levels to pixels determined to be affected by saturation and / or remove or compensate for lost or unreliable depth information, for example, in a manner related to illumination power and / or integration time as described above.

[0103] The above steps (steps 300 to 314, 400 to 406, and 416 to 420) are repeated (step 316) until depth maps are no longer needed.

[0104] In the above embodiments, the saturation detection technique is similar to motion detection because compensation is applied to the higher-order harmonic frequency signal analyzed against a reference fixed threshold. During normal operation, the saturation detection threshold C is selected. sat So that it exceeds the motion detection threshold C m And thus, saturation is incorrectly detected as motion without the additional filtering / correction described above. However, in another embodiment, the compensated amplitude α of the higher-order harmonic frequency signal can be compared. i The amplitude of the fundamental frequency signal a1 of the photon mixer pixel device 102 relative to the saturation limit value L sat The available headroom can be used to distinguish motion artifacts from saturation. Headroom can be determined from the saturation limit L. sat The calculation is performed by subtracting the amplitude a1 of the fundamental frequency signal.

[0105] refer to Figure 19 The distortion determination unit 136 is configured to identify the saturation of the photon mixer pixel device 102 according to the aforementioned saturation clearance technique. In this example, the distortion determination unit 136 and Figure 10 The description differs from the previous example, as follows: The distortion determination unit 136 includes a first summing unit 200, whose first input 202 is coupled to a first input 134 of the distortion determination unit 136, and a second sign-inverted input 204 of the first summing unit 200 is coupled to the output of the scaling unit 206. The input of the scaling unit 206 is coupled to a second input 138 of the distortion determination unit 136. The output 208 of the first summing unit 200 is coupled to a first input of a second comparator 214, replacing the first comparator 212. The second input of the second comparator 214 is coupled to the output 216 of a second summing unit 218, and a first sign-inverted input 220 of the second summing unit 218 is coupled to a second input 138 of the distortion detection unit 136. The second input 222 of the second summing unit 218 is coupled to a third input 140 of the distortion determination unit 136, in this example, where the third input 140 provides a saturation limit value L that can be stored in the aforementioned digital memory. sat The output of the second comparator 214 is coupled to the distortion detection output 142 of the distortion determination unit 136, which is used as a saturation detection output in this example.

[0106] In operation ( Figure 3In this process, the scene is illuminated, and reflected light is measured per pixel by each photon mixer pixel device 102 of the system according to an indirect time-of-flight measurement technique. The DFT unit 110 generates components of the output vector in the manner described above with respect to the previous embodiment, the vector being about the fundamental frequency and higher harmonic frequencies (steps 300 to 304). As described above, the phase angle and amplitude with respect to the fundamental frequency signal are calculated by the Cartesian-polar coordinate converter 126, and the amplitude with respect to the higher harmonic frequency signal is calculated by the second amplitude calculation unit 132. In this example, the Cartesian-polar coordinate converter 126 therefore calculates a first extracted (measured) calculated amplitude a1 in the complex plane according to the fundamental frequency signals I and Q, and the second amplitude calculation unit 132 calculates a second extracted (measured) calculated amplitude a2 in the complex plane according to the values ​​of the second harmonic frequencies I and Q (steps 306 to 310). Figure 20 After receiving (steps 400 and 402) the first and second calculated amplitudes a1 and a2, the distortion determination unit 136 calculates (steps 404 and 406) a compensated second amplitude a2', which is formed by subtracting the scaled first calculated amplitude β2a1 from the second calculated amplitude a2 of the second harmonic frequency signal. Essentially simultaneously, the first calculated amplitude a1 is applied to the first sign-inverted input 220 of the second summing unit 218, and the saturation limit value L... sat The second input 222 is applied to the second summing unit 218. In response to these inputs, the second summing unit 218 generates (step 422) an output value, which is the saturation limit value L that constitutes the saturation threshold. sat And the first calculated amplitude a1 (i.e., L) sat The difference between –a1) and then the second comparator 214 compares the compensated second amplitude a2' with the saturation threshold L. sat –a1 is compared (step 424) to determine whether the level of the compensated second amplitude a2' is less than the saturation threshold L. sat –a1. When the compensated second amplitude a2′ is equal to or greater than the saturation threshold L sat When –a1, the second comparator 214 outputs (step 426) a saturation detection signal for the photon mixer pixel device 102, for example, a logic high, indicating that saturation has been detected. Otherwise, the second comparator 214 outputs (step 428), for example, a logic low, as a saturation detection signal.

[0107] Repeat the above steps (steps 300 to 314, 400 to 406, and 424 to 428) (step 316) until depth maps are no longer needed.

[0108] Referring to Figure 21(a), a scene containing a common reflectivity object related to the previous embodiment is illuminated using an illumination source with a duty cycle D that is not 0.5, reflecting the illumination light. The phase signal generator 112 also uses the same duty cycle D. The reflected light is received by the photon mixer pixel device 102, and the DFT unit 110 outputs components of the fundamental frequency vector V1 and the second harmonic vector V2. The Cartesian-polar coordinate converter 126 and the second amplitude calculation unit 132 calculate the first amplitude a1 and the second amplitude a2 as described above. Since the duty cycle D is not 0.5, i.e., the waveform of the continuous wave light signal is pulse-width asymmetric, the second harmonic frequency signal output by the DFT unit 110 is non-zero even when no saturation with respect to the scene occurs. Using a previously calculated scaling factor, the second amplitude a2 returns from its uncompensated value to zero to provide a value compatible with the calculated saturation threshold L. sat –a1 is compared to detect saturation (if present). Therefore, when the photon mixer pixel device 102 is not saturated by reflected light received from objects in the scene, subtracting the scaled first calculated amplitude β2a1 from the second calculated amplitude a2 of the second harmonic frequency signal yields a zero or substantially zero value, regardless of the duty cycle D, for the continuous wave light signal emitted by the aforementioned electromagnetic radiation source. In this example, the photon mixer pixel device 102 is not saturated, so the second calculated amplitude a2 remains at its nominal value. Therefore, the scaled first calculated amplitude β2a1 is the expected value of the second calculated amplitude a2, and thus the subtraction produces a zero or near-zero result. Therefore, the compensated second amplitude a2' is less than the saturation threshold L. sat –a1, therefore, will result in the output indication at saturation detection output 142 not detecting saturation. It should be understood that this saturation detection method is superior to the previously described saturation detection techniques because, for unsaturated images, the detection threshold can be set higher relative to saturated images. Therefore, a higher saturation threshold reduces the likelihood of compensated amplitudes caused by motion being incorrectly detected as saturation.

[0109] Turning to Figure 21(b), where the reflected light received by the photon mixer pixel device 102 originates from a highly reflective object in the scene, the first calculated amplitude a1 reaches a level L corresponding to the saturation of the photon mixer pixel device 102. sat This means that the photon mixer pixel device 102 is in a saturated state. Therefore, the second calculated amplitude a2 is at the nominal level that is typically associated with no detected saturation. However, the compensated second amplitude a2′, formed by subtracting the scaled first calculated amplitude β2a1 from the second calculated amplitude a2 of the second harmonic frequency signal, is reduced to zero or essentially zero by applying a compensation factor in the form of the scaled first calculated amplitude β2a1. At this level, the compensated second amplitude a2′ is equal to the saturation threshold L. sat–a1, therefore will cause the output at saturation detection output 142 to indicate that the photon mixer pixel device 102 is saturated or in a saturated state.

[0110] Referring now to Figure 21(c), when a highly reflective object in the scene causes the illumination light reflected from it to saturate the photon mixer pixel device 102, the compensated second amplitude a2′ becomes non-zero and exceeds the saturation threshold L. sat –a1, and this will result in the output at saturation detection output 142 indicating that the photon mixer pixel device 102 is saturated. Due to the saturation threshold L sat –a1's dynamic properties, in this case, the saturation threshold L sat –a1 becomes zero or slightly less than zero, thus falling below the motion detection threshold C. m Therefore, compared to the saturation detection techniques described above, this provides better detection of saturated pixels affected by motion, i.e., selectivity.

[0111] In another embodiment, the photonic mixer pixel device 102 is considered to be unaffected by the saturation limit value L. sat Constraints. When not subject to the saturation limit value L sat When constrained, a compensation factor is used to adjust the amplitude a1 of the fundamental frequency of the reflected light received by the photon mixer pixel device 102. Modeling is performed. This is essentially an estimate of the actual received light intensity of the photonic mixer pixel device 102. The calculated amplitude is determined by the distortion determination unit 136 and is actually a virtual amplitude because the photonic mixer pixel device 102 will eventually saturate and therefore has a saturation limit. The distortion calculation unit 136 also calculates the virtual amplitude a1 of the photonic mixer pixel device 102. and saturation limit value L sat The difference between them. This difference is the residual amplitude, i.e., the virtual amplitude a1. Exceeding the saturation limit L sat The amplitude. In order to detect the saturation of the photonic mixer pixel device 102, the remaining amplitude a1 can be... -L sat Compared to the amplitude ai' of the compensated higher-order harmonic frequency signal, it is clear from the explanation below that the remaining amplitude a1 -L sat Comparison with a threshold is sufficient, for example, the saturation threshold C. sat Set to a value specific to this example.

[0112] The scaling factor β can be adjusted. i The reciprocal or inverse of the value is applied to the amplitude α of the higher-order harmonic frequency signal generated by the DFT unit 110. i In order to model (e.g., project) the amplitude a1 of the fundamental frequency signal The unconstrained value mentioned above is called the virtual amplitude. (Reference) Figure 22 The configuration of the distortion calculation unit 136 is slightly different because it no longer requires the first calculated amplitude a1 from the Cartesian-polar coordinate converter 126. Instead, the second input 138 of the distortion calculation unit 136 is now used to provide the saturation limit value L stored in the digital memory. sat As described in the previous example, the first input 134 of the distortion calculation unit 136 is coupled to the input of the second scaling unit 224, which is configured to apply a scaling factor β. i The reciprocal of β i -1 The output of the second scaling unit 224 is coupled to the first input 226 of the third summing unit 228, and the second sign-inverted input 230 of the third summing unit 228 is coupled to the second input 138 of the distortion calculation unit 136. The output 232 of the third summing unit 228 is coupled to the first input of the third comparator 234. The second input of the third comparator 234 is coupled to the third input 140 of the distortion calculation unit 136, from which the aforementioned saturation threshold C is provided. sat The output of the third comparator 234 is coupled to the saturation detection output 142 of the distortion calculation unit 136.

[0113] During operation ( Figure 3 According to the indirect time-of-flight measurement technique, the scene is illuminated and reflected light is measured by each photon mixer pixel device 102 of the system on a pixel-by-pixel basis. The DFT unit 110 generates output vectors in the manner described above with respect to the previous embodiment, these vectors being about the fundamental frequency and higher harmonic frequencies (steps 300 to 304). As described above, the phase angle and amplitude about the fundamental frequency signal are calculated by the Cartesian-polar coordinate converter 126, and the second amplitude calculation unit 132 calculates the amplitude about the higher harmonic frequency signal. In this example, the Cartesian-polar coordinate converter 126 therefore calculates a first extracted (measured) calculated amplitude a1 in the complex plane according to the fundamental frequency I and Q values, and the second amplitude calculation unit 132 calculates a second extracted (measured) calculated amplitude a2 in the complex plane according to the second harmonic frequency I and Q values ​​(steps 306 to 310). Although the first calculated amplitude a1 is not used to determine saturation in this example, it can be used for other purposes, such as as a confidence indicator for selecting pixels for further processing. Reference Figure 23 Distortion determination unit 136 calculates (step 430) the reciprocal β of the scaling factor. i -1 And after receiving (step 432) the second calculated amplitude a2, the distortion determination unit 136 calculates (step 434) the virtual amplitude a1. The virtual amplitude a1 The second scaling unit 224 uses the reciprocal of the scaling factor β. i -1 The second calculated amplitude a2 is formed by scaling. Then, it is obtained from the virtual amplitude a1 through the third summing unit 228. Subtract the saturation limit L sat To calculate the remaining amplitude (step 436). Then, the third comparator 134 will calculate the remaining amplitude L. sat -a1 With saturation threshold C sat Compare (step 438) to determine the remaining amplitude L sat -a1 Is it greater than or equal to the saturation threshold C? sat When the remaining amplitude L sat -a1 Equal to or greater than the saturation threshold C sat When the saturation detection signal is detected, the third comparator 234 outputs (step 440) a saturation detection signal for the photon mixer pixel device 102, for example, a logic high signal, indicating that saturation has been detected. Otherwise, the third comparator 234 outputs (step 442) a logic low signal, for example, as a saturation detection signal.

[0114] Referring to Figure 24(a), a scene containing the aforementioned ordinary reflectivity object and reflecting the illumination light is illuminated using a lighting source with a duty cycle D that is not 0.5. The phase signal generator 112 also uses the same duty cycle D. The reflected light is received by the photonic mixer pixel device 102, and the DFT unit 110 generates the components of the fundamental frequency vector V1 and the second harmonic vector V2, respectively. As described above, the Cartesian-polar coordinate converter 126 and the second amplitude calculation unit 132 calculate the first amplitude a1 and the second amplitude a2 based on the components of the fundamental frequency vector V1 and the second harmonic vector V2. Since the duty cycle D is not 0.5, i.e., the waveform of the continuous wave light signal is pulse-width asymmetrical, the second calculated amplitude a2 of the second harmonic signal output by the DFT unit 110, as well as other higher harmonics, is non-zero even when there is no saturation of the photonic mixer pixel device 102 caused by the scene. The residual amplitude L calculated from the second calculated amplitude a2... sat -a1 Used as a measure of saturation, it can be used to calculate the threshold C. sat A comparison is made to detect saturation (if present). When the photon mixer pixel device 102 is not saturated by reflected light received from objects in the scene, the remaining amplitude L... sat -a1 It is a negative value and is unrelated to the duty cycle D of the continuous wave light signal emitted by the aforementioned electromagnetic radiation source. In this example, the photonic mixer pixel device 102 is not saturated, therefore the remaining amplitude L... sat -a1 It is negative. Because the remaining amplitude Lsat -a1 Less than the saturation threshold C sat (Not shown in Figure 24(a)), the output at saturation detection output 142 indicates that saturation was not detected.

[0115] Turning to Figure 24(b), where the reflected light received by the photonic mixer pixel device 102 comes from a highly reflective object in the scene, the first calculated amplitude a1 reaches level L corresponding to the photonic mixer pixel device 102 in a saturated state. sat Therefore, the second calculated amplitude a2 is at the nominal level typically associated with no detected saturation. However, the calculated residual amplitude L... sat -a1 It is zero or essentially zero. At this level, the remaining amplitude L sat -a1 Equal to the saturation threshold C sat Furthermore, the output at saturation detection output 142 indicates that the photon mixer pixel device 102 is saturated or in a saturated state.

[0116] Referring now to Figure 24(c), when a highly reflective object in the scene saturates the photon mixer pixel device 102 with the illumination light reflected from it, the residual amplitude L sat -a1 It becomes non-zero and positive, thus exceeding the saturation threshold C. sat Therefore, the output at saturation detection output 142 indicates that the photon mixer pixel device 102 is saturated.

[0117] In another embodiment, the system is an imaging system that includes a statistical processing engine that uses the residual amplitude L sat -a1 As input, it is used to identify saturated pixels or saturated regions at the image level. In this regard, the residual amplitude L can be used before the second measurement scene. sat -a1 This can be used to modify (for example) reduce one or more configuration / setting parameters, such as integration time and / or illumination power. Therefore, the second measurement is not expected to be less saturated or less saturated than the first measurement. In one example, the statistical processing engine analyzes the depth map generated with respect to the first frame period and determines the maximum residual amplitude, then uses the maximum residual amplitude as the basis for modifying one or more measurement parameters (e.g., the integration time and / or illumination power mentioned above) with respect to a second subsequent frame period. In another example, saturated regions of the first depth map image can be replaced by non-saturated corresponding regions of the second depth map image acquired in a second frame period acquired immediately after the first frame period. Furthermore, for some applications, statistical processing may include filtering and / or algorithmic processing to ignore small clusters of saturated pixels and / or saturation lines and avoid applying compensation to these pixels, as the performance of the application (e.g., image recognition) is not affected by such a small number of pixel saturations.

[0118] refer to Figure 25 The system generates (step 500) a first depth map constituting the first image capture within the first frame period, and uses the aforementioned technique to calculate the residual amplitude L with respect to each or substantially each photon mixer pixel device 102 of the system. sat -a1 Then, the first depth map is systematically and statistically evaluated (step 502) to identify the remaining amplitude L. sat -a1 The region in the first depth map that exceeds zero. Statistical analysis may include statistical analysis of the region reaching the saturation limit value L. sat The amplitude of the pixels, statistically exceeding the saturation limit value L. sat The remaining amplitude L of the pixel sat -a1 And analyze the collected data to identify those that produce values ​​above the saturation limit L. sat The maximum number of pixels with the highest amplitude. The remaining amplitude L with the highest count. sat -a1 It can be used as the saturation limit value L sat The above are the most common measurements of residual amplitude. The most common residual amplitude L sat -a1 and saturation limit value L sat The difference between them can be used as a measure of saturation and to modify the integration time, illumination intensity, or a combination thereof, through, for example, the saturation limit value L represented by the calculated amplitude difference. sat The ratio. In another example, the pixel amplitude and residual amplitude can be aggregated by combination. In this regard, the statistical processing engine can analyze the generated count data to identify the residual amplitude L. sat -a1 As used to determine (step 504) the given calculated residual amplitude Lsat -a1 Is it equal to or below an acceptable saturation threshold? In one example, the amplitude cell with the highest count can be used as the acceptable saturation threshold. In another example, a minimum count value can be predetermined and used to identify the amplitude cell with the highest count above the minimum count value; the identified amplitude cell is used as the acceptable saturation threshold, such as its midpoint.

[0119] If the saturation threshold is not exceeded, the saturation level is considered acceptable, and the above process (steps 500 to 504) is repeated, generating a further depth map using the same measurement parameters. However, if the saturation threshold is exceeded, one or more measurement parameters, such as the integration time and / or power of the electromagnetic radiation source (step 506), are modified to attempt to reduce saturation, and another depth map is generated in a second frame period immediately following the first frame period using adaptive measurement parameters (step 508). The remaining amplitude L in the first depth map is then identified. sat -a1 Regions with amplitudes greater than zero are identified in the corresponding second depth map and correlated with the remaining amplitude L in the first depth map. sat -a1 Measurements associated with regions identified as having a value greater than zero are replaced with the corresponding regions in the second depth map (step 510). The corrected depth map is then output for further processing. The above process (steps 500 to 512) is repeated until depth map generation is no longer needed.

[0120] Although in the example above, the residual amplitude L is used sat -a1 As a measure of excessive amplitude, a virtual amplitude a1 can be used. Replace the remaining amplitude L sat -a1 And the saturation threshold C can be increased accordingly. sat To compensate for virtual amplitude a1 Direct use, with the remaining amplitude L sat -a1 on the contrary.

[0121] In another example, modify Figure 22 The distortion determination unit 136 is configured to provide a first calculated amplitude a1 at the second input 138 of the distortion calculation unit 136, instead of providing a saturation limit value L at the second input 138. sat Regarding the third comparator 234, the saturation threshold C satAccordingly, this is increased, for example, by the reciprocal of the scaling factor βi. At the output 232 of the third summing unit 228, for positive values, the generated output value will correspond to the residual amplitude. Therefore, this residual amplitude can be used to identify saturated pixels and / or modify one or more configuration / setting parameters to mitigate or correct the effects of saturation in a manner similar to the measurement of the residual amplitude described above.

[0122] When a pixel has not experienced saturation, the calculated residual amplitude and the first calculated amplitude a1 are approximately equal, therefore the difference between these two values ​​is evaluated as zero, which is less than the saturation threshold C. sat Therefore, the output of the third comparator 234 produces, for example, a logic low, as a saturation detection signal indicating whether the pixel is not saturated or is saturated. When the pixel is saturated, the calculated residual amplitude and the first calculated amplitude a1 remain substantially equal, so the difference between these two values ​​is evaluated as zero. Therefore, for example, the third comparator 234 still produces a logic low. However, when the pixel is saturated, the calculated residual amplitude becomes greater than the first calculated amplitude a1, so the difference between these two values ​​is evaluated as non-zero, and when the difference is greater than the saturation threshold C... sat For example, the output of the third comparator 234 generates a logic high, which serves as a saturation detection signal indicating pixel saturation.

[0123] Those skilled in the art should understand that the implementations described above are merely examples of various implementations conceivable within the scope of the appended claims. Indeed, regarding the above example of detecting saturation as distortion, it should be understood that although the use of even-order harmonic frequency signals has been described, odd-order harmonic frequency signals can also be used for saturation detection. In the context of equation (4) above, “i∈N” is relative to “i∈2N”. Regarding the first example above concerning the detection of motion artifacts as distortion, even-order harmonic frequency signals can be used to detect motion artifacts. However, it should be understood that in this example and the other examples described herein, even-order harmonic frequencies can include the 0th harmonic frequency. Indeed, in this example, the even-order harmonic frequency signal can be a combination of two or more even-order harmonic frequency signals allowed by the number of phase offset values ​​employed, such as a combination of the 0th harmonic frequency and the second harmonic frequency.

[0124] Although the scaling factor β and scaling factor β described in this article -1 The reciprocal is calculated by the distortion determination unit 136, but those skilled in the art should understand that these factors can be pre-calculated and stored, for example, during the camera setup phase.

Claims

1. A motion or saturation determination device, comprising: The light source is configured to emit light based on indirect time-of-flight measurement technology; An optoelectronic device is configured to receive electromagnetic signals and convert the electromagnetic signals into multiple electrical output signals corresponding to multiple predetermined phase offset values ​​applied within a frame period, according to the indirect time-of-flight measurement technique; the optoelectronic device is also configured to store each of the multiple electromagnetic signals. A signal processing circuit is configured to process the plurality of electrical output signals according to the indirect time-of-flight measurement technique to calculate a plurality of measurement vectors substantially parallel to and derived from the plurality of electrical output signals generated within the frame period, the plurality of measurement vectors being relative to a plurality of frequencies, and including a first measurement vector relative to a fundamental frequency and a second measurement vector relative to a non-fundamental frequency; wherein The signal processing circuit is further configured to: Determine the first amplitude of the first measurement vector; Determine the second amplitude of the second measurement vector; Calculate a scaled version of the first amplitude; Calculate the difference between the second amplitude and the scaled version of the first amplitude; The difference is compared with a distortion determination threshold. as well as The motion or saturation of the optoelectronic device within the frame period is identified based on the results of the comparison.

2. The apparatus of claim 1, wherein the distortion determination threshold includes a motion threshold or a saturation threshold.

3. The apparatus of claim 1, wherein the calculation of the scaled version of the first amplitude includes applying a scaling factor to the first amplitude.

4. The apparatus of claim 1, wherein the signal processing circuitry is further configured to subtract a scaled version of the first amplitude from the second amplitude.

5. A motion detection device, comprising: The light source is configured to emit light based on indirect time-of-flight measurement technology; An optoelectronic device is configured to receive electromagnetic signals and convert the electromagnetic signals into multiple electrical output signals corresponding to multiple predetermined phase offset values ​​applied within a frame period, according to the indirect time-of-flight measurement technique; the optoelectronic device is also configured to store each of the multiple electromagnetic signals. A signal processing circuit is configured to process the plurality of electrical output signals according to the indirect time-of-flight measurement technique to calculate a plurality of measurement vectors substantially parallel to and derived from the plurality of electrical output signals generated within the frame period, the plurality of measurement vectors being relative to a plurality of frequencies, and including a first measurement vector relative to a fundamental frequency and a second measurement vector relative to a non-fundamental frequency; wherein The signal processing circuit is further configured to: Determine the first amplitude of the first measurement vector; Determine the second amplitude of the second measurement vector; Calculate the scaled version of the first amplitude; Calculate the difference between the second amplitude and the scaled version of the first amplitude; The difference is compared with a distortion determination threshold. as well as The motion of the optoelectronic device within the frame period is identified based on the comparison results. in, The non-fundamental frequency is an even harmonic frequency; and The signal processing circuit is further configured to calculate the absolute value of the difference and compare the absolute value of the difference with the distortion determination threshold.

6. The motion detection device of claim 5, wherein the distortion determination threshold is a predetermined motion threshold.

7. A saturation detection device, comprising: The light source is configured to emit light based on indirect time-of-flight measurement technology; An optoelectronic device is configured to receive electromagnetic signals and convert the electromagnetic signals into multiple electrical output signals corresponding to multiple predetermined phase offset values ​​applied within a frame period, according to the indirect time-of-flight measurement technique; the optoelectronic device is also configured to store each of the multiple electromagnetic signals. A signal processing circuit is configured to process the plurality of electrical output signals according to the indirect time-of-flight measurement technique to calculate a plurality of measurement vectors substantially parallel to and derived from the plurality of electrical output signals generated within the frame period, the plurality of measurement vectors being relative to a plurality of frequencies, and including a first measurement vector relative to the fundamental frequency and a second measurement vector relative to non-fundamental frequencies; wherein, The signal processing circuit is further configured to: Determine the first amplitude of the first measurement vector; Determine the second amplitude of the second measurement vector; Calculate the scaled version of the first amplitude; Calculate the difference between the second amplitude and the scaled version of the first amplitude; Compare the difference with a distortion determination threshold; and The saturation of the optoelectronic device within the frame period is identified based on the comparison results. The distortion determination threshold is a predetermined saturation threshold. Or one of them, The optoelectronic device has an associated saturation limit; and The signal processing circuit is configured to calculate the saturation threshold constituting the distortion determination threshold by calculating the available clearance between the first amplitude and the saturation limit of the optoelectronic device.

8. A saturation detection device, comprising: The light source is configured to emit light based on indirect time-of-flight measurement technology; An optoelectronic device is configured to receive electromagnetic signals and convert the electromagnetic signals into multiple electrical output signals corresponding to multiple predetermined phase offset values ​​applied within a frame period, according to the indirect time-of-flight measurement technique; the optoelectronic device is also configured to store each of the multiple electromagnetic signals. A signal processing circuit is configured to process the plurality of electrical output signals according to the indirect time-of-flight measurement technique to calculate a plurality of measurement vectors substantially parallel to and derived from the plurality of electrical output signals generated within the frame period, the plurality of measurement vectors being relative to a plurality of frequencies, and including a first measurement vector relative to a fundamental frequency and a second measurement vector relative to a non-fundamental frequency; wherein The signal processing circuit is further configured to: Determine the second amplitude of the second measurement vector; The inverse of the scaling factor is applied to the second amplitude to determine the virtual amplitude of the first measurement vector; Calculate the difference between the virtual amplitude and the saturation limit associated with the optoelectronic device; The difference is compared with a distortion determination threshold. as well as The saturation of the optoelectronic device within the frame period is identified based on the results of the comparison.

9. The apparatus of claim 8, wherein: The distortion determination threshold is a predetermined saturation threshold; and The signal processing circuit is configured to compare the calculated difference with the predetermined saturation threshold.

10. An imaging system, comprising: Optoelectronic device array, including optoelectronic devices; The saturation detection device as claimed in claim 8 is configured to detect the saturation of each optoelectronic device in the optoelectronic device array; in The signal processing circuit is configured to perform a first image capture using the optoelectronic device array; The signal processing circuit is configured to statistically analyze the difference between the saturation limit and the calculated difference with respect to the actual received intensity of the optoelectronic array, and identify regions in the optoelectronic array that experience saturation with respect to the first image capture; and The signal processing circuit is configured to modify the performance parameters associated with the first image capture and perform a second image capture that implements the modified performance parameters.

11. A method for determining motion or saturation of an optoelectronic device for an indirect time-of-flight distance measurement apparatus, the method comprising: Light is emitted based on indirect time-of-flight measurement technology; In response to the received optical signal and corresponding to a plurality of predetermined phase offset values ​​applied within a frame period according to the indirect time-of-flight measurement technique, a plurality of electrical output signals are generated and stored. The plurality of electrical output signals are processed according to the indirect time-of-flight measurement technique to calculate a plurality of measurement vectors that are substantially parallel and derived from the plurality of electrical output signals generated within the frame period, the plurality of measurement vectors being relative to a plurality of frequencies, and including a first measurement vector relative to the fundamental frequency and a second measurement vector relative to a non-fundamental frequency; Determine the first amplitude of the first measurement vector; Determine the second amplitude of the second measurement vector; Calculate the scaled version of the first amplitude; Calculate the difference between the second amplitude and the scaled version of the first amplitude; The difference is compared with a distortion determination threshold. as well as The motion or saturation of the optoelectronic device within the frame period is identified based on the results of the comparison.

12. A method for determining the saturation of an optoelectronic device for an indirect time-of-flight distance measurement apparatus, the method comprising: Light is emitted based on indirect time-of-flight measurement technology; In response to the received optical signal and corresponding to a plurality of predetermined phase offset values ​​applied within a frame period according to the indirect time-of-flight measurement technique, a plurality of electrical output signals are generated and stored. The plurality of electrical output signals are processed according to the indirect time-of-flight measurement technique to calculate a plurality of measurement vectors that are substantially parallel and derived from the plurality of electrical output signals generated within the frame period, the plurality of measurement vectors being relative to a plurality of frequencies, and including a first measurement vector relative to the fundamental frequency and a second measurement vector relative to a non-fundamental frequency; Determine the second amplitude of the second measurement vector; The inverse of the scaling factor is applied to the second amplitude to determine the virtual amplitude of the first measurement vector; Calculate the difference between the virtual amplitude and the saturation limit associated with the optoelectronic device; The difference is compared with a saturation threshold; as well as The saturation of the optoelectronic device within the frame period is identified based on the results of the comparison.

Images