Spread phase of time-of-flight modulated light

Abstract

Adaptive phase unwrapping for a time-of-flight camera. The scene is illuminated with modulated light of two or more frequencies, where these frequencies do not share a common integer denominator. The modulated light reflected from objects within the scene is received at a sensor array. The received modulated light is then processed and weighted in a complex domain to determine the unwrapped phase for each of the two or more modulated light frequencies.

Description

Background Technology

[0001] Time-of-flight (ToF) imaging systems can be used to generate depth images of an environment, where each pixel of the depth image represents the distance to a corresponding point in the environment. The distance to a point on the imaging surface in the environment is determined based on the length of the time interval (i.e., ToF) during which light emitted by the imaging system travels to that point and then returns to the sensor array in the imaging system. An optical ToF camera measures this interval for a number of points on the surface, thereby assembling a depth image where the depth coordinates for each pixel in the depth image are proportional to the ToF observed at that pixel. Summary of the Invention

[0002] This "Summary" is provided to illustrate, in a simplified form, the selection of concepts further described in the following "Detailed Description." This "Summary" is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to addressing any or all of the shortcomings pointed out in any part of this disclosure.

[0003] Using a time-of-flight camera, a scene is illuminated with modulated light of two or more frequencies, where the frequencies do not share a common integer denominator. The modulated light reflected from objects within the scene is received at the sensor array of the time-of-flight camera. The received modulated light is then processed to determine the unfolding phase of each of the two or more modulated lights at each frequency. Attached Figure Description

[0004] Figure 1 An example electronic device with an embedded or peripheral digital camera is shown.

[0005] Figure 2 An example digital camera is shown.

[0006] Figure 3 An example method for determining the unfolding phase via a time-of-flight camera is shown.

[0007] Figure 4 An example method for adjusting phase expansion parameters based on noise conditions is shown.

[0008] Figure 5A An example graph showing the expansion through rate is provided.

[0009] Figure 5B It shows Figure 5A A scaled-down version of the example diagram.

[0010] Figure 6AAn example diagram of phase expansion up to 16.67 m is shown for noise conditions.

[0011] Figure 6B The following is shown for the zero-noise condition. Figure 6A An example of phase unfolding diagram.

[0012] Figure 6C An example diagram of phase expansion up to 50m is shown for noise conditions.

[0013] Figure 6D The following is shown for the zero-noise condition. Figure 6C An example of phase unfolding diagram.

[0014] Figure 6E It shows Figure 6C A truncated version of the example graph.

[0015] Figure 6F It shows Figure 6D A truncated version of the example graph.

[0016] Figure 7 A non-limiting embodiment of the computing system is illustrated schematically. Detailed Implementation

[0017] Optical time-of-flight (ToF) imaging has become a dominant depth imaging technique due to the development of high-speed, high-resolution optical sensor arrays. "Phase-based" optical ToF imaging is an important variant of this technique, where depth is calculated based on the phase lag of modulated light reflected from an object. Devices employing this technique are increasingly being found in industrial and consumer applications, including equipment automation, gaming and virtual reality, biometrics, and facial recognition. This disclosure relates to improvements in ToF imaging that utilize modulated light with two or more frequencies having no common integer denominator to achieve multi-frequency imaging. This, in turn, allows for the tuning of the frequencies used, thereby improving depth measurement, reducing power consumption, and / or providing other advantages.

[0018] Figure 1 Four different examples of electronic devices (100A-D) with embedded or peripheral digital cameras (102A-D) are illustrated. Device 100A is a smartphone including camera 102A. Device 100B is a personal computer including webcam 102B. Device 100C is a video game system including peripheral 3D camera 102C. Device 100D is a virtual reality headset including 3D camera 102D. In these and other device environments, the correction methods disclosed herein can be applied to the output of these and other types of user-facing and / or world-facing digital cameras.

[0019] Figure 2 An aspect of an example imaging environment 200 including a digital camera 202 in one implementation is illustrated. The digital camera 202 includes: a sensor array 204 having a plurality of sensor elements 206 (e.g., pixels), a controller 208, and an objective lens system 210. The objective lens system 210 is configured to focus an image of at least one surface 220 of a photographic object 222 onto the sensor array 204. The controller 208 is configured to collect and process data from the various sensor elements 206, thereby constructing a digital image of the object. The controller 208 may be implemented across one or more computing devices, examples of which are described herein. Figure 7 discuss.

[0020] The digital image constructed by the controller 208 can be represented as a numerical array, where each array represents a set of pixels (X, Y). j Each pixel in the array provides a value S. j The X, Y position of each pixel in the digital image is mapped to the associated sensor element 206 of the sensor array 204, and further mapped to the corresponding associated trajectory 224 on the surface 220 via the objective system 210. In some implementations, the mapping of image pixels to sensor elements can be a 1:1 mapping, but other mappings such as 1:4, 4:1, etc., can also be used. In some implementations, the digital camera 202 can be configured to acquire a time-resolved sequence of digital images of the object 222—i.e., video.

[0021] Pixel (X, Y) j S j There are no particular restrictions on the dimensions of the values. In some examples, S j It can be a specified pixel (X, Y). j The brightness is a real or integer scalar value. In other examples, S j It can be a specified pixel (X, Y). j A vector of real or integer values ​​of color—for example, using the scalar component values ​​of the red, green, and blue color channels. In other examples, S j It can be preprocessed to include complex values. Where a and b are integers or real numbers. As described in more detail below, the complex value S... j This can be used to represent the signal response of the sensor element of an optical ToF camera that uses continuous wave (CW) modulation and phase discrimination to resolve radial distance. Such a camera is referred to herein as a "CW-ToF camera".

[0022] Figure 2The digital camera 202 is a CW-ToF camera, configured to resolve radial distance Z to a plurality of points j on surface 220 of the photographed object 222. To implement phase-based ToF imaging, the CW-ToF camera 202 includes a modulated light emitter 230 and a sensor array 204 having an analog and / or digitally modulated electronic shutter 232. The modulated light emitter 230 can be configured to emit electromagnetic radiation of any frequency detectable by the sensor element 206. For example, the modulated light emitter 230 can be an infrared and / or near-infrared light-emitting diode (LED) or a laser diode (LD), and the sensor array can be a high-resolution array of complementary metal-oxide-semiconductor (CMOS) sensor elements 206. Located behind the objective system 210 and a wavelength filter 234, which can be an external optical component of the sensor or a filter layer directly on the sensor 204, the sensor array is configured to image light from the modulated light emitter 230, which is reflected from surface 220 and returns to the camera. Other optical ToF cameras may include different variations of optical, light emitter, and / or sensor arrays—for example, charge-coupled device (CCD) sensor arrays or microlens objective arrays.

[0023] The electronic shutter 232 can refer to a controlled voltage signal, including any suitable modulated waveform, having an adjusted bias optionally applied simultaneously to certain electrode structures of the various sensor elements 206 of the sensor array 204. In some examples, the electrode structure receiving the controlled voltage signal with bias may include a current collector that, depending on the level of the bias voltage signal, causes photoelectrons generated within the sensor element 206 to drift to the current collector and the current via the voltage converter is measured as current or voltage. In some examples, the electrode structure receiving the controlled bias voltage signal may include a gate that, depending on the level of the bias voltage signal, causes photoelectrons to drift to the current collector or to the voltage converter.

[0024] The controller 208 of the CW-ToF camera 202 includes a modulation control engine 240, which is configured to modulate the light emitter 230 and synchronously modulate the electronic shutter 232 of the sensor array 204. In some examples, the light emitter 230 and the electronic shutter 232 are modulated at one or more predetermined frequencies having a predetermined angular phase shift. The predetermined angular phase shift The delay of the electronic shutter modulation relative to the light emitter modulation is controlled. In some examples, "modulation" as used herein refers to the fundamental harmonics of a sinusoidal or digitized quasi-sinusoidal waveform and / or a rectangular waveform, which simplifies the analysis. However, this feature is not absolutely necessary, as modulation using other waveforms can be substituted.

[0025] In some implementations, the sensor array 204 images a component of the reflected light, which causes the transmitter to modulate and lag a series of predetermined phase shifts. Each phase shift in the sensor array. The shutter acquisition engine 242 of the controller 208 is configured to interrogate the sensor array 204 to retrieve the acquired signal value S from each sensor element 206. j A digital image captured in this manner is called the "raw shutter". The raw shutter 243 can be represented as having a shutter speed provided for each sensor element 206. Specific actual strength value S j And the coordinates (X, Y) of the position of the specific sensor element 206 within the sensor array 204. j An associated array of values. This is achieved by capturing values ​​with three or more distinct phase offsets. Three or more consecutive raw shutters 243 can be configured to reveal the actual phase lag of the light reflected back to each sensor element 206. The "phase image". The phase image is a sequence of images specified for each sensor element j. And the coordinates (X, Y) of the position of the specific sensor element 206 within the sensor array 204. j An associated array of values. In some implementations with signal preprocessing, each signal value S... j It is a complex number Where 'a' is the signal component in phase with the transmitter modulation, and 'b' is the signal component that lags the transmitter modulation by 90°. In this case, the complex signal value S j With modulus ||S j ||Related and with phase lag

[0026]

[0027] In implementations where the phase-independent reflectivity of the object is also of interest, a given phase image can be replaced by its modulus or by the square of its modulus for each complex signal value S. j This image is then processed. In this paper, this image is referred to as an "effective brightness" image.

[0028] Using data from single-phase images or the raw shutter group of the components, the radial distance Z between the depth camera and the surface point imaged at each sensor element j can be conditionally estimated. j More specifically, the depth can be solved using the following formula.

[0029]

[0030] Where c is the speed of light, f is the modulation frequency, and N is a non-negative integer.

[0031] When the depth value Z j The above solution is unique when the entire range is no greater than half the distance light travels in one modulation period c / (2f), in which case N is a constant. Otherwise, the solution is indeterminate and periodic. Specifically, with the same phase lag... Surface points with depth differences of c / (2f) can be observed. Depth image data can only be resolved to this extent—for example, data from a single-phase image or the corresponding raw shutter triplet—which is called "aliasing" or "wrapping".

[0032] To resolve depths greater than c / (2f), additional phase images can be calculated using the raw shutter speeds acquired at different modulation frequencies. In some examples, three, four, five, or more frequencies can be used; in others, two frequencies are sufficient. The combined inputs from all the raw shutter speeds (e.g., nine in the case of three frequencies and six in the case of two frequencies) are sufficient to uniquely determine each Z. j Redundant depth imaging of the same object and image frames that provides non-periodic depth estimation is called "dealiasing" or "unwrapping"; this function is performed in the unwrapping engine 244 of the controller 208. The unwrapped depth image, or "radial distance map," can be represented as having a radial distance value Z provided for each pixel. j And the coordinates (X, Y) of the specified pixel position. j An associated array of values. For example, the modulation control engine 240 can simultaneously output modulation signals to the phase shifter. For each modulation frequency f m The phase shift step size (>=3) can be set to equal distances within 2π. Then, the phase-shifted rf signal can be used to demodulate the signal at each pixel.

[0033] In some implementations, pixels in an unfolded depth image can be classified into one or more segments according to a single-layer or multi-layer (i.e., hierarchical) classification scheme. The segmentation engine 246 of controller 208 can be configured to specify the classification. In some examples, pixels can be classified as foreground or background. In some examples, pixels classified as foreground can be further classified as human objects. In some examples, pixels classified as human objects can be further classified as "object head," "object hand," etc. The classified digital image can be represented as having a signal value S provided for each pixel. j and category value C j And the coordinates (X, Y) of the specified pixel position. j An associated array of numbers.

[0034] Optionally, in a video implementation, model fitting can be applied to track the motion of the classified depth image segments frame by frame. In an example where the depth imaging object includes a person, the classified depth image segment corresponding to the hand can be segmented from the rest of the object. In depth video applications, the hand segment can then be tracked using a sequence of depth image frames and / or a kinematic model. For example, the tracked hand segment can be used as input for a virtual reality video game or as gesture input for controlling a computer. Tracking can be performed in the tracking engine 248 of the controller 208. Naturally, the method in this paper can be extended to various other segmentation and tracking tasks that can be performed on the output of a phase-based CW-ToF camera.

[0035] Using a single higher radio frequency can increase depth resolution, but the unambiguity of doing so is limited (e.g., 0.75m at 200MHz). Therefore, multiple higher radio frequencies are typically used to extend the measurement distance following phase unfolding techniques. Multiple depth maps of the same environment can be captured using emitted light with different modulation frequencies, thereby improving accuracy in environments with different reflectivities and / or partially occluded objects.

[0036] Current unrolling techniques are performed in the phase domain (phase vector domain). This can lead to many limitations. For example, the values ​​of multiple frequencies are restricted to have an integral relationship with the common denominator between the frequencies (e.g., [153,162,198] / 9 = [17 18 22]) to provide a repeatable phase period where all frequencies periodically overlap at a known distance. Typically, the fundamental frequency is relatively low to set the unrolling distance, and the applied frequencies are relatively high. If there is no common integer, the frequencies will still meet, but this may occur at infinity or some other irrational number.

[0037] This limits the flexibility of constructing economically adaptable ToF camera systems, as the array of potential frequency configurations is limited to common correlation values. Therefore, many frequency configurations that could improve power efficiency may not be available. The phase domain unrolling process increases computationally as the number of frequencies increases. For example, in methods using lookup tables, the size and parameters of the lookup table change whenever the number of frequencies and configurations change. Furthermore, system robustness may not automatically adapt, as the phase domain at different frequencies may have different phase noise levels.

[0038] This disclosure presents example systems and methods for performing phase expansion in the complex domain, where vector phase expansion is transformed into a scalar phase expansion process. This approach allows 3D ToF cameras to use any optimized set of frequencies without being restricted to a common integer denominator, thereby achieving adaptive system power efficiency. For the same or even higher number of frequencies, the computational power required to perform scalar expansion is less than that of phase vector-based methods. Scalar expansion methods in the complex domain can be weighted according to effective brightness, thus generating an automatically adaptive system as the system ages or encounters unbalanced phase noise at each frequency. This expansion method can handle any number of frequencies and any type of frequency combination using the same general formula, enabling adaptive, user-friendly ToF cameras that can self-optimize for a given imaging environment. While this disclosure focuses on amplitude continuous wave applications based on modulated light, such as infrared imaging, the techniques disclosed herein are also applicable to radar and other range measurement techniques that rely on phase detection of waveforms at different frequencies to determine distance.

[0039] Figure 3 A flowchart of an example method 300 for phase unfolding using a time-of-flight camera such as digital camera 202 is shown. At 310, method 300 includes illuminating a scene with modulated light (e.g., infrared light) having two or more frequencies having no common integer denominator. This set of two or more frequencies may include integer frequencies and / or fractional frequencies. For example, a modulation controller such as modulation controller 240 may be configured to generate a radio frequency modulated signal. A modulation light transmitter such as modulation light transmitter 230 may be configured to receive the radio frequency modulated signal from the modulation controller and illuminate the scene with the modulated light. As a non-limiting example, the sets of example frequencies (in MHz) may include {151, 167, 197}, {160.123, 168.456, 198.789}, {96.9, 102, 114, 141.3, 2016}, and {68, 2, 1225, 148}.

[0040] At 320, method 300 includes receiving modulated light reflected from objects within the scene at a sensor array. The sensor array may include a subset of sensor elements tuned and / or filtered to receive infrared light at one or more emission frequencies, as described for sensor array 204.

[0041] At 330, method 300 includes processing the received infrared light to determine the envelope phase vector of the modulated light at each frequency. For example, for each frequency of modulated light received at the sensor array, the synthesized capture signal can be expressed as:

[0042] Where k = 1, 2, ... Nk And m = 1, 2, ... N m (Equation 3)

[0043] Where m is the number of frequencies, and where V m，k The modulation frequency f is represented by m The sensor voltage output (e.g., capture signal) is m≥2. The capture intensity can be two-dimensional and has indices (i, j) corresponding to position and distance. CM m It is the modulation frequency f m The captured common-mode signal represents the DC component of the signal. AB m It is the modulation frequency f m The effective brightness is related to the phase signal and is contributed by the effective light emitted from the modulated light emitter. This allows the difference in illumination power at different frequencies to be considered in the downstream equations. The modulation frequency f is represented by m The flight time corresponds to the phase. ψ k This represents the equidistant phase shift step size within the 2π modulus (k >= 3). The phase shift step size can be the same for each frequency condition. N k N represents the total number of phase shift steps at each frequency, where it is assumed that they are the same at every frequency. m This represents the total number of frequencies output by the modulated optical transmitter.

[0044] Although Equation 3 does not include higher harmonics, it can be used to expand the captured phase vector. Equation 3 can be transformed into a normal equation and its accompanying derivation:

[0045]

[0046]

[0047]

[0048]

[0049] The regular equation applies the least squares method, which uses a numerical filter in the complex domain to effectively filter Gaussian noise from the phase vector. By including the effective luminance, amplitude modulation across different frequencies can be considered. When the amplitude is low, the phase error tends to be large, thus including more phase noise. Considering the effective luminance allows for weighting of different frequencies to reduce the error introduced into the overall system. Therefore, the phase expansion becomes robust when the phase error is unbalanced.

[0050] For example, it can be based on each specific frequency f m Solve equations 4a, 4b, and 4c to derive the three system variables:

[0051] CM m=1 / N k ∑ k (V m，k (Equation 5a);

[0052]

[0053]

[0054] Where I(m) = ∑ k [V m，k sin(ψ k )], and represents the imaginary part of the complex value; R(m)=∑ k [V m，k coS(ψ k )], and represents the real part of a complex value; And it represents the wrapped phase in the 2π modulus following the arctangent operation expressed in Equation 5b; t dm Represents each frequency f m Package flight time.

[0055] At 340, method 300 includes determining the unwound phase of each of two or more modulated lights at different frequencies. As used herein, the unwound phase represents the phase lag corresponding to the time of flight of the modulated light. Previous phase unwound methods have been implemented via processing a set of phase vectors. This is accomplished by finding the true time of flight. However, as mentioned above, these methods are inflexible in terms of frequency selection and become increasingly computationally expensive as the number of frequencies used increases. Therefore, optimizing the frequency set using previously used methods is challenging, leading to suboptimal camera operation, which can be energy inefficient.

[0056] Therefore, optionally, at 350, method 300 includes converting the wrapped phase vector into a phase scalar in the complex domain. For example, by converting the phase vector at all frequencies f m In the case of both, further solving equation 5c yields an expansion formula with unified constraints on the complex values ​​R(m) and I(m) in the complex domain:

[0057]

[0058] Where T d Represents a single specific frequency f m0 The flight time of the unfolding (e.g., the highest frequency). Considering this value together with the package flight time given in Equation 5b, we obtain the following relationship:

[0059] 2πf m0 T d =2πM+2πf m0 t dm0(M = 0, 1, 2...M) max (Equation 6a). Substituting the relationship from Equation 6a into Equation 6, we get:

[0060] Where M is the frequency f m0 The number of wrapping cycles of the 2π modulus. From the derivation conditions of equation 4c, it can be deduced that if the integer M makes the function U in equation 7... M When an extreme value (e.g., a minimum) is reached, the value of M is the number of repetitions, 2π, so the expansion process can be assumed to be in progress. This formula produces a phase scalar, which is represented by a single unknown integer (M) constrained in the complex domain, thus transforming the vector phase expansion process into a scalar phase expansion process.

[0061] Therefore, optionally, at 360, method 300 includes determining the expanded phase in the complex domain based on a phase scalar. For example, the value of M can be based on minimizing U. M The value of M is determined by the value of M such that 0 ≤ M ≤ M max For example, equation 7 can be rewritten using equations 5b and 5c:

[0062]

[0063] Equation 8a can be considered equivalent to equations 6 and 7. Constants It can be discarded, but is kept here for completion. For f among them... m =f m0 Under the given conditions, expand the phase expression. Having frequency f m0 One of them is redundant and can be eliminated, resulting in:

[0064]

[0065] If the frequency f m0 If an integer M satisfies equation 8a or 8b, then all other frequencies f m The expanded phase can be expressed as an ideal case or an integer multiple of 2π, assuming an ideal case. In the presence of significant noise, the expanded phase term can be expressed as an integer (N... 0m The residual ΔN of the integer and the integer m To represent, that is: Therefore, equation 8b is now equivalent to:

[0066]

[0067] Where 2πΔN m It is a small amount, therefore cos(2πΔN) m )≈1-0.5·(2πΔNm ) 2 Equation 8c can be simplified to:

[0068]

[0069] Discarding all constant parameters, we get:

[0070]

[0071] Equation 8e can be rewritten as:

[0072] This allows equation 8e or equation 8f to be measured using AB at instant. m Or the ratio of previously measured sums to the average. It can be a constant for each sensor element, regardless of the signal strength under ideal conditions.

[0073] Integer residual ΔN m This can be represented as a rounding operation with the following relationship:

[0074]

[0075] As an example, from 0 to M max The total integer count is equal to the specific frequency f. m0 The package count within the designed unfolding distance (e.g., the highest frequency used by the system) is 2π. The unfolding process can be summarized by the following example: the radio wavelength of the highest radio frequency can be determined. For example, if f m0 =180MHz, then the wavelength is 1.6667m. M can be determined for the package phase vector within the determined maximum unfolding distance. max The value. For example, at the designed deployment distance of 15m, And it can be rounded to the integer 18. M can be set to 0, 1, 2... (M) within equation 6e or 6f. max -1), for example, 0, 1, 2...18, to determine which value of M0 minimizes U. M The unfolding phase of the modulated light at each frequency can then be determined based on at least M. This produces all unfolding phases Ψ, denoted by M0. m Along with the package phase as follows:

[0076] Where m = 1, 2, 3..., (Equation 9)

[0077] Method 300 and the accompanying formula thus enable the use of multiple frequencies, unrestricted by integral relationships between frequencies having a common denominator, thereby generating the flexibility of a switchable and adaptable ToF camera system that allows for cost-effectiveness. Method 300 allows the 3D ToF system to use any frequency configuration determined to achieve accurate depth measurement within environmental range with reduced power consumption. Frequency combinations can be selected based on unfolding distance to improve power efficiency, reduce noise associated with depth jitter, etc. As an example, during a first condition, the scene can be illuminated with modulated light of a first set of three or more frequencies. During a second condition, the scene can be illuminated with modulated light of a second set of three or more frequencies, wherein at least one frequency of the second set differs from any frequency in the first set.

[0078] This expansion method maps the multidimensional phase vector to a scalar with only one unknown integer (M), thereby reducing the computational burden as the frequency of the modulated light increases. Furthermore, system robustness can be determined by the system's effective brightness AB of the received modulated light. m Automatic adjustment or weighting. Equation 8 reveals the robust expansion results under Gaussian noise conditions above the threshold. Some variations of equations 6-8f can be modified, for example:

[0079]

[0080] Equation 10 can effectively determine the unfolding phase, but it is not optimized for significant Gaussian noise conditions. Therefore, the unfolding results may be inaccurate when the phase noise exceeds a threshold. This can be described as the unfolding failure probability or unfolding pass rate under noisy conditions, defined as the following ratio:

[0081] UR = [1 - (Deployment failure count at full deployment distance / Total deployment count at full designed deployment distance)]% (Equation 11)

[0082] Figure 4 Method 400 illustrates an example method for adjusting phase unfolding parameters based on noise conditions. For example, a ToF camera employing a three-frequency system {f1, f2, f3} is used to describe a robust optimization effort to improve the unfolding pass rate under given noise conditions. However, it should be understood that similar methods can be applied to systems using smaller or larger frequency groups.

[0083] At 410, method 400 includes determining a first frequency of light output by the optical emitter. For example, the first frequency of light may be the highest frequency of light output by the optical emitter. The first frequency can be arbitrary, predetermined, or based on operating conditions. For example, the highest frequency may be based on the desired depth resolution and / or desired range of other lower frequencies in the frequency group. As an example, f3 may be set to 198 MHz, but higher or lower frequencies may also be used.

[0084] At 420, method 400 includes determining the maximum unfolding range of the scene to be imaged. The maximum unfolding range can be determined empirically, estimated, predetermined, or arbitrary, and / or can be based on a determined first (e.g., highest) frequency. As an example, the unfolding distance L can be set to 16m, but alternatively, higher or lower distances can be used.

[0085] At 430, method 400 includes scanning one or more other frequencies within the maximum expansion range. For example, if f3 is determined, the desired ranges for f1 and f2 can be determined, and ToF measurements can be performed for each lower frequency with interval resolution while keeping the higher frequencies constant. For example, f1 and f2 can be scanned over the two frequencies in the range of 50-189 MHz, although larger or smaller ranges can be used. The ranges of f1 and f2 do not have to be the same or even overlap. The frequency interval resolution can be any suitable value, such as 1 MHz. In some examples, the lowest frequency can be predetermined, and only intermediate frequencies can be scanned. The upper limit of the available frequencies may be limited by the product of the frequency (f) and the ToF pixel modulation efficiency ME (f), for example, f·ME (f). However, the frequency range scanned can include frequencies greater than this upper limit. This allows for the selection of optimal frequency settings later if the characteristics of the device, system, and / or environment change in the future.

[0086] At 440, method 400 includes adjusting the noise level of each captured image to within a predetermined standard deviation of phase noise. Given a noise perturbation, such as Gaussian noise applied to Equation 3, the noise level can be equalized. For example, Equation 3 indicates that the captured noise level can be adjusted to bring the phase noise within a standard deviation of 9.5 degrees, although any suitable value can be used. This may produce noise simulation results as shown in Equations 5b and 5c under the fully expanded range L.

[0087] At 450, method 400 includes determining the unwound phase of the modulated light for each group of frequencies. For example, unwound phases can be performed in the complex domain based on equations 8e and 9. Each result can then be compared to a result that does not account for noise perturbations. A value can then be assigned to each group of frequencies, for example, using equation 9. This value could indicate, for example, whether there are more comparison results within 360 degrees (+ / - noise) or greater than 360 degrees (+ / - noise) of each other.

[0088] At 460, method 400 includes indicating the frequency group with the highest unfolding pass rate. For example, the unfolding diagram can be drawn using Equation 9, where at a fixed f3 (e.g., 198 MHz), UR = RU(f1, f2).

[0089] Figure 5A This is Figure 500, showing the expanded pass rate at each phase noise standard deviation under 9.5-degree phase noise conditions, scaled from 50-100% as shown on the right side of the figure. Visually, this scaling resolution may be too low to determine the desired frequency configuration.

[0090] Figure 5B Figure 520 is shown, indicating a portion of the unfolding through-rate graph 500. For convenience of frequency selection, this graph is scaled from 99-100%. All visible points on the graph have a high unfolding probability or through-rate, as shown in Equation 9.

[0091] For systems using more than 3 frequencies, optimization of the expanded frequencies can be performed by scanning all frequencies, iteratively from the lower frequency number to the higher frequency number under given noise conditions, using the same process described above, although visualizing the results on a 2D graph may not be feasible.

[0092] Figure 6A and Figures 6B-6C Two examples are shown, illustrating the expanded results of 3-frequency and 5-frequency systems without using rational integers between these frequencies.

[0093] Figure 6A Figure 600 illustrates example phase unfolding results up to 16.67 m at frequencies of f1 = 160.123 MHz, f2 = 168.456 MHz, and f3 = 198.789 MHz. Under phase noise conditions of 1 standard deviation (approximately 9.3 degrees), the unfolding pass rate / probability is approximately 99.46%. Figure 6B Figure 610 shows an example phase unfolding result indicating Figure 600, but under zero noise (ideal) conditions.

[0094] Figure 6CFigure 620 illustrates example phase unfolding results up to 50 m at frequencies of f1 = 96.9 MHz, f2 = 102 MHz, f3 = 114 MHz and f3 = 141.3 MHz, f4 = 201.6 MHz. Under 1 standard deviation phase noise conditions (approximately 15 degrees), the unfolding pass rate / probability is approximately 99.66%. Increasing the number of frequencies results in a larger unfolding distance and greater robustness. Figure 6D Figure 630 shows an example phase unfolding result indicating Figure 620, but under zero noise conditions. Figure 6E Figure 640, showing a close-up of the portion of Figure 620, illustrates a distance of approximately 10 m. Figure 6F Figure 650, showing a close-up portion of Figure 630, illustrates the value for a distance of approximately 10 m under zero-noise conditions.

[0095] In some embodiments, the methods and processes described herein can be attached to a computing system of one or more computing devices. In particular, such methods and processes can be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0096] Figure 7 A non-limiting embodiment of a computing system 700 capable of performing one or more of the methods and processes described above is schematically illustrated. The computing system 700 is shown in a simplified form. The computing system 700 can take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices. The computing system 700 can be an example of a controller 208.

[0097] The computing system 700 includes a logic machine 710 and a memory machine 720. The computing system 700 may optionally include a display subsystem 730, an input subsystem 740, a communication subsystem 750, and / or... Figure 7 Other components not shown.

[0098] The logic machine 710 includes one or more physical devices configured to execute instructions. For example, the logic machine may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, change the state of one or more components, achieve technical effects, or otherwise achieve desired results.

[0099] A logic machine may include one or more processors configured to execute software instructions. Additionally or alternatively, a logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The logic machine's processor may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Individual components of the logic machine may optionally be distributed across two or more independent devices that may be remotely located and / or configured for coordinated processing. Various aspects of the logic machine may be virtualized and executed by remotely accessible, networked computing devices configured with cloud computing capabilities.

[0100] The storage device 720 includes one or more physical devices configured to hold instructions executable by a logic machine to implement the methods and processes described herein. When such methods and processes are implemented, the state of the storage device 720 can be transformed—for example, to hold different data.

[0101] Storage device 720 may include removable and / or built-in devices. Storage device 720 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, magnetic tape drive, MRAM, etc.). Storage device 720 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location-addressable, file-addressable, and / or content-addressable devices.

[0102] It should be understood that the memory 720 includes one or more physical devices. However, aspects of the instructions described herein are alternatively propagated via a communication medium (e.g., electromagnetic signals, optical signals, etc.) that is not maintained by the physical devices for a finite duration.

[0103] The logic unit 710 and the memory unit 720 can be integrated together into one or more hardware logic components. For example, such hardware logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASICs / ASICs), program- and application-specific standard products (PSSPs / ASSPs), system-on-a-chip (SoCs), and complex programmable logic devices (CPLDs).

[0104] The terms "module," "program," and "engine" can be used to describe an aspect of a computing system 700 implemented to perform a specific function. In some cases, a module, program, or engine can be instantiated by executing instructions held in memory 720 via logic machine 710. It should be understood that different modules, programs, and / or engines can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can include individual or grouped executable files, data files, libraries, drivers, scripts, database records, etc.

[0105] When included, the display subsystem 730 can be used to present a visual representation of the data held by the memory 720. This visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data held by the memory, and thus transform the state of the memory, the state of the display subsystem 730 can also be transformed to visually represent changes in the underlying data. The display subsystem 730 may include one or more display devices using virtually any type of technology. Such display devices may be combined with the logic machine 710 and / or the memory 720 in a shared enclosure, or such display devices may be peripheral display devices.

[0106] When included, the input subsystem 740 may include or interface with one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may include or interface with a selected Natural User Input (NUI) component. Such components may be integrated or peripheral, and the translation and / or processing of input actions may be performed on-machine or off-machine. Example NUI components may include a microphone for speech and / or voice recognition; an infrared, color, stereo, and / or depth camera for machine vision and / or gesture recognition; a head tracker, eye tracker, accelerometer, and / or gyroscope for motion detection and / or intent recognition; and an electric field sensing component for assessing brain activity.

[0107] When included, the communication subsystem 750 can be configured to communicatively couple the computing system 700 to one or more other computing devices. The communication subsystem 750 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem may allow the computing system 700 to send and / or receive messages to and / or from other devices via a network such as the Internet.

[0108] In one example, a method for a time-of-flight camera includes illuminating a scene with modulated light having two or more frequencies, the two or more frequencies not having a common integer denominator; receiving the modulated light reflected from objects in the scene at a sensor array; and determining the unfolded phase for each of the two or more modulated lights. In such an example or any other example, additionally or alternatively, determining the unfolded phase for each of the two or more modulated lights includes: processing the received modulated light to determine a wrapping phase vector for each frequency; converting the wrapping phase vector into a phase scalar in a complex domain; and determining the unfolded phase in the complex domain based on the phase scalar. In any or any other example of the foregoing examples, additionally or alternatively, the phase scalar is represented by a single unknown integer M, where M is the number of wrapping periods for a 2π mode at a given modulated light frequency. In any or any other example of the foregoing examples, additionally or alternatively, the method includes determining the M of the wrapping phase vector within a determined maximum unfolding distance. max The value; and based on minimizing U M The value of M is determined by the value of M such that 0 ≤ M ≤ M. max ,in And N k It is the total phase shift step size at each frequency, and ABm is the total phase shift step size at each frequency f. m The effective brightness at that location, and t d，m It is at each frequency f m The time of flight of the package at the specified location. In any or any other of the foregoing examples, additionally or alternatively, the unfolding phase of the modulated light for the frequency is determined at least based on M. In any or any other of the foregoing examples, additionally or alternatively, Gaussian noise is filtered from the phase vector using a digital filter in the complex domain. In any or any other of the foregoing examples, additionally or alternatively, the system robustness is weighted based on the effective brightness of the received modulated light.

[0109] In another example, a time-of-flight camera includes: a modulated light emitter configured to illuminate a scene with modulated light having two or more frequencies, the two or more frequencies not having a common integer denominator; a sensor array configured to receive modulated light reflected from objects within the scene; and a controller configured to: process the received modulated light to determine a wrap-around phase vector for each frequency of the modulated light; and determine an unfolded phase for each of the two or more frequencies of the modulated light. In such an example or any other example, additionally or alternatively, the controller is configured to: convert the wrap-around phase vector into a phase scalar in a complex domain; and determine the unfolded phase in the complex domain based on the phase scalar. In any or any other example of the foregoing examples, additionally or alternatively, the phase scalar is represented by a single unknown integer M, where M is the number of wrap-around periods for the 2π mode of the modulated light at a given frequency. In any or any other example of the foregoing examples, additionally or alternatively, the controller is configured to determine the M of the wrap-around phase vector within a determined maximum unfolded distance. max The value of U; and based on minimizing U M The value of M is determined by the value of M such that 0 ≤ M ≤ M. max ,in And N k It is the total phase shift step size at each frequency, AB m It is at each frequency f m The effective brightness at that location, and t d，m It is at each frequency f m The time of flight of the package at that location. In any or any other of the foregoing examples, additionally or alternatively, the unfolding phase of the modulated light at a specific frequency is at least based on U. M Generate for U M The value of M is determined by the extreme value of the value. In any or any other of the foregoing examples, additionally or alternatively, the controller is configured to filter Gaussian noise from the phase vector using a digital filter in the complex domain. In any or any other of the foregoing examples, additionally or alternatively, the controller is configured to weight the system robustness based on the effective brightness of the received modulated light.

[0110] In yet another example, a method for a time-of-flight camera includes: illuminating a scene with modulated light of three or more frequencies; receiving, at a sensor array, the modulated light reflected from objects within the scene; processing the received modulated light to determine a wrap-around phase vector for each frequency of the modulated light; converting the phase vector into a phase scalar in a complex domain; and determining an unfolded phase in the complex domain based on the phase scalar. In such an example or any other example, additionally or alternatively, the phase scalar is represented by a single unknown integer M, where M is the number of wrap-around periods of the 2π mode of the modulated light for a given frequency. In any or any other example of the foregoing examples, additionally or alternatively, converting the phase vector into a phase scalar in a complex domain includes generating a complex value R(m) = ∑ k [V m，k cos(ψ k )] and I(m)=∑ k [V m，k sin(ψ k The expansion formula for unified constraints in a complex domain: In any or any other of the foregoing examples, additionally or alternatively, the method includes a U-based approach. M Minimize to determine the value of M such that 0 ≤ M ≤ M max ,in In any of the foregoing examples or any other examples, additionally or alternatively, the method includes: during a first condition, illuminating the scene with modulated light of a first group of three or more frequencies; and during a second condition, illuminating the scene with modulated light of a second group of three or more frequencies, wherein at least one frequency in the second group is different from any frequency in the first group. In any of the foregoing examples or any other examples, additionally or alternatively, the three or more frequencies of modulated light are selected based on an optimization process, the optimization process including: determining modulated light of a first transmission frequency; determining a maximum deployment range for the scene; scanning one or more additional frequencies of the three or more frequencies of modulated light over the maximum deployment range; adjusting the noise level of the received modulated light to within a predetermined phase noise standard deviation; determining the deployment phase of the modulated light for each group of frequencies; and indicating the frequency group with the highest deployment pass rate.

[0111] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered limiting, as many variations are possible. The particular procedures or methods described herein may represent one or more of any number of processing strategies. Therefore, the various actions shown and / or described may be performed in the shown and / or described order, in another order, in parallel, or omitted. Similarly, the order of the above processes may be changed.

[0112] The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations disclosed herein, as well as any and all their equivalents.

Claims

1. A time-of-flight camera, comprising: A modulated light emitter is configured to illuminate a scene with modulated light having two or more frequencies, the two or more frequencies not having a common integer denominator; A sensor array is configured to receive the modulated light reflected from objects within the scene; The controller is configured as follows: The received modulated light is processed to determine the wrap-around phase vector for each frequency of the modulated light; The unfolding phase of each modulated light in the two or more modulated lights for the said two or more frequencies shall be determined by at least the following: Transform the wrapped phase vector into a phase scalar in the complex domain; as well as Determine the expanded phase in the complex domain based on the phase scalar; and The robustness of the system is weighted based on the effective brightness of the received modulated light.

2. The time-of-flight camera of claim 1, wherein the phase scalar is represented by a single unknown integer M, where M is the number of wrapping cycles of the 2π mode of the modulated light for a given frequency.

3. The time-of-flight camera according to claim 2, wherein the controller is further configured to: Within the determined maximum unfolding distance, determine the M of the package phase vector. max The value; and Based on the generation of U M The extreme values ​​of the values ​​are used to determine the value of M such that 0 ≤ M ≤ M. max ,in And N k It is the total phase shift step size at each frequency, AB m It is at each frequency f m The effective brightness at that location, and t d,m It is at each frequency f m Package flight time.

4. The time-of-flight camera of claim 3, wherein the unfolding phase of the modulated light for a frequency is based at least on the generation of the phase for the U... M The value of M is determined by the extreme values ​​of the values.

5. The time-of-flight camera according to claim 1, wherein the controller is further configured to: Gaussian noise is filtered from the phase vector using a digital filter in the complex domain.

6. A method for a time-of-flight camera, comprising: Illuminate the scene with modulated light of three or more frequencies; At the sensor array, the modulated light reflected from objects within the scene is received; The received modulated light is processed to determine the wrap-around phase vector for each frequency of the modulated light; Transform the phase vector into a phase scalar in the complex domain; as well as The expanded phase in the complex domain is determined based on the phase scalar; The system robustness is based on a weighted average of the effective brightness of the received modulated light.

7. The method of claim 6, wherein the phase scalar is represented by a single unknown integer M, where M is the number of wrapping cycles of the 2π mode of the modulated light for a given frequency.

8. The method of claim 7, wherein converting the phase vector into a phase scalar in the complex domain comprises generating a complex value R(m) = ∑ k [V m,k cos(ψ k )] and I(m)=∑ k [V m,k sin(ψ k The expansion formula for the unified constraint in the complex domain is as follows:

9. The method according to claim 7, further comprising: Based on U M Minimize the value of M such that 0 ≤ M ≤ M max ,in 10. The method of claim 6, further comprising: During the first condition period, the scene is illuminated with a first set of modulated light of three or more frequencies; as well as During the second condition, the scene is illuminated with modulated light of three or more frequencies from a second group, wherein at least one frequency in the second group is different from any frequency in the first group.

11. The method of claim 6, wherein the modulated light of the three or more frequencies is selected based on an optimization process, the optimization process comprising: Modulated light with a first transmission frequency; Determine the maximum expansion range for the given scenario; One or more additional frequencies of the modulated light at the three or more frequencies are scanned over the maximum spread range; Adjust the noise level of the received modulated light to within a predetermined phase noise standard deviation; Determine the unfolding phase of the modulated light for each frequency group; and Indicates the frequency group with the highest unfolding pass rate.

12. A method for a time-of-flight camera, comprising: Illuminate a scene with modulated light having two or more frequencies, wherein the two or more frequencies do not have a common integer denominator; At the sensor array, the modulated light reflected from objects within the scene is received; as well as The unfolding phase of each modulated light in the two or more modulated lights for the said two or more frequencies shall be determined by at least the following: The received modulated light is processed to determine the wrap-around phase vector for each frequency of the modulated light; Transform the wrapped phase vector into a phase scalar in the complex domain; as well as The expanded phase in the complex domain is determined based on the phase scalar; The system robustness is based on a weighted average of the effective brightness of the received modulated light.

13. The method of claim 12, wherein the phase scalar is represented by a single unknown integer M, where M is the number of wrapping cycles of the 2π mode of the modulated light for a given frequency.

14. The method of claim 13, further comprising: Within the determined maximum unfolding distance, determine the M of the package phase vector. max The value; as well as Based on making U M To minimize the value of M, determine the value of M such that 0 ≤ M ≤ M. max ,in And N k It is the total phase shift step size at each frequency, AB m It is at each frequency f m The effective brightness at that location, and t d,m It is at each frequency f m Package flight time.

15. The method of claim 14, wherein the unfolding phase of the modulated light for a frequency is determined at least based on M.

16. The method of claim 12, wherein Gaussian noise is filtered from the phase vector using a digital filter in the complex domain.

Images