Time-of-flight measurement using linear inverse function

By using linear inverse methods and polynomial function modeling, combined with two-point temperature calibration and spatial correction, the problem of depth measurement error caused by the temperature dependence of the light source and image sensor is solved, and high-precision depth sensing over a wide temperature range is achieved.

CN113678024BActive Publication Date: 2024-11-12MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080028952.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-04-15
Filing Date
2020-03-26
Publication Date
2024-11-12
Estimated Expiration
2040-03-26

AI Technical Summary

Technical Problem

Existing time-of-flight depth sensing systems, when operating at high peak laser power, suffer from phase and amplitude sensitivity due to the temperature dependence of the light source and image sensor, leading to depth measurement errors, especially performance degradation in the depth image data processing pipeline after phase unwrapping.

Method used

The temperature dependence of the light source and image sensor is calibrated using a linear inverse method. By obtaining the complex phase of each sample at each light source modulation frequency, the temperature dependence is modeled using a polynomial function. Combined with two-point temperature calibration and spatial correction, the influence of temperature on depth measurement is mitigated.

Benefits of technology

Millimeter-level depth measurement accuracy was achieved over a wide temperature range, reducing errors caused by temperature variations and improving the precision of the depth sensing system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113678024B_ABST
    Figure CN113678024B_ABST
Patent Text Reader

Abstract

An example provides a time-of-flight depth imaging system configured to modulate light emitted from a light source to illuminate an environment with the modulated light, and for each of one or more modulation frequencies, integrate an image at each of a plurality of phase steps, and sense the temperature of the light source and / or the image sensor via one or more temperature sensors to obtain a measured temperature. The instructions are also executable to, and for each of one or more pixels of the image sensor: determine a complex phasor based on the measured temperature using an inverse linear function for each modulation frequency, determine a phase shift between light emitted from the light source and light from the light source that is reflected back by the environment based on the complex phasor, and output a depth value for the pixel based on the phase shift.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Time-of-flight depth sensors can use a phase shift method to measure distance. In such an example, a periodically modulated laser beam is emitted toward a target object, the reflected light is received at the image sensor, and for each pixel of the image sensor, the phase of the received light is compared with the phase of the emitted light. The depth (distance) from each pixel of the time-of-flight sensor to the target object can then be determined based on the phase shift. Summary of the invention

[0002] This summary is provided to introduce a selection of concepts that are further described in the detailed description below in a simplified form. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.

[0003] Examples are disclosed that relate to mitigating the effects of temperature on components of a time-of-flight depth sensing system. One example provides a time-of-flight depth imaging system that includes a modulated time-of-flight image sensor, a light source, and one or more temperature sensors configured to sense the temperature of the light source and / or the image sensor. The example also includes a processor and a storage device that includes instructions executable by the processor to: modulate light emitted from the light source to illuminate an environment with the modulated light, and for each of one or more modulation frequencies, integrate the image at each of a plurality of phase steps, and sense the temperature of the light source and / or the image sensor via one or more temperature sensors to obtain a measured temperature. The instructions may also be executable to: for each of one or more pixels of the image sensor, using a linear inverse function for each modulation frequency, determine a complex phasor based on the measured temperature, determine a phase offset between light emitted from the light source and light from the light source that is reflected back by the environment based on the complex phasor, and output a depth value for the pixel based on the phase offset. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 A block diagram of an example time-of-flight depth imaging system is shown.

[0005] Figure 2 An example Argand plot is shown representing the phase and amplitude of a sample captured by a time-of-flight depth camera with a light source that has no temperature dependence.

[0006] Figure 3 An example Argand plot is shown representing the phase and amplitude of a sample captured by time-of-flight depth with a light source that exhibits temperature dependence.

[0007] Figure 4 A graph of phase shift as a function of light source temperature is shown for nine example phase samples.

[0008] Figure 5 A graph of the amplitude as a function of the source temperature is shown for nine example phase samples.

[0009] Figure 6 A series of polynomial curves fit to data representing phase error as a function of light source temperature for selected phase samples across a population of depth sensors are shown.

[0010] Figure 7 The residual phase error versus temperature for selected phase samples on a population of depth sensors when determining depth using a linear inverse method with a single temperature point calibration is shown.

[0011] Figure 8 The residual phase error versus temperature for selected phase samples across a population of depth sensors when determining depth using the linear inverse method with a two-point temperature calibration is shown.

[0012] Fig. 9 A graph depicting depth accuracy as a function of ambient temperature using the linear inverse method and a two-point temperature calibration is shown.

[0013] Fig.10 A block diagram depicting the flow of data within a time-of-flight depth sensing system is shown.

[0014] Fig.11 An example method of operating a time-of-flight system is shown.

[0015] Fig.12 A block diagram of an example computing system is shown. DETAILED DESCRIPTION

[0016] Amplitude modulated continuous wave (AMCW) time of flight (ToF) depth sensing systems work by illuminating a scene with modulated light and measuring the time it takes for the light to travel to an object in the scene and back to the sensor. The sensor works by correlating the backscattered light with a reference modulated signal, enabling the phase shift induced in the modulating waveform to be measured and the time it took the light to travel to be inferred, thereby calculating the distance to the object. Since phase wraparound causes ambiguity in the measured distance, multiple modulation frequencies are often used to increase the maximum unambiguous range. Detailed descriptions of such systems are available in the literature.

[0017] AMCW ToF sensing systems can operate at high peak laser powers to overcome the amount of background ambient light in some environments, such as outdoor spaces or indoor spaces with bright lighting. To further reduce noise from background light, it may be desirable to emit the total laser energy of an image frame in a short period of time to limit the amount of ambient light that is integrated. Using a high-power laser light source helps to shorten the integration time. However, the phase and amplitude of the light emitted by such a high-power light source may be sensitive to the temperature of the light source. In a time-of-flight sensing system that calculates the phase based on a discrete Fourier transform (DFT) bin of phase measurement, this may result in centimeter-level errors in the sensed depth value. For example, when a laser light source heats up within a frame during a series of phase steps, the electro-optical properties of the first phase step, such as peak optical power output, threshold current, and mean phase, may be different from those of the second or third phase step. Ambient temperature may further affect such electro-optical properties. The image sensor used to integrate the reflected light may also have a spatial temperature dependency that varies as a function of the pixel position on the image sensor, further complicating accurate depth measurement.

[0018] Various methods can be used to mitigate the errors caused by this temperature sensitivity. One possible solution is to maintain the light source at a substantially constant temperature during depth sensing. However, for many practical systems, the hardware for maintaining temperature may be too bulky and / or expensive. As another possible solution, a time-of-flight depth sensing system can use a pixel-by-pixel linear correction after conversion to depth values. However, such an approach may be applied late in the depth image data processing pipeline after phase unwrapping, resulting in performance degradation. In addition, this pixel-by-pixel linear correction may not address the spatial temperature dependence of the image sensor.

[0019] Thus, examples are disclosed involving calibrating such temperature dependence of a light source and image sensor in a time-of-flight imaging system for accurate depth sensing. In short, rather than using a Fourier transform-based approach to process depth data, the disclosed examples use a linear inverse approach to determine the complex phasor for each sample acquired at each light source modulation frequency. As explained in more detail below, using the disclosed example linear inverse approach, the phase and amplitude can be arbitrary compared to the Fourier transform-based approach. Depending on the calibration temperature used, the disclosed examples can help achieve millimeter-level accuracy over a wide temperature range (e.g., from 10° to 80° Celsius).

[0020] Figure 1An example time-of-flight depth sensing system 10 is shown configured to image a target 12. The depth camera 10 includes a modulated light source 14, an imaging array 16, and a controller 18. The depth camera may also include various other components, such as an imaging lens 20 and / or a wavelength filter, which may be disposed in front of the array.

[0021] Modulated light source 14 is configured to irradiate target 12 with amplitude modulated light of sinusoidal, rectangular or any other shape. Modulated light source may include one or more suitable laser light sources, such as LEDs, VCSELs or edge emitting lasers.

[0022] The imaging array 16 comprises an array of depth sensing pixels. The imaging array is configured to receive pulsed or modulated probe light reflected from the target 12. Each pixel of the array is configured to present an output signal that depends on the distance from the depth camera to the core of the target 12 imaged onto that pixel.

[0023] continue Figure 1 , the controller 18 includes a logic subsystem 62 and a storage subsystem 64. The controller 18 includes logic for providing a modulated drive signal to the light source 14 and the modulated imaging array 16 to synchronize the operation of these components. The imaging array 16 works by correlating the incident light with a reference waveform (which may vary for each measurement) and storing the associated value within each pixel. The value stored for a particular illumination and sensor modulation waveform combination may be referred to as an integral. The controller is also configured to read the output signal from each pixel of the imaging array to enable calculation of a depth map of the target 12. In some embodiments, repeated depth imaging may be used to assemble a time-resolved series of depth maps (depth video).

[0024] As mentioned above, the power of the light emitted from the camera's light source is modulated by a periodic waveform. The light source may be modulated at multiple frequencies per depth image frame, and depth values ​​may be determined at multiple phase steps for each modulation frequency. Figure 2 An example Argand plot 200 is shown, which represents the phase and amplitude of samples captured by a time-of-flight depth camera with a light source that has no temperature dependence. In this example, the three phase samples are sampled with the same intrinsic correlation waveform shape, but are offset by 120 degrees by changing the phase relationship between the light source modulation waveform and the sensor reference waveform. From the perspective of the fundamental frequency of the correlation waveform, each phase sample can be considered as a projection 202 of the real signal 204 onto a sampling vector 206, which represents the effective phase and amplitude of the phase sample. As mentioned above, existing methods calculate the phase of each frequency by calculating the discrete Fourier transform (DFT) bin corresponding to the fundamental frequency of the three phase samples, as represented by the following equation:

[0025]

[0026] in is a complex depth measure, and is the phase step size. The complex depth measurement encodes the amplitude of the modulated illumination backscattered from the scene and measured by the sensor in the modulus, and the time-of-flight induced phase shift in the complex parameter. The phase shift for each frequency can be calibrated so that the depth can be calculated without unknown phase shifts due to the optoelectronic properties of the illumination and the sensor.

[0027] exist Figure 2 In the example of , it is assumed that each vector is equidistant, at 0, 120, and 240 degrees, representing the phase angle of the emitted light, and each vector is of equal length, representing the amount of light emitted for that particular phase sample. However, in reality, due to the temperature dependence of the laser light source mentioned above, the vectors may actually be offset in angle and vary in length, because the phase angle and amplitude of the emitted light are functions of temperature. Figure 3 An example Argand plot 300 is shown representing the phase and amplitude of a sample captured by a time-of-flight depth camera with a temperature-dependent light source, and an example vector representing the phase and amplitude of a three-phase sample affected by temperature changes is depicted. Imaging sensors also have temperature dependence, but it is usually dominated by the light source.

[0028] The terms "effective phase" and "effective amplitude" may be used to refer to the phase and amplitude of the light source as it would be measured by the sensor itself under ideal conditions for each individual phase step. In some embodiments, the effective phase and amplitude for each phase step is determined by performing a phase scan, where the phase relationship of the sensor reference and illumination modulation waveforms is advanced each frame. Once enough samples have been collected over at least 360 degrees of phase step advancement, a DFT may be performed on each integral over the collected frames, and the effective phase and amplitude for each individual phase step may be determined. In contrast, standard methods perform DFTs on different phase steps of the same frequency within a frame, rather than performing a DFT on each integral (corresponding to a particular phase step) over multiple frames. The disclosed phase scanning method allows for measurement of changes in phase and amplitude behavior as a function of the phase step order, which may not be possible with standard methods.

[0029] Variations in the effective phase and amplitude as a function of temperature can cause the negative fundamental frequency of the associated waveform to alias onto the fundamental frequency, resulting in two-cycle linearity errors. Existing solutions model the effective phase and / or amplitude of each complex depth measurement as a whole and correct for any fixed phase offset or scale, but do not model the individual phase steps and their resulting impact on range linearity.

[0030] Therefore, instead of calculating DFT bins, the examples disclosed herein use a linear inverse function to calculate the complex phasor based on the estimated phase and amplitude values. The disclosed examples allow for arbitrary variations in phase and amplitude while correcting for errors caused by temperature variations.

[0031] In order to apply the complex phasor method disclosed herein to real-time time-of-flight depth measurement, the phase and amplitude of the light sources as a function of temperature can be modeled before runtime, for example using data obtained from a light source cluster. An example modeling process is described in more detail below. The resulting temperature model can then be applied at runtime to help calculate the complex phasor for each phase step, thereby correcting for the effect of light source temperature on depth measurement.

[0032] Figure 4 A graph 400 of phase shift for each sample as a function of temperature is shown for nine example samples of an example light source, where samples 1, 2, and 3 correspond to phase samples at a first modulation frequency, samples 4, 5, and 6 correspond to phase samples at a second modulation frequency, and samples 7, 8, and 9 correspond to phase samples at a third modulation frequency. Figure 5 A graph 500 of amplitude versus temperature for the same samples 1-9 is shown. As can be seen, for different modulation frequencies, the phase and amplitude each vary as a function of the light source temperature. In addition, each phase sample within a modulation frequency also varies based on the light source temperature. Therefore, temperature calibration methods that apply the same calibration to each phase sample and / or each frequency may encounter linearity or phase unwrapping errors. In contrast, the linear inverse examples disclosed herein can avoid such errors.

[0033] Figure 4 and Figure 5 The curves of can each be modeled by a polynomial, such as a cubic polynomial or other suitable function, to form a predetermined temperature calibration model. To determine the temperature calibration model, temperature dependency data for each phase sample and each modulation frequency can be collected over a sample population of the light source using each integrated phase scan, and each temperature dependency curve for each phase sample at each frequency can be fit to a polynomial function or other suitable function. Figure 6 A graph depicting multiple polynomial functions fitted to temperature data for a first phase sample at a selected modulation frequency given a known target distance is shown. Similar data can be obtained and fitted to other phase samples and other modulation frequencies. As can be seen from this figure, the phase error and variation between samples across the population increases with increasing temperature. From such data, a mean population temperature model (previously referenced predetermined temperature calibration model) can be generated from multiple fitted polynomial functions to calibrate the depth camera and correct the temperature dependence data of the light source when calculating depth from the image sensor.

[0034] Figure 7 The cross-sectional view of the first phase step at the selected modulation frequency using a single point temperature calibration with linear inverse phase and amplitude calculation is shown. Figure 6 Figure 2. Residual phase error versus light source temperature on an example population of depth sensors. The example shown here was calibrated at approximately 30°C. As can be seen, the phase error is proportional to Figure 6 The phase error is much smaller than that of the . However, higher temperatures result in slightly higher variances compared to lower temperatures. In addition, as mentioned above, the plotted Figure 6 The polynomial function of diverges as temperature increases. Thus, when using a single-point calibration, the mean temperature model may not accurately fit the depth sensor population over the entire operating temperature range to the degree desired by some applications.

[0035] Thus, in some examples, the mean temperature model can be adjusted on a per-unit basis using a two-point temperature calibration and known population variations. In such examples, instead of using a universal population calibration at a fixed temperature, the fitted polynomial function can be further constrained by a second calibration temperature. To this end, the polynomial function (or other suitable function) can be distorted using measurements made at the second calibration temperature to match measurements at two temperatures. As an example, correction basis vectors determined from principal component analysis can be applied to the polynomial coefficients to account for variations in the polynomial or other function for the population. Figure 8 The residual phase error vs. temperature after applying an example two-point temperature calibration on an example population of depth sensors is shown. The second calibration temperature used for these examples is approximately 60° C. As can be seen, the error of the two-point temperature calibration can be within millimeters over a relatively wide temperature range.

[0036] After modeling the temperature dependence of the phase and amplitude of the light source population, the model can be applied during runtime using a linear inverse method.An example parameterization applicable to both phase and amplitude in this scheme is as follows.

[0037]

[0038] This function assumes that there are two temperature sensors in the depth sensing system, one of which is configured to sense the temperature of the light source (t laser ), and one is configured to sense the temperature of the image sensor (t sensor). In other examples, the term for image sensor temperature may be omitted. Furthermore, in some examples, additional spatial corrections may be applied in the depth data processing pipeline to account for the effect of sensor temperature on the complex phasor calculations, as described below. In examples where the depth sensing system includes more than one light source temperature sensor, the above parameterization may also include separate coefficients for each of the multiple light source temperatures. Although the above example equation (2) is a polynomial function, it will be appreciated that any other suitable function may be used.

[0039] The temperature sensor may be read at any frequency, for example for each integration / phase step, or once per modulation frequency, or once per frame, or at any other rate. In some cases, it may be read only once during operation.

[0040] Continuing, for each frequency, the complex phasor c can be calculated via a linear inverse function. An example calculation using the Moore-Penrose pseudo-inverse function is as follows.

[0041] c=[1,i]R(H T H) -1 H T (3)

[0042] in,

[0043] And among them, α k =f k , amplitude(t laser , t sensor ), (5)

[0044] θ k =f k,phase (t laser , t sensor ), (6)

[0045] is the original capture, and is the rotation matrix encoding the fixed spatial phase correction. As shown, the complex phasor calculation can include all phase steps for that frequency (three in this example).

[0046] In some embodiments, the phase relationship between the sensor reference and the illumination modulation waveform for each integration varies from frame to frame. In this case, a corresponding offset is applied to the phase model for each phase step to correct for the variation. This allows a corrected distance measurement to be produced despite the change in phase relationship. In some embodiments, the system can perform a phase scan continuously, with the phase relationship of each integration advancing per frame; as a result, systematic biases due to aliasing of harmonics of the correlated waveforms can be reduced. In some embodiments, this is combined with time averaging techniques.

[0047] Furthermore, in some embodiments, the phase steps for each frequency are not equally spaced around 360 degrees (as opposed to the standard approach which uses equally spaced phase steps). In some embodiments, this is combined with time averaging to reduce the negative impact of any related waveform harmonic aliasing on range linearity.

[0048] In some embodiments, one or more cameras have phase scans performed at multiple temperatures to determine the effective phase and amplitude for each integration or phase step, collecting for each camera data similar to or equivalent to Figure 4 and Figure 5 The data of the camera temperature data is provided to provide a corpus of camera temperature data. In some embodiments, the corpus temperature model is created by fitting a polynomial curve to the combined phase and amplitude data. In various embodiments, a polynomial curve is fitted to the phase / amplitude versus temperature data for each camera at each integration.

[0049] In addition, in some embodiments, a single-point temperature calibration for a particular camera is performed by performing a phase scan of all modulation frequencies using a target of known properties (such as a flat wall) at a known or controlled temperature. Another potential target configuration is a retroreflector or mirror at a known distance with a diffuser above the imaging lens to simulate imaging an object at a constant radial distance. The corpus temperature model for the effective phase is adjusted by applying an offset or mathematical transformation so that the model for each integral intersects the true value at the known or measured calibration temperature. In the case of a polynomial model, some embodiments may include: adjusting the constant term of the polynomial so that this condition holds. This adjusted model is then used during operation of the camera to calculate the effective phase for each integral / phase step. The same or similar transformation is performed on the corpus temperature model for the effective amplitude so that the effective amplitude measured at the calibration temperature intersects the adjusted model for each integral. In the case of a polynomial model, some embodiments may include scaling the polynomial so that it intersects at the calibration temperature.

[0050] In some embodiments, additional calibrations are performed at different temperatures. The adjusted corpus temperature model for effective phase and amplitude is mathematically transformed in such a way that the adjusted model intersects all measurements of effective phase and amplitude. In some embodiments using two temperature calibrations, this is achieved by adding linear correction factors to the model so that the adjusted model intersects the first and second effective phase and amplitude measurements. In other embodiments, this is achieved by creating orthogonal basis vectors on the coefficients of the polynomial model and performing a weighted addition on the adjusted corpus temperature model so that the adjusted model intersects all measurements of effective phase and amplitude.

[0051] In some embodiments, orthogonal basis vectors for the effective phase are generated by transforming a polynomial model fitted to the corpus temperature data in the corpus for each integral of each camera so that the effective phase for each integral of each camera in the corpus intersects zero at the calibration temperature of the particular camera to be calibrated. In some embodiments, this is achieved by adding a constant offset to the fitted polynomial model. Assuming that the calibration temperature effective phase is fixed to zero, a principal component analysis is performed to generate a first principal component of the variation in the corpus for the effective phase of each integral. When an additional temperature calibration is performed at a second temperature, the basis vectors corresponding to the first principal component are used to correct the adjusted effective phase model by weighted addition so that the adjusted model intersects the first measurement and the second measurement. In another embodiment, a third temperature calibration step is performed and the basis vectors corresponding to the second principal component are used to adjust the effective phase model so that it intersects all temperature calibration measurements of the effective phase. In another embodiment, this is extended to an arbitrary number of temperature measurements.

[0052] Additionally, in some embodiments, for each camera to be calibrated using phase scanning, multiple calibrations are performed at different temperatures. The model is fit directly to the effective amplitude and phase measured over temperature. This is used during runtime to perform temperature correction. In some embodiments, this can be a linear fit over the measured laser temperature for each integrated phase and amplitude. In another embodiment, the fitting operation can use the statistics of a corpus of camera temperature data as a statistical prior and solve a maximum a posteriori estimation problem to determine the optimal model to apply at runtime. In some embodiments, the statistical prior can be a multivariate Gaussian distribution describing the coefficients of a polynomial that is fit to the effective phase and effective amplitude of each integration over the corpus of camera temperature data. In some embodiments, the same (or similar) approach is applied to the effective amplitude.

[0053] In some embodiments, the fitted model may not exactly intersect the measured effective phase and amplitude.

[0054] Furthermore, in some embodiments, a fixed spatial phase correction for each pixel is calculated from a known target and used to calculate the phase correction applied at runtime after calibration (referred to as R in equation (3) above).

[0055] It should be understood that other mathematically similar or equivalent methods may exist within the scope of the present disclosure, including where the calibration temperature is stored and the corpus temperature model is used to estimate the change in effective phase and amplitude relative to the effective phase and amplitude at calibration without creating a modified temperature calibration model for each camera. In other embodiments, the corpus temperature model is used to adjust the spatial phase calibration so that it is represented as if it were at a specified temperature, such as zero degrees Celsius.

[0056] Fig. 9 Shows the application of Figure 8 Example real-world performance of a time-of-flight depth sensing system following the linear inverse method of two-point temperature calibration illustrated in FIG. As shown, an error within a range of approximately ±2 mm can be achieved over a range of temperatures.

[0057] As mentioned above, image sensor performance may also exhibit some temperature dependence. Even for an ideal light source maintained at a constant temperature, changes in sensor temperature may cause spatial sensor performance changes, thereby affecting complex phasor measurements. For example, at each phase step, the image may be integrated by modulating the image sensor using a delay locked loop (DLL) in the image sensor, which intentionally introduces a phase delay horizontally across the pixels of the image sensor. Each pixel in the image sensor has a sampled phasor whose phase may vary. Therefore, the spatial variation in the image sensor may be a function of the sensor signal being delayed from pixel to pixel horizontally (left to right relative to the pixel in the sensor) due to the DLL and vertically (top to bottom) due to the (multiple) pixel clock drivers in combination with transmission lines affecting the clock lines in the imaging array. These properties may change with changes in temperature, resulting in depth measurement errors.

[0058] Thus, in addition to temperature correction based on light source temperature, spatial temperature correction based on sensor temperature may also be applied. In some examples, a horizontal correction vector may be applied to each row of pixels in the image sensor, and a vertical correction vector may be applied to each column of pixels. The horizontal correction vector and the vertical correction vector may be determined based on population data associated with phase error across the sensor as a function of temperature. For example, the vectors may be calculated as follows.

[0059] Horizontal correction vector = (t sensor -t calibration )h c

[0060] Vertical correction vector = (t sensor -t calibration )v c

[0061] The basis vector h used for the horizontal correction vector c is a linear function of the number of columns and is used as the basis vector v for the vertical correction vector c is a quartic function of the number of rows. The disclosed examples are not limited to basis vectors that can be modeled as polynomials; in some embodiments, the entire basis vectors are stored explicitly or a different type of model is used.

[0062] In some embodiments, the spatial temperature basis vectors are generated by measuring the effective phase of one or more cameras at two different temperatures, performing a subtraction of the effective phase at one temperature from the effective phase at the other temperature and averaging across all cameras. The spatial correction factor per degree Celsius is determined by dividing the effective phase difference by the difference between the two temperatures. In other embodiments, more temperatures are used. In some embodiments, the horizontal correction vector h is determined by a weighted least squares linear fit of the raw effective phase spatial correction factor per degree Celsius values ​​in the horizontal and vertical directions. c and the vertical correction vector v c , where the weights are the mean complex moduli of the measured signals. In other embodiments, any method known to those skilled in the art may be used to fit the correction vector to the available data.

[0063] Fig.10 A block diagram of an example depth engine 1000 within a time-of-flight depth camera that can implement the above-described example temperature correction is shown. As described above, raw data 1002 and laser and sensor temperatures 1004 are provided as inputs to a temperature correction deprojection block 1006, which uses a linear inverse-based approach to determine a complex phasor while compensating for light source temperature dependency (and potential image sensor temperature dependency). Continuing, at block 1008, phase and effective brightness (AB) (e.g., amplitude) are determined from the complex phasor. Next, as described above, spatial correction for the temperature dependency of the image sensor can be performed at 1010 using input from the image sensor temperature sensor at block 1012.

[0064] Next, a phase unwrapping block 1014 takes the determined phase measurements and performs phase unwrapping, which helps use multiple phase samples at each modulation frequency to disambiguate between possible distances. Performing spatial correction before phase unwrapping can help prevent de-aliasing errors or linearity errors that may be introduced in the phase unwrapping stage, but in some examples, spatial correction can occur after phase unwrapping. At 1018, the phase and amplitude measurements can then be converted into depth, xyz measurements, and / or reflectivity, which can then be output from the depth engine at 1020. Optionally, filters can be applied at any stage of the pipeline, which can include processing filters, image filters, box filters, and / or other suitable filters. Subsets of these outputs or additional outputs can be provided, such as the sum of the amplitudes of the individual modulation frequencies measured by the sensor.

[0065] Fig.11 An example method 1100 for correcting time-of-flight depth measurements is shown. The method 1100 includes, at 1102, modulating light emitted from a light source to illuminate an environment with the modulated light. The method 1100 also includes, at 1104, integrating an image at each of a plurality of phase steps for each of one or more modulation frequencies, and at 1105, sensing a temperature of the light source and / or the image sensor via one or more temperature sensors to obtain a measured temperature.

[0066] For each image integrated at each phase step, at 1106, and for each pixel of one or more pixels of the image sensor, at 1110, the method 1100 includes, at 1112, determining a complex phasor based on the temperature of the light source and / or the image sensor using a linear inverse function. At 1114, the complex phasor may be determined based on a predetermined temperature calibration model that relates the light source phase and amplitude to the temperature of the light source. The predetermined temperature calibration model may be based on the phase and amplitude behavior of a device population as a function of the light source temperature determined during factory calibration / manufacturing. In some examples, the predetermined temperature calibration model may be based on calibration at two or more different temperature points, while in other examples, the calibration model may be based on a single calibration temperature point. Next, for each pixel, at 1116, a phase offset between the light emitted from the light source and the light reflected back from the light source by the environment is determined based on the complex phasor. As described above, determining the phase offset based on the complex phasor using a linear inverse function may allow the use of arbitrary phase steps and frequencies. In some examples, unequally spaced phase steps may be deliberately used and changed over time to help improve accuracy.

[0067] The method 1100 also includes, at 1118, correcting the determined phase offset to account for spatially dependent errors of the image sensor. This can be accomplished, for example, by applying a horizontal correction vector to one or more rows of pixels of the image sensor and applying a vertical correction vector to one or more columns of pixels of the image sensor at 1120. The method 1100 then includes, at 1122, outputting a depth value for each pixel. Thus, the resulting depth value accounts for temperature dependencies of the light source and the image sensor, which may be more accurate than methods that use per-pixel linear corrections to determine depth values.

[0068] Fig.12 A non-limiting embodiment of a computing system 1200 that can perform one or more of the above methods and processes is schematically shown. The computing system 1200 is shown in a simplified form. The computing system 1200 can take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices. For example, the computing system 1200 can represent a depth camera 10.

[0069] The computing system 1200 includes a logic subsystem 1202 and a storage subsystem 1204. The computing system 1200 may optionally include a display subsystem 1206, an input subsystem 1208, a communication subsystem 1210, and / or Fig.12 Other components not shown.

[0070] Logic subsystem 1202 includes one or more physical devices configured to execute instructions. For example, a logic machine may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, implement a technical effect, or otherwise achieve a desired result.

[0071] The logic subsystem 1202 may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic subsystem 1202 may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processor of the logic subsystem 1202 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential processing, parallel processing, and / or distributed processing. The various components of the logic subsystem 1202 may optionally be distributed among two or more separate devices, which may be remotely located and / or configured for coordinated processing. Various aspects of the logic subsystem 1202 may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0072] Storage subsystem 1204 includes one or more physical devices configured to store instructions executable by logic subsystem 1202 to implement the methods and processes described herein. When such methods and processes are implemented, the state of the storage machine may be transformed—for example, to store different data.

[0073] The storage subsystem 1204 may include removable and / or built-in devices. The storage subsystem 1204 may include optical storage (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor storage (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic storage (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), etc. The storage subsystem 1204 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices.

[0074] It should be appreciated that storage subsystem 1204 includes one or more physical devices. However, aspects of the instructions described herein may alternatively be propagated by a communication medium (eg, electromagnetic signals, optical signals, etc.) that is not retained by a physical device for a finite duration.

[0075] Aspects of the logic subsystem 1202 and the storage subsystem 1204 may be integrated together into one or more hardware logic components. Such hardware logic components may include field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0076] The term "engine" may be used to describe an aspect of the computing system 1200 that is implemented to perform a particular function. In some cases, an engine may be instantiated via the logic subsystem 1202 executing instructions held by the storage subsystem 1204. It should be understood that different engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The term "engine" may encompass a single or a group of executable files, data files, libraries, drivers, scripts, database records, etc.

[0077] When included, the display subsystem 1206 can be used to present a visual representation of the data maintained by the storage subsystem 1204. The visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored by the storage machine, thereby converting the state of the storage machine, the state of the display subsystem 1206 can also be converted to visually represent the changes in the underlying data. The display subsystem 1206 may include one or more display devices utilizing almost any type of technology. Such a display device can be combined with the logic subsystem 1202 and / or the storage subsystem 1204 in a shared shell, or such a display device can be a peripheral display device.

[0078] When included, the input subsystem 1208 may include or interface with one or more user input devices, such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the conversion and / or processing of input actions may be processed on-board or off-board. Examples of NUI components may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity.

[0079] When included, the communication subsystem 1210 can be configured to communicatively couple the computing system 1200 with one or more other computing devices. The communication subsystem 1210 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem can be configured to communicate via a wireless telephone network or a wired or wireless local or wide area network. In some embodiments, the communication subsystem can allow the computing system 1200 to send and / or receive messages to and / or from other devices via a network such as the Internet.

[0080] Another example provides a time-of-flight depth imaging system, which includes a modulated time-of-flight image sensor, a light source, one or more temperature sensors configured to sense the temperature of the light source and / or the image sensor, a processor, and a storage device including instructions executable by the processor to: modulate light emitted from the light source to illuminate an environment with the modulated light, for each of one or more modulation frequencies, integrate an image at each phase step of a plurality of phase steps, sense the temperature of the light source and / or the image sensor via one or more temperature sensors to obtain a measured temperature, and for each of one or more pixels of the image sensor, using an inverse linear function for each modulation frequency, determine a complex phasor based on the measured temperature, determine a phase shift between light emitted from the light source and light from the light source that is reflected back by the environment based on the complex phasor, and output a depth value for the pixel based on the phase shift. The complex phasor may be additionally or alternatively determined based on a predetermined temperature calibration model that associates the effective light source phase and amplitude sensed by the image sensor for each individual phase step with the temperature sensed by the light source and / or the image sensor. A predetermined temperature calibration model for an individual camera may additionally or alternatively be determined based on calibration at two or more temperatures. The predetermined temperature calibration model may additionally or alternatively be determined based on principal component analysis of a population of time-of-flight depth imaging devices. The predetermined temperature calibration model for the effective phase for each phase step and the predetermined temperature calibration model for the effective amplitude for each phase step may additionally or alternatively be polynomials. The predetermined temperature model may additionally or alternatively be adjusted for each device based on measurements of the effective phase and / or effective amplitude for each individual phase step using phase scanning. The spacing between phase steps may additionally or alternatively be non-uniform over 360 degrees. Additionally or alternatively, the phase relationship between the sensor reference waveform and the illumination modulation waveform for each integration may change between temporally adjacent frames. The instructions may additionally or alternatively also be executed to: perform spatial temperature correction for each frequency. In the case where the temperature sensor is a first temperature sensor, the time-of-flight depth imaging system may additionally or alternatively include a second temperature sensor configured to sense the temperature of the image sensor, and wherein the instructions are also executable to: correct the determined phase offset to take into account the spatially dependent error of the sensor by applying a horizontal correction vector to one or more rows of pixels of the image sensor and applying a vertical correction vector to one or more columns of pixels of the image sensor, the horizontal correction vector and the vertical correction vector being based on the temperature of the image sensor.

[0081] On a time-of-flight depth imaging system including an image sensor, a light source, and a temperature sensor configured to sense the temperature of the light source and / or the image sensor, another example provides a method, the method comprising: modulating light emitted from the light source to illuminate an environment with the modulated light, integrating an image at each phase step in a plurality of phase steps for each modulation frequency in one or more modulation frequencies, and for each image integrated at each phase step, sensing the temperature of the light source and / or the image sensor via a temperature sensor, and for each pixel in one or more pixels of the image sensor, using a linear inverse function for each modulation frequency, determining a complex phasor based on the sensed temperature, determining a phase shift between light emitted from the light source and light from the light source that is reflected back by the environment based on the complex phasor, and outputting a depth value for the pixel. Determining the complex phasor may additionally or alternatively include: determining the complex phasor based on a predetermined temperature calibration model that associates the effective phase and amplitude of the light source with the temperature of the light source. The predetermined temperature calibration model may additionally or alternatively be determined based on calibration at two or more temperatures. The predetermined temperature calibration model may additionally or alternatively be a polynomial. Additionally or alternatively, the phase relationship between the sensor reference waveform and the illumination modulation waveform for each integration may be varied between temporally adjacent frames. The method may additionally or alternatively include, for each image integrated at each phase step, correcting the determined phase offset to account for spatially dependent errors of the image sensor.

[0082] Another example provides a time-of-flight depth imaging system, which includes a modulated time-of-flight image sensor, a light source, a first temperature sensor configured to sense the temperature of the light source, a second temperature sensor configured to sense the temperature of the image sensor, a processor and a storage device, the storage device including instructions executable by the processor to: modulate light emitted from the light source to illuminate an environment with the modulated light, for each of one or more modulation frequencies, integrate an image at each of a plurality of phase steps, for each image integrated at each phase step, sense the temperature of the light source via the first temperature sensor, sense the temperature of the image sensor via the second temperature sensor, for each of one or more pixels of the image sensor, determine a complex phasor based on the temperature of the light source using an inverse linear function, and determine a phase offset between light emitted from the light source and light reflected from the light source by the environment based on the complex phasor, correct the determined phase offset to take into account a spatially dependent error of the image sensor by applying a horizontal correction vector to one or more rows of pixels of the image sensor and a vertical correction vector to one or more columns of pixels of the image sensor, the horizontal correction vector and the vertical correction vector being based on the temperature of the image sensor, and output a depth value for each pixel. The horizontal correction vector and the vertical correction vector may each additionally or alternatively be based on a phase error as a function of the temperature of the image sensor. The horizontal correction vector may additionally or alternatively include a linear function of the number of columns that is applied to each row of pixels in the image sensor. The vertical correction vector may additionally or alternatively include a quartic function that is a function of the number of rows that is applied to each column of pixels in the image sensor.

[0083] It should be understood that the configuration and / or method described herein is exemplary in nature, and these specific embodiments or examples should not be considered restrictive, because many variations are possible. The specific routines or methods described herein can represent one or more of any number of processing strategies. Thus, the various actions illustrated and / or described can be performed in the order illustrated and / or described, performed in other orders, performed in parallel, or omitted. Likewise, the order of the above process can be changed.

[0084] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents.

Claims

1. A time-of-flight depth imaging system, comprising: modulated time-of-flight image sensor; light source; one or more temperature sensors configured to sense the temperature of the light source and / or image sensor; Processor; and A storage device including instructions executable by the processor to modulating light emitted from the light source to illuminate an environment with the modulated light, integrating an image at each of a plurality of phase steps for each of one or more modulation frequencies and sensing the temperature of the light source and / or image sensor via the one or more temperature sensors to obtain a measured temperature, and For each of the one or more pixels of the image sensor, Determining the complex phasor based on the measured temperature using a linear inverse function for each modulation frequency, determining a phase shift between the light emitted from the light source and the light from the light source reflected by the environment based on the complex phasor, and A depth value for the pixel is output based on the phase offset.

2. A time-of-flight depth imaging system according to claim 1, wherein the complex phase quantity is determined based on a predetermined temperature calibration model, which associates the effective light source phase and amplitude sensed by the image sensor for each individual phase step with the temperature sensed by the light source and / or the image sensor. 3 . The time-of-flight depth imaging system of claim 2 , wherein the predetermined temperature calibration model for an individual camera is determined based on calibration at two or more temperatures. 4 . The time-of-flight depth imaging system of claim 3 , wherein the predetermined temperature calibration model is determined based on a principal component analysis of a population of time-of-flight depth imaging devices.

5. The time-of-flight depth imaging system of claim 2, wherein the predetermined temperature calibration model for the effective phase of each phase step and the predetermined temperature calibration model for the effective amplitude of each phase step are polynomials.

6. The time-of-flight depth imaging system of claim 2, wherein for each device, the predetermined temperature model is adjusted using phase scanning, based on measurements of the effective phase and / or effective amplitude for each individual phase step.

7. The time-of-flight depth imaging system of claim 2, wherein the spacing between the phase steps is non-uniform over 360 degrees.

8. The time-of-flight depth imaging system of claim 2, wherein the phase relationship between the sensor reference waveform and the illumination modulation waveform for each integration varies between temporally adjacent frames.

9. The time-of-flight depth imaging system of claim 1, wherein the instructions are further executable to perform spatial temperature correction for each frequency.

10. A time-of-flight depth imaging system according to claim 9, wherein the temperature sensor is a first temperature sensor configured to sense the temperature of the light source, and further includes a second temperature sensor configured to sense the temperature of the image sensor, and wherein the instructions are also executable to: correct the determined phase offset by applying a horizontal correction vector to one or more rows of pixels of the image sensor and applying a vertical correction vector to one or more columns of pixels of the image sensor to take into account the spatially dependent error of the image sensor, the horizontal correction vector and the vertical correction vector being based on the temperature of the image sensor.

11. A method, applied to a time-of-flight depth imaging system comprising an image sensor, a light source, and a temperature sensor configured to sense the temperature of the light source and / or the image sensor, comprising: modulating light emitted from the light source to illuminate an environment with the modulated light; integrating the image at each of a plurality of phase steps for each of the one or more modulation frequencies; as well as For each image integrated at each phase step, sensing the temperature of the light source and / or the image sensor via the temperature sensor, and For each of the one or more pixels of the image sensor, determining a complex phasor based on the sensed temperature using an inverse linear function for each modulation frequency, determining a phase shift between the light emitted from the light source and the light from the light source reflected by the environment based on the complex phasor, and Output the depth value for the pixel.

12. The method of claim 11, wherein determining the complex phasor comprises: The complex phasor is determined based on a predetermined temperature calibration model that relates light source effective phase and amplitude to the temperature of the light source.

13. The method of claim 12, wherein the predetermined temperature calibration model is determined based on calibration at two or more temperatures. The method of claim 12 , wherein the predetermined temperature calibration model is a polynomial.

15. The method of claim 11, wherein the phase relationship between the sensor reference waveform and the illumination modulation waveform for each integration is varied between temporally adjacent frames.

16. The method according to claim 11, further comprising: For each image integrated at each phase step, the determined phase offset is corrected to account for spatially dependent errors of the image sensor.

17. A time-of-flight depth imaging system, comprising: modulated time-of-flight image sensor; light source; a first temperature sensor configured to sense a temperature of the light source; a second temperature sensor configured to sense a temperature of the image sensor; processor; and a storage device comprising instructions executable by the processor to: modulating light emitted from the light source to illuminate an environment with the modulated light, integrating the image at each of a plurality of phase steps for each of the one or more modulation frequencies, For each image integrated at each phase step, sensing the temperature of the light source via the first temperature sensor, sensing a temperature of the image sensor via the second temperature sensor, For each of the one or more pixels of the image sensor, determining a complex phasor based on the temperature of the light source using an inverse linear function, and determining a phase offset between the light emitted from the light source and light from the light source reflected by the environment based on the complex phasor, correcting the determined phase offset to account for spatially dependent errors of the image sensor by applying a horizontal correction vector to one or more rows of pixels of the image sensor and a vertical correction vector to one or more columns of pixels of the image sensor, the horizontal correction vector and the vertical correction vector being based on the temperature of the image sensor, and Outputs the depth value for each pixel.

18. The time-of-flight depth imaging system of claim 17, wherein the horizontal correction vector and the vertical correction vector are both based on a phase error that is a function of a temperature of the image sensor.

19. The time-of-flight depth imaging system of claim 17, wherein the horizontal correction vector comprises a linear function of the number of columns, the linear function being applied to each row of pixels in the image sensor.

20. The time-of-flight depth imaging system of claim 17, wherein the vertical correction vector comprises a quartic function that is a function of the number of rows applied to each column of pixels in the image sensor.