Time-of-flight camera

Abstract

A time-of-flight camera for distance measurement according to the phase measurement principle is proposed, comprising a modulated light source and a time-of-flight sensor with multiple pixels. The sensor has two pixel classes, which are synchronously controlled with different modulation signals and frequencies. A CW pixel class is divided into groups with different phase positions, while a UR pixel class has only one group with one phase position. The modulation frequencies of the two classes are in an integer multiple of each other. During an integration interval, a CW clock sequence is run through, with the illumination synchronized such that light pulses are sent out corresponding to the modulation signals, where either only one pulse or n - 1 pulses are sent out during the CW clock sequence,

Description

[0001] The invention relates to a time-of-flight camera according to claim 1.

[0002] Time-of-flight cameras or time-of-flight camera systems refer in particular to all time-of-flight or 3D-TOF camera systems that obtain time-of-flight information from the phase shift of emitted and received radiation. PMD cameras with photonic mixer detectors (PMDs), such as those described in DE 197 04 496 C2, are particularly suitable as time-of-flight or 3D-TOF cameras.

[0003] DE 10 2012 204 512 A1 relates to a time-of-flight sensor for measuring the phase of modulated light. The sensor comprises several time-of-flight pixels as photonic mixer detectors. These are connected to modulation drivers and at least one phase shifter. The sensor is designed so that it can be operated in at least two areas with different phase measurement modes. A first area is located in the center of the sensor, the second in the edge area. This allows for flexible adjustment of the measurement times and modes, for example, to minimize motion artifacts and increase measurement accuracy. By using multi-phase modes, several phase positions can be recorded simultaneously during an exposure, which reduces spatial resolution but reduces motion artifacts.The invention also includes a method for operating the sensor, in which during an integration time in the first area all pixels are operated with the same phase position and in the second area groups of pixels are operated with different phase positions.

[0004] DE 10 2022 102 992 A1 relates to a time-of-flight distance measuring system, in particular a time-of-flight camera system, comprising an illumination unit for emitting and a time-of-flight sensor for receiving and demodulating modulated light. A modulator generates the modulation signal for the illumination unit and sensor. During distance measurement, the modulation signal is provided with a first modulation frequency as a pseudo-noise signal for selecting a distance range. During a control measurement, the modulation signal is provided as a common-mode signal with a second, lower modulation frequency. This enables the comparison of the amplitudes from both measurements to validate the distance measurement. The system uses pseudo-random binary sequences (PN sequences) such as maximum-sequence sequences (MLS) due to their advantageous autocorrelation properties to enable accurate and noise-resistant measurements.However, when using PN sequences directly, unequal numbers of logical zeros and ones can lead to performance problems. By substituting the bits of the basic PN sequence with sub-bit sequences, a symmetric charge carrier distribution is achieved, making MLS usable for modulating a PMD depth image sensor. The system offers advantages such as multi-target capability and the reduction of mixed phases and stray light. The plausibility of the measurements is verified by an additional measurement at a lower frequency to avoid incorrect measurements.

[0005] US 2020 / 0167942 A1 relates to a method for performing depth measurements with an image sensor. Continuous wave phase measurements and coded modulation measurements are performed for at least one pixel. A mask value is determined from the coded modulation measurement, which is applied to the distance value calculated from the phase measurements to obtain a masked distance value without phase winding ambiguity. This technique improves the accuracy of depth measurements by eliminating ambiguities that can arise from phase windings. The method comprises performing multiple phase measurements with different phase differences and determining the mask value by comparing the coded modulation measurement with a threshold value. The mask value is then applied to the distance values ​​to obtain masked distance values.An image processing system includes a sensor with multiple pixels that generate signals in response to received light, and a control circuit that controls the measurement and application of mask values. This technique can also be used to filter image data, limiting the data to objects within a specified distance range.

[0006] DE 10 2021 113 743 A1 relates to a method and a device for time-of-flight (ToF) detection of a scene. A ToF sensor performs multiple ToF measurements at a first modulation frequency to obtain measured values. Each correlation function of these measurements is periodic and has an increasing amplitude over the distance within the sensor's measuring range. This enables the distance to an object to be determined based on the measured values. The ToF sensor emits modulated light pulses into the scene and detects the reflected light pulses. The sensor's processing circuitry determines the distance based on the measured values. To overcome the ambiguity of the distance measurement, additional ToF measurements can be performed at a second modulation frequency. The correlation functions of the second measurements are also periodic and have an increasing amplitude.The distance is determined by comparing the phase shifts of the first and second measurements. This method reduces glare and saturation problems for nearby objects and improves ToF detection. Modulation codes such as Barker codes or m-sequences are used to control the light pulses and reference signals.

[0007] The object of the invention is to simplify the determination of a unique distance of a time-of-flight camera that operates according to the phase measurement principle.

[0008] The object is achieved by a time-of-flight camera according to claim 1.

[0009] Some of the terms used are explained in more detail below: - Continuous wave or CW measurement refers to an amplitude-modulated continuous light emission, whereby a CW measurement determines a phase difference between the emitted and received modulated light. - Code modulation or CM measurements use coded modulation or so-called pseudo-noise modulation instead of uniform modulation. - Unambiguous range (UR) describes the maximum range for which an unambiguous distance can be determined in a measurement based on the phase measurement principle. For distances greater than the wavelength of the modulation signal, the measurement is no longer unambiguous. The unambiguous range can be increased by performing measurements with several different modulation frequencies. - dToF: direct Time of Flight method, all distance measurement methods that determine a distance directly from the 'time of flight' of an emitted light - iToF: indirect Time of Flight method, all methods which, in contrast to dToF methods, determine indirect parameters dependent on the time of light, such as a phase shift between a transmitted and received modulation signal, to determine the distance.

[0010] The realization of the most accurate and long-range distance measurement possible with iToF systems represents a particular challenge, since the distance noise behaves linearly, whereas the uniqueness region size behaves inversely to the modulation frequency of a continuous wave measurement.

[0011] Typically, this contradiction is resolved by cleverly combining several consecutive individual measurements. For this purpose, a CW measurement is combined with either another CW measurement or a coded modulation measurement. In the first case, the uniqueness range of the combined measurement is increased by a factor of typically 4 to 8 by selecting the uniqueness ranges of the individual measurements according to the Chinese remainder theorem. In the second case, the position of the uniqueness range of the CW measurement is determined by the CM measurement, thus multiplying the size of the combined uniqueness range.

[0012] Both methods have different advantages and disadvantages. A common disadvantage of both methods is the need for additional measurements, which require additional measurement time, readouts, and electrical power. In many cases, this also involves limitations regarding laser safety due to the additional active illumination of the measurement scene.

[0013] The object of the invention is to overcome this disadvantage.

[0014] The problem is solved by the inventive procedure by enabling simultaneous CW and CM measurement.

[0015] The effective uniqueness range of a continuous wave measurement according to the iToF principle can thus be multiplied by additional code modulation measurements while maintaining the distance noise.

[0016] They show: Fig. 1 schematically shows a time-of-flight camera system, Fig. 2 a modulated integration of generated charge carriers, Fig. 3 a cross-section through a PMD time-of-flight pixel with a potential distribution, Fig. 4 Courses of charge integration depending on the phase shift and position, Fig. 5 a relation of the phase shift in an IQ diagram, Fig. 6 a time-of-flight sensor with three groups of time-of-flight pixels that can be operated with different modulation signals Fig. 7 a modulation scheme for a time-of-flight sensor according to Fig. 6, Fig. 8 a time-of-flight sensor with six groups of time-of-flight pixels that can be operated with different modulation signals Fig. 9 a modulation scheme for a time-of-flight sensor according to Fig. 8, Fig. 10 typical autocorrelation functions for an inventive operation, Fig. 11 a modified modulation scheme for a time-of-flight sensor according to Fig. 8, Fig. 12 Extension of the module scheme from Fig. 9 for an additional UR modulation.

[0017] Fig. 1 shows a measuring situation for an optical distance measurement with a time-of-flight camera, as known, for example, from DE 197 04 496 A1.

[0018] The time-of-flight camera system 1 comprises a transmitting unit or an illumination module 10 with an illumination 12 and an associated beam-shaping optics 15 as well as a receiving unit or time-of-flight camera 20 with a receiving optics 25 and a time-of-flight sensor 22.

[0019] The time-of-flight sensor 22 has at least one time-of-flight pixel, preferably also a pixel array, and is designed in particular as a PMD sensor. The receiving optics 25 typically consist of several optical elements to improve the imaging properties. The beam-shaping optics 15 of the transmitting unit 10 can be designed, for example, as a reflector or lens optic. In a very simple embodiment, optical elements can be omitted on both the receiving and transmitting sides.

[0020] The measuring principle of this arrangement is essentially based on the fact that, based on the phase shift of the emitted and received light, the transit time and thus the distance traveled by the received light can be determined. For this purpose, the light source 12 and the transit time sensor 22 are jointly modulated via a modulator 30 with a specific modulation signal M 0 with a base phase position φ 0In the example shown, a phase shifter 35 is also provided between the modulator 30 and the light source 12, with which the base phase φ 0 of the modulation signal M 0 the light source 12 by defined phase positions φ var For typical phase measurements, phase positions of φ var = 0°, 90°, 180°, 270°.

[0021] According to the set modulation signal, the light source 12 transmits an intensity-modulated signal S p1 with the first phase position p1 or p1 = φ 0 + φ var This signal S p1 or the electromagnetic radiation is reflected by an object 40 in the case shown and, due to the distance traveled, arrives with a phase shift Δφ(t L ) with a second phase position p2 = φ 0 + φ var + Δφ(t L ) as received signal S p2to the time-of-flight sensor 22. In the time-of-flight sensor 22, the modulation signal M 0 with the received signal S p2 mixed, whereby the phase shift or the object distance d is determined from the resulting signal.

[0022] Furthermore, the system has a modulation control device 27 which, depending on the measuring task at hand, determines the phase position φ var the modulation signal M 0 changes and / or sets the modulation frequency via a frequency oscillator 38.

[0023] Infrared light-emitting diodes are preferably suitable as the illumination source or light source 12. Of course, other radiation sources in other frequency ranges are also conceivable, in particular, light sources in the visible frequency range.

[0024] The basic principle of phase measurement is shown schematically in Fig. 2. The upper curve shows the time course of the modulation signal M 0 which controls the illumination 12 and the light transit time sensor 22. The light reflected by the object 40 is received as a signal S p2 according to its light travel time t L phase shifted Δφ(t L ) to the time-of-flight sensor 22. The time-of-flight sensor 22 collects the photonically generated charges q over several modulation periods in the phase position of the modulation signal M 0 in a first accumulation gate Ga and in a phase position M shifted by 180° 0 + 180° in a second accumulation gate Gb. From the ratio of the charges qa, qb collected in the first and second gate Ga, Gb, the phase shift Δφ(t L ) and thus determine a distance d of the object.

[0025] Fig. Figure 3 shows a cross-section through a pixel of a photonic mixer detector, such as that known from DE 197 04 496 C2. The modulation photogates Gam, G0, Gbm form the light-sensitive region of a PMD pixel. Depending on the voltage applied to the modulation gates Gam, G0, Gbm, the photonically generated charges q are directed either to one or the other accumulation gate or integration node Ga, Gb. The integration nodes can be configured as a gate or as a diode.

[0026] Fig. 3b shows a potential curve in which the charges q flow towards the first integration node Ga, while the potential according to Fig. 3c, the charge q flows toward the second integration node Gb. The potentials are specified according to the applied modulation signals. Depending on the application, the modulation frequencies are preferably in the range of 1 to 100 MHz. For example, a modulation frequency of 1 MHz results in a period of one microsecond, so the modulation potential changes every 500 nanoseconds.

[0027] In Fig. 3a also shows a readout unit 400, which may already be part of a PMD time-of-flight sensor embodied as CMOS. The integration nodes Ga, Gb, embodied as capacitors or diodes, integrate the photonically generated charges over a plurality of modulation periods. In a known manner, the voltage then applied to the gates Ga, Gb can be tapped off with high impedance, for example, via the readout unit 400. The integration times should preferably be selected such that the time-of-flight sensor or the integration nodes and / or the light-sensitive regions do not saturate for the expected amount of light. The readout paths of the two integration nodes Ga, Gb can also be referred to as A and B channels.

[0028] Fig. 4a and Fig. 4b shows the charge difference Δq = q a - q b / (q a + q b ) depending on the phase shift Δφ(tL ) of the received light signal S p2 with different phase positions. The Fig. 4a shows a curve for an unshifted modulation phase M 0 with a phase position φ var = 0°.

[0029] When the signal S p2 without phase shift i.e. Δφ(t L ) = 0°, for example, if the transmission signal S p1 is directed directly onto the sensor, the phases of the modulation M 0 and from the received signal S p2 identical, so that all generated charge carriers are detected in phase synchronism at the first gate Ga and thus a maximum difference signal with Δq = 1 is present.

[0030] With increasing phase shift, the charge at the first accumulation gate Ga decreases and at the second accumulation gate Gb increases. With a phase shift of Δφ(t L) = 90°, the charge carriers qa, qb are equally distributed at both gates Ga, Gb, and the difference is thus zero, and after a 180° phase shift, it is "-1". As the phase shift continues to increase, the charge at the first gate Ga increases again, resulting in the charge difference rising again, reaching a maximum again at 360° or 0°.

[0031] Mathematically, this is a correlation function of the received signal S p2 with the modulating signal M 0 . q(τ)=∫0τSp2(t−τ)M0(t)dt

[0032] When modulating with a square wave signal, the correlation function is a triangular function, as already shown. When modulating with a sine wave signal, for example, the result would be a cosine function.

[0033] How Fig. 4a shows, a measurement of the phase with a phase position is only possible up to a phase shift Δφ(t L ) ≤ 180° clearly.

[0034] For maximum detection of the phase shift, the IQ (in-phase quadrature) method is known, in which two measurements are carried out with phase positions shifted by 90°, for example with the phase position φ var = 0° and φ var = 90°. The result of a measurement with the phase position φ var = 90° is in Fig. 4b.

[0035] The relationship between these two curves can be shown in a known way, for example for sinusoidal curves in an IQ diagram according to. Fig. 5. As a first approximation, this representation can also be easily applied to the triangular functions shown.

[0036] The phase angle can then be determined in a known way using an arctan function: φ=arctanΔq(90°)Δq(0°)

[0037] For example, to compensate for asymmetry of the sensor, additional phase measurements shifted by 180° can be carried out, so that the phase angle can be determined as follows. ϕ=arctan2Δq(90°)−Δq(270°)Δ(0°)−Δq(180°)

[0038] From the transit time-related phase shift Δφ(t L ) a distance can be determined in a known manner for object distances d that are smaller than half the wavelength λ of the modulation frequency d ≤ λ / 2. d=Δφ(tL)λ2π⋅12

[0039] For distances d > λ / 2 there is usually no possibility to measure the phase shift absolutely, so that the determined phase shift can no longer be clearly assigned to a distance value.

[0040] The core idea of ​​the invention is to design a time-of-flight sensor in such a way that different time-of-flight pixels can be operated with different phase positions and modulation signals in order, for example, to be able to operate a CW modulation and a coded modulation CM simultaneously.

[0041] The invention proposes to realize the CW and CM measurements simultaneously without additional readout of the imager.

[0042] For this purpose, CW and CM measurements are carried out in parallel on different pixels.

[0043] Fig. Figure 6 shows a possible design of a time-of-flight sensor according to the invention with two pixel classes that can be operated with different modulation frequencies, wherein within the pixel classes, groups of pixels with different modulation phase positions can be controlled. In the example shown, a CW pixel class is provided, with CW pixels CW-M0, CW-M90, which are operated with a modulation phase position of 0° and 90°. Furthermore, a UR pixel class is provided, which in the example shown is operated only with a first phase position of 0°. 0° and 90° here are synonymous with a real and imaginary part of the phase measurement.

[0044] For symmetry reasons, it is also advantageous to measure the complementary phase position for the CW pixels and the orthogonal phase position for the UR pixel to fully cover the desired depth range, ie, for example, in a second measurement, the complementary phase position shifted by 180° is measured for the CW pixels and the orthogonal phase position shifted by 90° is measured for the UR pixels.

[0045] Alternatively, it may also be provided, as in Fig. 8, pixels for the complementary and orthogonal phase position are provided on the time-of-flight sensor. In the example according to Fig. 8 therefore provides 6 pixel groups, namely four CW pixels CW-M0, CW-M90, CW-M180, CW-270 and two UR pixels UR-M0, UR-M90

[0046] Fig. Figure 7 shows a preferred modulation scheme for the illumination and a time-of-flight sensor or a respective time-of-flight pixel according to Fig. 6. In the illustrated case, the CW modulation signal of the CW pixels has a modulation frequency 4 times higher than the UR modulation signal, whereby the UR modulation signal has no special coding.

[0047] Starting with the 4 times higher modulation frequency, the CW modulation signal passes through 4 CW periods during one period of the UR modulation signal. These 4 CW periods form a CW clock sequence.

[0048] According to the invention, the length of the CW and UR periods or CW and UR modulation frequencies is varied by an integer factor n f differentiate. With f1 = n * f2.

[0049] Accordingly, n CW periods are passed through during one UR period. The n CW periods form a CW clock sequence. The illumination is now designed such that only one light pulse is emitted synchronously with the CW modulation signal during one CW clock sequence. Alternatively, it can also be provided that n-1 light pulses are emitted during the CW clock sequence. In both cases, the light pulses preferably have a duration of at least 25% of the CW period length.

[0050] In the example shown, only one light pulse is emitted within n=4 periods of the CW modulation signal or within one period of the UR modulation signal.

[0051] In a second measurement, the complementary and orthogonal phase positions are preferably measured. The complete set of measurements is described in Fig. 9, which can be measured either in two measurements with the time of flight sensor according to Fig. 6 or in a measurement with a time-of-flight sensor according to Fig. 8 can be carried out.

[0052] In Fig. Figure 10 shows the resulting autocorrelation functions (AKF) of the measurement according to the invention, and it is immediately apparent that the combination of the information enables an unambiguous distance determination across four CW uniqueness ranges. For this purpose, the position determination within the uniqueness range of the CW pixels is combined with the determination of the position of the uniqueness range from the information of the UR pixels.

[0053] The UR pixels register the arrival of the individual emitted light pulse and the position of the CW uniqueness area can be clearly determined from the light travel time.

[0054] However, the disadvantage compared to a conventional CW measurement is the optical duty cycle of the CW measurement, which is reduced by a factor of four, which leads to a lower amplitude of the correlation signal per integration time.

[0055] The lighting scheme according to Fig. 11 avoids the strong reduction of the optical signal by not emitting a single light pulse, but by omitting a single light pulse or emitting n-1 light pulses.

[0056] In this scheme, the location of the CW uniqueness region is detected by the absence of a pulse. Consequently, the signal amplitude of the CW pixels is enhanced, while the noise level of the UR pixels is increased by the additional noise of the received light.

[0057] However, since only a simple A / B evaluation is necessary for the UR measurement, this does not represent a significant limitation. For the sake of simplicity, Fig. 11 only one CW modulation signal CW-M0 is recorded as an example, of course the other phase positions are recorded accordingly and included in the evaluation.

[0058] A possible implementation of the modulation scheme from Fig. For example, 11 on a pixel array can combine 3 different pixels that are read out in 2 frames: • one CW pixel for CW measurement with 0° and 180° phase difference • one CW pixel for CW measurement with 90° and 270° phase difference • one UR pixel for A- and B-CM measurement (0°, 90°)

[0059] Depending on the application and the resolution of the iToF module, it may be useful to distribute a few UR pixels across the imager and assume the same UR area for the areas surrounding these pixels. Adaptive selection of the density and position of the UR pixels depending on the scene being viewed is also conceivable.

[0060] The scheme from Fig. 11 can be expanded by adding additional UR pixels with a lower modulation frequency to increase the combined uniqueness range. With each additional UR measurement, the uniqueness range can be doubled.

[0061] Fig. Figure 12 shows such an extension, where 3 UR signals are used to distinguish 8 CW unambiguous regions.

[0062] The described scheme can be easily adapted in various ways for CW measurements with a different number of CW phase measurements. For example, for a 3-tap measurement with 3 CM measurements, the following implementation in 3 frames is possible: • one CW pixel for CW measurement with 0°, 120° and 240° phase difference • one UR pixel for A, B and C CM measurement

[0063] Furthermore, the UR modulation can be implemented as so-called coded modulation, particularly in the form of a Gray code. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 197 04 496 C2 [0002, 0025] DE 10 2012 204 512 A1

[0003] DE 10 2022 102 992 A1

[0004] US 2020 / 0167942 A1

[0005] DE 10 2021 113 743 A1

[0006] DE 197 04 496 A1

[0017]

Claims

A time-of-flight camera designed for distance measurement according to a phase measurement principle, - with an illumination for emitting modulated light, - and with a time-of-flight sensor with multiple time-of-flight pixels for receiving the emitted modulated light reflected from a scene, - wherein the time-of-flight sensor has two pixel classes (CW, UR), namely a CW and a UR pixel class, which can be synchronously controlled with different CW and UR modulation signals (CW-M, UR-M) and with different CW and UR modulation frequencies (f1, f2), - wherein the CW pixel class (CW) has at least two CW pixel groups (CW-M0, CW-M90) that are operated with different phase positions (M0, M90, M180, M270) of the CW modulation frequency (f1), and UR pixel class (UR) has at least one first UR pixel group (UR-M0) which is operated with a first phase position (M0) of the UR modulation frequency (f2),- wherein the CW modulation frequency (f1) is an integer multiple n of the UR modulation frequency (f2) with f1 = n * f2, so that the CW periods repeat n times during a UR period, - wherein n CW periods form a CW clock sequence, - wherein at least one CW clock sequence and at least one UR period are run through in parallel during an integration interval, - and wherein the illumination is designed such that either only 1 light pulse or n - 1 light pulses are emitted synchronously with the CW modulation signal within a CW clock sequence, - wherein the light of the illumination is detected simultaneously by both pixel classes (CW, UR) in each integration interval, and a relative distance value is determined from the measurement of the CW pixel group and a uniqueness range is determined from the measurement of the UR pixel group, - wherein then from the relative distance value and the An absolute distance value is determined in the uniqueness range. Time-of-flight camera according to claim 1, wherein the CW pixel class (CW) has four CW pixel groups with different phase positions of the CW modulation frequency (f1), and wherein the UR pixel class (UR) has two UR pixel groups with different phase lengths of the UR modulation frequency (f2). A time-of-flight camera according to claim 1, wherein the time-of-flight sensor has two CW pixel groups and only one UR pixel group, wherein the time-of-flight camera is designed such that in a first measuring phase the first CW pixel group (CW-M0, CW-M90) is operated with a 0° phase position, the second CW pixel group with a 90° phase position and the one UR pixel group (UR-M0) with a 0° phase position, wherein in a second measuring phase the first and second CW pixel groups (CW-M0, CW-M90) are operated with the complementary phase position of 180° and 270° and the first UR pixel group (UR-M0) is operated with the orthogonal phase position of 90°. Time-of-flight camera according to one of the preceding claims, in which, in addition to the first UR pixel group, one or more further UR pixel groups are provided, which can be operated with modulation signals of lower frequency or longer wavelength in relation to the first UR modulation signal in order to increase the uniqueness range.

Images