Light time-of-flight camera

Abstract

The invention relates to a time-of-flight camera designed for measuring the distance to objects. The core of the camera consists of a time-of-flight sensor containing at least one time-of-flight pixel. This pixel is part of a system that emits modulated light to measure distances to objects. The camera uses the phase detection principle to perform the measurements. For the precise determination of the distance (d w For a portion of the light-time-of-flight pixels of an object, at least two separate measurements (A1, A2, ...) are performed. These measurements use different modulation frequencies (f1, f2, ...), each with its own uniqueness ranges (UR1, UR2, ...). In each of these measurements (A i ) a residual distance (d i = mod(d, UR i )) determined. The remaining distances (d i) from the at least two measurements are first expressed in the form of residual distance tuples (d1, d2, ...) a sub-area (M) or a corresponding sub-area index M of an extended total distance curve (UR) extd ). The final assignment of the sub-area indices M takes into account a 'nearest neighbor filter' and / or a temporal filter of the initially assigned sub-area indices (M).

Description

[0001] The invention relates to a time-of-flight camera according to the preamble of claim 1.

[0002] Time-of-flight cameras are intended to include, in particular, cameras that obtain time-of-flight information from the phase shift of emitted and received radiation. PMD cameras with photonic mixer detectors (PMDs) are particularly suitable as time-of-flight or 3D TOF cameras, as described in DE 197 04 496 A1.

[0003] To extend the uniqueness range of such a camera, multiple modulation frequencies are preferably used. For example, various approaches are described in DE 10 2013 214 677 B3, DE 10 2013 207 647 A1, and DE 10 2018 104 668 A1.

[0004] The aim of the inventions is to improve the accuracy of multi-frequency methods.

[0005] The object is achieved by the camera according to the invention according to claim 1.

[0006] Advantageously, a time-of-flight camera is provided for distance determination, comprising a time-of-flight sensor with at least one time-of-flight pixel, with an illumination for emitting a modulated light, wherein the time-of-flight camera is designed for distance measurements according to a phase measurement principle and is designed such that for at least some of the light travel time pixels for determining a distance of an object, at least two measurements are carried out, each with different modulation frequencies and thus different uniqueness ranges, wherein a residual distance is measured in each measurement, wherein, based on these residual distances from the at least two measurements, the residual distances are initially assigned to a sub-area or a corresponding sub-area index M of a total distance curve of a modulo diagram, wherein a final assignment of the sub-area indices M is carried out depending on a 'nearest neighbor filtering' and / or a temporal filtering of the initially assigned sub-area indices M, wherein the distance is determined by projecting the remaining distances onto the sub-area with the value of the finally assigned sub-area index M.

[0007] This procedure has the advantage that, before a possibly complex calculation of a distance, an assignment to a sub-area of ​​an overall distance curve is first ensured, which simplifies the subsequent distance calculations and contributes to the reduction of distance noise.

[0008] They show: Fig. 1 schematically shows the basic principle of photonic detection, Fig. 2 a modulated integration of the generated charge carriers, Fig. 3 a distance measurement with a wavelength, Fig. 4 a distance measurement with two different wavelengths, Fig. 5 an inventive procedure, Fig. 6 remaining distances in a modulo diagram, Fig. 7 a nearest neighbor filtering.

[0009] In the following description of the preferred embodiments, like reference numerals designate like or comparable components.

[0010] Fig. 1 shows a measuring situation for an optical distance measurement with a time-of-flight camera, as known, for example, from DE 197 04 496 A1.

[0011] The time-of-flight camera system 1 comprises a transmitting unit or an illumination module 10 with an illumination 12 and an associated beam-forming optics 15 as well as a receiving unit or time-of-flight camera 20 with a receiving optics 25 and a time-of-flight sensor 22.

[0012] The time-of-flight sensor 22 has at least one time-of-flight pixel, preferably also a pixel array, and is designed in particular as a PMD sensor. The receiving optics 25 typically consist of several optical elements to improve the imaging properties. The beam-shaping optics 15 of the transmitting unit 10 can be designed, for example, as a reflector or lens optic. In a very simple embodiment, optical elements can be omitted on both the receiving and transmitting sides.

[0013] The measuring principle of this arrangement is essentially based on the fact that, based on the phase shift of the emitted and received light, the transit time and thus the distance traveled by the received light can be determined. For this purpose, the light source 12 and the transit time sensor 22 are jointly modulated via a modulator 30 with a specific modulation signal M owith a base phase position φ0. In the example shown, a phase shifter 35 is also provided between the modulator 30 and the light source 12, with which the base phase φ0 of the modulation signal M0 of the light source 12 can be shifted by defined phase positions φ var For typical phase measurements, phase positions of φ var = 0°, 90°, 180°, 270°.

[0014] According to the set modulation signal, the light source 12 sends an intensity-modulated signal S p1 with the first phase position p1 or p1 = φ0 + φ var This signal S p1 or the electromagnetic radiation is reflected by an object 40 in the case shown and, due to the distance travelled, arrives with a phase shift Δφ(t L ) with a second phase position p2 = φ0 + φ var + Δφ(t L ) as received signal S p2to the time-of-flight sensor 22. In the time-of-flight sensor 22, the modulation signal M o with the received signal S p2 mixed, whereby the phase shift or the object distance d is determined from the resulting signal.

[0015] To improve measurement accuracy and / or expand the unambiguous range, it is advantageous to perform the time-of-flight measurements at different modulation frequencies. For this purpose, the modulator 30 is connected to a modulation control unit 38, which can preferably specify modulation frequencies within a predetermined frequency spectrum.

[0016] The modulator 30 could, for example, be designed as a frequency synthesizer, controlled for the respective measurement task via the modulation control unit 38. Switching between quartz oscillators with fixed frequencies is also conceivable.

[0017] Furthermore, the receiving unit 20 is connected to an evaluation unit 27. The evaluation unit 27 can optionally also be a component of the receiving unit 20 and in particular also part of the time-of-flight sensor 22. The task of the evaluation unit 27 is to determine and / or evaluate phase shifts based on the received signals in relation to the modulation frequency. The mixing of the received light beams with the modulation frequency preferably takes place in the time-of-flight sensor 22 or PMD sensor. Furthermore, the modulation control unit 38 can also be a component of the evaluation unit 27. In particular, it can also be provided that the evaluation unit 27 completely or partially assumes the function of the modulation control unit 38.

[0018] Infrared light-emitting diodes are preferably suitable as the illumination source or light source 12. Of course, other radiation sources in other frequency ranges are also conceivable, in particular, light sources in the visible frequency range are also considered.

[0019] The basic principle of phase measurement is shown schematically in Fig. 2. The upper curve shows the time course of the modulation signal M0, which controls the illumination 12 and the time-of-flight sensor 22. The light reflected by the object 40 arrives as the received signal S p2 according to its light travel time t L phase shifted Δφ(t L) to the time-of-flight sensor 22. The time-of-flight sensor 22 collects the photonically generated charges q over several modulation periods in the phase position of the modulation signal M0 in a first accumulation gate Ga and in a phase position M0 + 180° shifted by 180° in a second accumulation gate Gb. From the difference Δq of the charges qa, qb collected in the first and second gate Ga, Gb, the phase shift Δφ(t L ) and thus determine a distance d of the object.

[0020] For more precise phase determination and to cover the full 360° angular range, a second measurement is preferably performed with a modulation signal shifted by 90°. In this case, either the sensor or the illumination is phase-shifted by this angle.

[0021] The phase angle can then be determined in a known manner using an arctan function or arctan2 function: φ=arctanΔq(90°)Δq(0°)

[0022] For example, to compensate for asymmetry of the sensor, additional phase measurements shifted by 180° can be carried out, so that the phase angle can be determined as follows. φ=arctanΔq(90°)−Δq(270°)Δ(0°)−Δq(180°)

[0023] From the Fig. 2 shown runtime-related phase shift Δφ(t L ) can be calculated for object distances d that are smaller than half the wavelength λ of the modulation frequency d ≤ λ / 2 a distance can be determined in a known manner. d=Δφ(tL)λ2π⋅12

[0024] For distances d > λ / 2 there is usually no possibility to measure the phase shift absolutely, so that the determined phase shift can no longer be clearly assigned to a distance value.

[0025] Fig. 3 shows an example in which the object 40 has a distance d from the transmitter 10 of d=2λ+R2 , whereby of course the total distance travelled to the receiver 20 is twice as large, namely D = 2d = 4λ + R

[0026] To increase the uniqueness range, as in Fig. 4, is intended to determine an object distance d with at least two modulation frequencies or modulation wavelengths. For the sake of simplicity, Fig. 4 shows the total distance D between transmitter 10 and receiver 20. Within the common uniqueness range of the two wavelengths λ1, λ2, which is typically spanned by the lowest common multiple of the wavelengths λ1, λ2, the following distance equation applies: D=2d=n1λ1+R1=n2λ2+R2 with Ri=Dmodλi=φi(fi,D)λi2π where the relative phase shift φ, which depends on the modulation frequency and the object distance, i (f i ,D) applies: φi(fi,D)≡D⋅fic⋅2π mod 2π=Dλi⋅2π mod 2π

[0027] The relative phase shift (φ i (f i ,D) is thus a measure of the remaining piece R in the distance measurement i . If the relative phase shift φ i (f i ,D) into a distance value D i (f i ,D), the remaining piece R corresponds i exactly this distance value D i with R i ,D i · ∈ [0,λ i ]. Ri=Di=2di=φiλi2π

[0028] For the distance determination, two phase shifts φ recorded for different modulation frequencies f1, f2 can now be used. 1 / 2 (f 1 / 2 ,D) a solution for the distance equation presented above can be found.

[0029] In principle, to extend the uniqueness range of indirect TOF camera systems, multiple measurements at different frequencies can be combined. With such multi-frequency methods, the region of the extended uniqueness range in which the measurement point is located is usually first determined, and then the exact distance is calculated. Due to noise or faulty measurements, this initial process step and rough localization can be inaccurate, leading to large distance errors.

[0030] According to the invention, the determination of the position in the extended uniqueness range is not only carried out pixel by pixel, but information from temporally or spatially neighboring regions is incorporated before an accurate distance image is generated.

[0031] As previously described, TOF camera systems measure distances in a scene by determining the time of flight of light from a light source to the object and back to the camera. The measured time of flight is directly proportional to the distance to be determined. Instead of determining the actual time of flight, indirect TOF systems measure the phase shift between a light source with a specific frequency f mod modulated outgoing light signal and the received signal. The phase shift φ is proportional to the propagation time and thus also to the measured distance d. However, due to the periodicity of the phase, a measured phase shift can correspond to several distances. The unambiguous range (UR) is therefore limited to the distance up to half the wavelength of the transmitted signal. d=φ2π⋅UR with UR=clight2fmod, where c lightthe speed of light. An effective and well-known method for extending the uniqueness range is to combine measurements of different frequencies. This then encompasses half the wavelength of the frequency from the greatest common divisor of all measured frequencies. URextd=clight2⋅gcd(f1,f2,...fn)

[0032] The measured distances are only available as remaining distances (d1,..., d n ), where d1 = mod(d w , UR1),..., d n = mod(d w , UR n ).

[0033] The actual distance d w of an object can be calculated as a combination of the measured remaining distances (d1, ..., d n ) and an integer multiple of the individual UR specific to the respective modulation frequency. For an ideal measurement without errors or noise, there is a vector S of integer values ​​(m1, m2, ..., m n ) for which applies: dw=d1+m1⋅UR1=d2+m2⋅UR2=…=dn+mn⋅URn

[0034] The determination of the integer values ​​m1, ..., m n can be understood as a rough localization of the area of ​​the extended UR in which the measuring point is located. Real measurements (d̃ i ) are subject to deviations and noise F i , so that d˜1=d1+F1+m1UR1 d˜2=d2+F2+m2UR2 ⋯d˜n=dn+Fn+mn+URn

[0035] This means that the vector S = (m1, m2 ... m n ) such that d̃1 ≈ d̃2 ≈ ... ≈ d̃ n . The actual distance can then be calculated from the (weighted) individual measurements (d̃1, ..., d̃ n ) can be determined.

[0036] If the noise of the individual measurements or the deviations are too large, a “wrong” vector S = (m1, m2 ... m n ), the error to the true distance is much larger than the original noise.

[0037] The following procedure is intended to prevent these errors or to correct them subsequently: If S is calculated for each pixel at each measurement time, S(x, y, t) can be understood as a temporal sequence of images in which it is recorded in which area of ​​the extended uniqueness region each pixel lies at the current time.

[0038] By filtering this data, individual errors caused by noise can be corrected. The filtering can be - spatial (local, over x,y), - temporal (time, over t) - or a combination of the two domains.

[0039] Apart from distance edges at the edge of objects and finer structures that represent larger distance jumps in the measured scene, it can be assumed that pixels that are close to each other in space and time lie in the same area of ​​the extended uniqueness region. An effective reduction of the distance values ​​incorrectly assigned in the uniqueness region can therefore be achieved by low-pass filtering S(x, y, t). A simple or weighted moving average, for example, would be conceivable. However, median filtering of the data appears more promising for various reasons. On the one hand, edges and structures are preserved and only objects that are smaller than the filter size are changed. On the other hand, no information that was not previously contained in the image is generated, since median filtering ensures that the pixel to be filtered is assigned to a vector S onto which other nearby pixels are also projected.

[0040] Each of these approaches is facilitated by the fact that each vector S can be uniquely represented by a numerical value M, which is suitable for filtering and, ideally, is linearly related to the final distance. Especially for median filtering, it is advantageous to use the smallest possible number of integer values. One possibility is described here as an example:

[0041] In Fig. 5 shows an example of a measurement with two frequencies: f1=15 MHz, UR1=10 m f2=12 MHz, UR2=12.5m enables an extended uniqueness range of UR extd = 50 m. This results in eight unique vectors (S0 = (0,0), S1 = (1,0), S2 = (1, 1), ..., S7 = (4, 3)), each describing a sub-area.

[0042] If n is the number of measurements and the final distance d is already calculated, it is obvious to define M as M=∑i=1nmi with mi=|dURi| For the above example, M can take the integer values ​​0 to 7, which allows for efficient median filtering. The corrected distance can then be easily calculated as d=di+mi+URi with mi from SM

[0043] Fig. 6 shows the facts according to Fig. 5 in a so-called modulo diagram. The total distance curve (UR extd ) depending on distance pairs (d i (f i ), d j (f j )) or remaining distance pairs (d i , d j ) of the selected modulation frequencies f i , f j . The total distance curve extends in the modulo diagram into several sub-areas M. Furthermore, the Fig. 6 several measured distance pairs or remaining distance pairs. In the example shown, the measured distance pairs marked with filled points are located near a sub-area with the sub-area index M = 0 and are assigned to this sub-area. The distance pairs with the unfilled points are located near the sub-areas M with the indices M = 7 and M = 2 and would typically be assigned to these sub-areas.

[0044] In order to avoid such potential misallocations, as already described, it is intended to filter the indices of the initially assigned sub-areas temporally and / or spatially and to carry out the final sub-area allocation depending on this filtering.

[0045] Fig.Figure 7 shows a 'nearest neighbor filtering', in which the middle pixel is corrected or assigned from the initially assigned sub-area index M = 7 to M = 0, taking into account the neighboring pixels. Of course, the next but one or more neighboring pixels can also be included here.

[0046] After filtering the sub-area indices, for each measured distance pair d i , d j or for each remaining distance pair, for example by projecting the measuring points onto the associated sub-area of ​​the total distance curve, a 'true' distance d w be determined.

[0047] The term distance pair is intended to include higher dimensioned n-tuples, especially 3-tuples. List of reference symbols 10 lighting module 12 Lighting 22 Light transit time sensor 27 Evaluation unit 30 Modulator 35 phase shifters, lighting phase shifters 38 Modulation control unit Δφ(t L ) runtime-related phase shift φ var Phase position φ0 base phase M0 modulation signal p1 first phase p2 second phase Sp1 transmit signal with first phase Sp2 Receive signal with second phase λ wavelength D Total path length D ij determined distance values d object distance, distance d ij determined distance values QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 197 04 496 A1 [0002, 0010] DE 10 2013 214 677 B3

[0003] DE 10 2013 207 647 A1

[0003] DE 10 2018 104 668 A1

[0003]

Claims

[1] Time-of-flight camera for distance determination, with a time-of-flight sensor with at least one time-of-flight pixel, with a lighting system for emitting a modulated light, the time-of-flight camera for distance measurements is designed according to a phase measurement principle and is designed in such a way that, for at least a part of the time-of-flight pixels, for the determination of a distance (d w ) of an object, at least two measurements (A1, A2, ...) are carried out, each with different modulation frequencies (f1, f2, ...) and thus different unambiguousness ranges (UR1, UR2, ...), whereby in each measurement (A i ) a remaining distance (d i = mod(d, UR i )) is measured, Starting from these remaining distances (d i ) from at least two measurements (A1, A2, ...) the remaining distances (d i), preferably in the form of residual distance tuples (d1, d1, ...), initially a sub-area (M) or a corresponding sub-area index M of a total distance curve (UR extd ) are assigned, wherein a final assignment of the sub-area indices M is carried out depending on a 'nearest neighbor filtering' and / or a temporal filtering of the initially assigned sub-area indices (M), where the distance (d w ) by projecting the remaining distances (d i ) to the sub-area, with the value of the finally assigned sub-area index M. [2] Time-of-flight camera according to claim 1, wherein for the determination of the partial area index (M) an estimate of the partial area index (M(x, y, t)) is first determined for the given pixel, which is then offset against the estimates for spatially adjacent pixels and / or temporally preceding measurements. [3] Time-of-flight camera according to one of the preceding claims, in which the calculation of the initial sub-areas or the estimates of the sub-area index (M(x, y, t)) is carried out via a mean, a median, a weighted mean, or a weighted median. [4] Time-of-flight camera according to one of the preceding claims, wherein the filtering of the initial partial area index or the estimate of the partial area index (M) is carried out on the basis of an initially estimated combined distance and the known uniqueness ranges UR used in the measurement i is calculated.

Images