Light time-of-flight camera

Abstract

Light time-of-flight camera for distance measurement with a time-of-flight sensor with at least one time-of-flight pixel, with a lighting system for emitting modulated light, the time-of-flight camera is designed for distance measurements according to a phase measurement principle and is designed in this way, that, a) for at least a part of the light-time-of-flight pixels, for determining a distance (d w ) of an object at least two measurements (A1, A2, ...) are carried out with different modulation frequencies (f1, f2, ...) and thus different uniqueness ranges (UR1, UR2, ...), whereby in each measurement (A i ) a residual distance (d i = mod(d, UR i )) is measured, b) starting from these residual distances (d i ) for each light travel time pixel, initially a sub-area index M of a total distance curve (UR) extd) is assigned, c) a field of the initially assigned sub-area indices M is processed using a "nearest neighbor filter" and / or a temporal filter, to generate a corrected, final sub-area index (M_final) for the respective light-time-of-flight pixel, and only then, d) the distance (d w ) by projecting the remaining distances (d i ) is determined on that sub-area of ​​the total distance curve which is defined by the corrected, final sub-area index (M_final).

Description

[0001] The invention relates to a time-of-flight camera according to the preamble of claim 1.

[0002] Time-of-flight cameras are intended to include, in particular, cameras that derive time-of-flight information from the phase shift of emitted and received radiation. PMD cameras with photomixing detectors (PMDs), as described in DE 197 04 496 A1, are especially suitable as time-of-flight or 3D-TOF cameras.

[0003] To extend the uniqueness range of such a camera, several modulation frequencies are preferably used. Various approaches are described, for example, in documents DE 10 2013 214 677 B3, DE 10 2013 207 647 A1 and DE 10 2018 104 668 A1.

[0004] German patent DE 10 2020 127 332 A1 discloses a method for operating a time-of-flight camera system on a vehicle, in which the vehicle's movement is detected. Raw distance values ​​are determined for a detection frame using a first modulation frequency and converted into distance values. These distance values ​​are then calculated back to distance values ​​of a previous detection frame, taking into account the vehicle's movement in the interim. By comparing the calculated distance value with a raw distance value of the same pixel in the previous frame, the determined distance value is output as valid if the threshold is not met, and this process is repeated.

[0005] German patent DE 10 2014 204 423 A1 discloses a method for operating a time-of-flight camera system in which object distance is determined based on a phase shift and the detection range comprises at least two uniqueness ranges. In an initialization phase, objects in the detection range are detected, assigned to the respective uniqueness ranges, and at least one distance-dependent parameter is recorded for each object. In the subsequent position determination, the object's spatial position is determined, and the distance-dependent parameter is then recorded again. The determined spatial position is output as valid if a check shows that the distance-dependent parameter is plausible for this spatial position.

[0006] The purpose of the inventions is to improve the accuracy of multi-frequency methods.

[0007] The problem is solved by the camera according to claim 1.

[0008] Advantageously, a time-of-flight camera is provided for distance determination. with a time-of-flight sensor with at least one time-of-flight pixel, with a lighting system for emitting modulated light, the time-of-flight camera is designed for distance measurements according to a phase measurement principle and is designed in such a way that for at least a part of the light travel time pixels, at least two measurements with different modulation frequencies and thus different uniqueness ranges are carried out to determine the distance of an object, where a residual distance is measured in each measurement, where, starting from these residual distances from the at least two measurements, the residual distances are initially assigned to a sub-area or a corresponding sub-area index M of a total distance curve of a modulo diagram, where a final assignment of the sub-area indices M is made depending on a “nearest neighbor filtering” and / or a temporal filtering of the initially assigned sub-area indices M, where the distance is determined by projecting the remaining distances onto the sub-area, with the value of the finally assigned sub-area index M.

[0009] This approach has the advantage that, before a potentially complex calculation of a distance, an assignment to a sub-area of ​​a total distance curve is ensured, which simplifies the subsequent distance calculations and contributes to reducing distance noise.

[0010] They show: Fig. 1. Schematically, the basic principle of photomixed detection, Fig. 2 a modulated integration of the generated charge carriers, Fig. 3 a distance measurement using a wavelength, Fig. 4. A distance measurement using two different wavelengths, Fig. 5 a procedure according to the invention, Fig. 6 residual distances in a modulo diagram, Fig. 7. Nearest neighbor filtering.

[0011] In the following description of preferred embodiments, identical reference numerals denote identical or comparable components.

[0012] Fig. Figure 1 shows a measurement situation for an optical distance measurement with a time-of-flight camera, as is known, for example, from DE 197 04 496 A1.

[0013] The time-of-flight camera system 1 comprises a transmitter unit or illumination module 10 with an illumination 12 and an associated beam shaping optics 15, as well as a receiver unit or time-of-flight camera 20 with a receiving optics 25 and a time-of-flight sensor 22.

[0014] The time-of-flight sensor 22 has at least one time-of-flight pixel, preferably also a pixel array, and is in particular designed as a PMD sensor. The receiving optics 25 typically consist of several optical elements to improve the imaging properties. The beam-shaping optics 15 of the transmitting unit 10 can, for example, be designed as a reflector or lens optics. In a very simple embodiment, optical elements can optionally be omitted on both the receiving and transmitting sides.

[0015] The measurement principle of this arrangement is essentially based on the fact that, starting from the phase shift of the emitted and received light, the travel time and thus the distance traveled by the received light can be determined. For this purpose, the light source 12 and the light travel time sensor 22 are connected via a modulator 30 together with a specific modulation signal M owith a basic phase position φ0. In the example shown, a phase shifter 35 is also provided between the modulator 30 and the light source 12, with which the basic phase φ0 of the modulation signal M0 of the light source 12 is shifted by defined phase positions φ var can be shifted. For typical phase measurements, phase positions of φ are preferably used. var = 0°, 90°, 180°, 270° used.

[0016] Depending on the set modulation signal, the light source 12 sends an intensity-modulated signal S p1 with the first phase position p1 or p1 = φ0 + φ var off. This signal S p1 or, in the case shown, the electromagnetic radiation is reflected by an object 40 and arrives with a corresponding phase shift Δφ(t) due to the distance traveled. L ) with a second phase position p2 = φ0 + φ var + Δφ(t L ) as a received signal S p2on the light time-of-flight sensor 22. In the light time-of-flight sensor 22, the modulation signal M is o with the received signal S p2 mixed, whereby the phase shift or the object distance d is determined from the resulting signal.

[0017] To improve measurement accuracy and / or to extend the range of uniqueness, it is advantageous to perform the light transit time measurements with different modulation frequencies. For this purpose, the modulator 30 is connected to a modulation control unit 38, which can preferably specify modulation frequencies within a predetermined frequency spectrum.

[0018] The modulator 30 could, for example, be configured as a frequency synthesizer, which is controlled via the modulation control unit 38 for the respective measurement task. Switching between crystal oscillators with fixed frequencies is also conceivable.

[0019] Furthermore, the receiving unit 20 is connected to an evaluation unit 27. The evaluation unit 27 can optionally also be a component of the receiving unit 20 and, in particular, also a component of the light time-of-flight sensor 22. The task of the evaluation unit 27 is to determine and / or evaluate phase shifts based on the received signals in relation to the modulation frequency. The mixing of the received light beams with the modulation frequency preferably takes place in the light time-of-flight sensor 22 or PMD sensor. Furthermore, the modulation control unit 38 can also be a component of the evaluation unit 27. In particular, it can also be provided that the evaluation unit 27 takes over the function of the modulation control unit 38 completely or partially.

[0020] Infrared LEDs are preferably suitable as the illumination source or light source 12. Of course, other radiation sources in other frequency ranges are also conceivable, in particular light sources in the visible frequency range.

[0021] The basic principle of phase measurement is schematically represented in Fig. Figure 2 shows the upper curve of the modulation signal M0, which controls the illumination 12 and the light-time-of-flight sensor 22. The light reflected from object 40 arrives as the received signal S. p2 according to its light travel time t L phase-shifted Δφ(t L) to the time-of-flight sensor 22. The time-of-flight sensor 22 collects the photonically generated charges q over several modulation periods in the phase of the modulation signal M0 in a first accumulation gate Ga and in a phase shifted by 180° M0 + 180° in a second accumulation gate Gb. From the difference Δq of the charges qa, qb collected in the first and second gates Ga, Gb, the phase shift Δφ(t) can be determined. L ) and thus determine a distance d of the object.

[0022] For more precise phase determination and to cover the full 360° angular range, a second measurement is preferably performed with a modulation signal shifted by 90°. In this process, either the sensor or the illumination is shifted in phase by this angle.

[0023] The phase angle can then be determined in a known manner using an arctan function or arctan2 function: φ=arctanΔq(90°)Δq(0°)

[0024] To compensate for sensor asymmetry, for example, additional phase measurements shifted by 180° can be performed, so that the phase angle can be determined as follows. φ=arctanΔq(90°)−Δq(270°)Δ(0°)−Δq(180°)

[0025] From the in Fig. 2 shown runtime-related phase shift Δφ(t L ) can be used for object distances d that are smaller than half the wavelength λ of the modulation frequency Determine a distance d ≤ λ / 2 in a known manner. d=Δφ(tL)λ2π⋅12

[0026] For distances d > λ / 2, there is usually no way to measure the phase shift absolutely, so that the determined phase shift can no longer be uniquely assigned to a distance value.

[0027] Fig. Figure 3 shows an example where the object 40 is at a distance d from the transmitter 10. d=2λ+R2 exhibits, whereby of course the total distance traveled to receiver 20 is twice as large, namely D = 2d = 4λ + R

[0028] To increase the range of uniqueness, it is, as in Fig. Figure 4 schematically illustrates the intention to determine an object distance d using at least two modulation frequencies or modulation wavelengths. For the sake of simplicity, in Fig. Figure 4 shows the total distance D between transmitter 10 and receiver 20. Within the common uniqueness range of the two wavelengths λ1, λ2, which is typically spanned by the least common multiple of the wavelengths λ1, λ2, the following distance equation applies: D=2d=n1λ1+R1=n2λ2+R2 with Ri=Dmodλi=φi(fi,D)λi2π where the relative phase shift φ depends on the modulation frequency and the object distance i (f i ,D) applies: φi(fi,D)≡D⋅fic⋅2πmod2π=Dλi⋅2πmod2π

[0029] The relative phase shift φ i (f i ,D) is therefore a measure of the remaining piece R in the distance measurement i . Is the relative phase shift φ i (f i ,D) into a distance value D i (f i Converted to ,D), the remaining piece corresponds to R i exactly this distance value D i with R i ,D i · ∈ [0, λ i ]. Ri=Di=2di=φiλi2π

[0030] Distance determination can now be achieved using two phase shifts φ recorded for different modulation frequencies f1, f2. 1 / 2 (f 1 / 2 ,D) a solution for the distance equation shown above must be found.

[0031] In principle, to extend the uniqueness range of indirect TOF camera systems, multiple measurements at different frequencies can be combined. With such multi-frequency methods, the region of the extended uniqueness range in which the measurement point is located is typically determined first, and then the exact distance is calculated. However, noise or faulty measurements can cause this initial process step and rough localization to be inaccurate, leading to significant distance errors.

[0032] According to the invention, it is provided that the determination of the position in the extended uniqueness range is not only carried out pixel by pixel, but that information from temporally or spatially adjacent regions is incorporated before an accurate distance image is generated.

[0033] As previously described, TOF camera systems measure distances in a scene by determining the travel time of light from a light source to the object and back to the camera. The measured light travel time is directly proportional to the distance being measured. Instead of determining the actual travel time, indirect TOF systems measure the phase shift between a light source emitting light at a specific frequency f and the object emitting light at a specific frequency f. mod The phase shift φ is proportional to the propagation time and thus also to the measured distance d. However, due to the periodicity of the phase, a measured phase shift can correspond to several distances. The unambiguous range (UR) is therefore limited to the distance up to half the wavelength of the emitted signal. d=φ2π⋅URwith UR=clight2fmod, where c lightThe speed of light. An effective and well-known method to extend the uniqueness range is to combine measurements of different frequencies. This then encompasses half the wavelength of the frequency from the greatest common divisor of all measurement frequencies. URextd=clight2⋅gcd(f1,f2,…fn)

[0034] The measured distances are only available as residual distances (d1,..., d) n ) known, where d1 = mod(d w , UR1), ..., d n = mod(d w , UR n ).

[0035] The actual distance d w The distance of an object can be calculated as a combination of the measured residual distances (d1, ..., d). n ) and an integer multiple of the individual UR specific to the respective modulation frequency. For an ideal measurement without errors or noise, there is a vector S of integer values ​​(m1, m2, ..., m). n ) for which the following applies: dw=d1+m1⋅UR1=d2+m2⋅UR2=⋯=dn+mn⋅URn

[0036] Determining the integer values ​​m1, ..., m n This can be understood as a rough localization, indicating in which area of ​​the extended UR the measurement point is located. Real measurements (d i However, these are subject to deviations and noise. i , so that d˜1=d1+F1+m1UR1d˜2=d2+F2+m2UR2…d˜n=dn+Fn+mnURn

[0037] This means that the vector S = (m1, m2 ... m n ) to determine such that d̃1 ≈ d̃2 ≈ ... ≈ d̃ n The actual distance can then be calculated from the (weighted) individual measurements (d̃1, ..., d̃). n ) are determined.

[0038] If the noise of the individual measurements or the deviations are too large, resulting in a "false" vector S = (m1, m2 ... m n When calculated, the error to the true distance is much larger than the original noise.

[0039] The following procedure is intended to prevent or subsequently correct these errors: If S is calculated for each pixel at each measurement time, S(x, y, t) can be understood as a temporal sequence of images that record in which area of ​​the extended uniqueness range each pixel lies at the current time.

[0040] Filtering this data aims to correct individual errors caused by noise. This filtering can be performed in... - spatial (local, over x,y), - temporal (time-related, over t) - or a combination of the two domains be carried out.

[0041] Apart from distance edges at the periphery of objects and finer structures that represent larger distance jumps in the measured scene, it can be assumed that pixels that are spatially and temporally close to each other lie in the same area of ​​the extended uniqueness domain. An effective reduction of distance values ​​incorrectly assigned within the uniqueness domain can therefore be achieved by low-pass filtering S(x, y, t). A simple or weighted moving average would be conceivable, for example. However, median filtering of the data appears more promising for several reasons. On the one hand, edges and structures are preserved, and only objects smaller than the filter size are modified. On the other hand, no previously unincorporated information is generated, since median filtering ensures that the pixel to be filtered is assigned to a vector S onto which other nearby pixels are also projected.

[0042] Each of these approaches is facilitated by the fact that every vector S can be uniquely represented by a numerical value M suitable for filtering, ideally with a linear relationship to the final distance. For median filtering in particular, it is advantageous to use as few integer values ​​as possible. One such possibility is described here as an example:

[0043] In Fig. Figure 5 shows an example measurement with two frequencies: f1=15 MHz,UR1=10 m f2=12 MHz,UR2=12.5 m enables an extended uniqueness range of UR extd = 50 m. This results in eight unique vectors (S0 = (0,0), S1 = (1, 0), S2 = (1, 1), ..., S7 = (4, 3)), each describing a sub-area.

[0044] If n is the number of measurements and the final distance d has already been calculated, then it is obvious, for example, to use M as M=∑i=1nmi with mi=⌊dURi⌋ to calculate. For the example above, M can therefore take on the integer values ​​0 to 7, which allows for efficient median filtering. The corrected distance can then be easily calculated as d=di+miURi with mi from SM

[0045] Fig. Figure 6 shows the facts according to Fig. 5 again in a so-called modulo diagram. A total distance curve (UR) is plotted. extd ) depending on distance pairs (d i (f i ), d j (f j )) or remaining distance pairs (d i , d j ) of the selected modulation frequencies f i , f j The total distance curve extends into several sub-areas M in the modulo diagram. Furthermore, the Fig. Six multiple measured distance pairs or residual distance pairs. In the example shown, the measured distance pairs marked with filled dots are located near a sub-area with sub-area index M = 0 and are assigned to this sub-area. The distance pairs with the unfilled dots are located near the sub-areas M with indices M = 7 and M = 2 and would typically be assigned to these sub-areas.

[0046] To avoid such potential misassignments, it is planned, as already described, to filter the indices of the initially assigned sub-areas temporally and / or spatially and to carry out the final sub-area assignment depending on this filtering.

[0047] Fig.Figure 7 shows a "nearest neighbor filter" where the middle pixel is corrected or assigned from the initially assigned sub-area index M = 7 to M = 0, taking its neighboring pixels into account. Of course, pixels beyond the next nearest or further neighbors can also be included.

[0048] After filtering the subrange indices, it is possible to calculate, in a known manner, for each measured distance pair d i , d j or, for each remaining distance pair, for example by projecting the measurement points onto the assigned sub-section of the total distance curve, a "true" distance d. w to be determined.

[0049] The term distance pair is intended to include, in particular, higher-dimensional n-tuples, especially 3-tuples. Reference symbol list 10 Lighting modules 12 Lighting 22 Light time-of-flight sensor 27 evaluation units 30 Modulator 35 Phase shifters, lighting phase shifters 38 Modulation control unit Δφ(t L ) runtime-related phase shift φ var Phase position φ0 basic phase M0 modulation signal p1 first phase p2 second phase Sp1 transmit signal with first phase SP2 Receive signal with second phase λ wavelength Total distance D ij determined distance values d object distance, distance d ij determined distance values

Claims

[1] Time-of-flight camera for distance measurement, with a time-of-flight sensor with at least one time-of-flight pixel, with a lighting system for emitting modulated light, the time-of-flight camera is designed for distance measurements according to a phase measurement principle and is designed in this way, that, a) for at least a part of the light-time-of-flight pixels, for determining a distance (d w ) of an object at least two measurements (A1, A2, ...) are carried out with different modulation frequencies (f1, f2, ...) and thus different uniqueness ranges (UR1, UR2, ...), whereby in each measurement (A i ) a residual distance (d i = mod(d, UR i )) is measured, b) starting from these residual distances (d i ) for each light travel time pixel, initially a sub-area index M of a total distance curve (UR) extd ) is assigned, c) a field of the initially assigned sub-area indices M is processed using a "nearest neighbor filter" and / or a temporal filter, to generate a corrected, final sub-area index (M_final) for the respective light-time-of-flight pixel, and only then, d) the distance (d w ) by projecting the remaining distances (d i ) is determined on that sub-area of ​​the total distance curve which is defined by the corrected, final sub-area index (M_final). [2] Time-of-flight camera according to claim 1, wherein for the determination of the sub-area index (M) an estimate of the sub-area index (M(x, y, t)) for the given pixel is first determined, which is then calculated with the estimates for spatially adjacent pixels and / or temporally preceding measurements. [3] Time-of-flight camera according to one of the preceding claims, wherein the calculation of the initial sub-areas or the estimates of the sub-area index (M(x, y, t)) is carried out using a mean, a median, a weighted mean, or a weighted median. [4] Time-of-flight camera according to one of the preceding claims, wherein the filtering of the initial sub-area index or the estimation of the sub-area index (M) is based on an initially estimated combined distance and the known uniqueness ranges UR used in the measurement. i is calculated.

Images