Light travel time camera system

Abstract

Method for operating a time-of-flight camera system with a time-of-flight sensor consisting of an array of time-of-flight pixels on a vehicle, in which the movement of the vehicle is detected and made available for calculations, comprising the steps: a) Determination of raw distance values ​​(mij(tn,fs)) to a first detection frame (tn) and with a first modulation frequency (fs), b) Conversion of the raw distance values ​​(mij(tn,fs)) into distance values ​​(dij(tn,fs,k)) under the condition dij(tn,fs,k) = mij(tn,fs) + k * EB(fs) with a k-value as a counter from 0 to a maximum k-value and a frequency-dependent uniqueness range (EB), c1) Conversion of the distance values ​​(dij(tn,fs,k)) for a respective pixel coordinate (ij) into coordinate values ​​(kij(tn,fs,k)), c2) Back-calculation of the coordinate values (kij(tn,fs,k))in coordinate values ​​(k'ij(tn-1,fs,k)) to a previous time (tn-1) taking into account the movement (T&Dgr; ) that took place between the two times (t1, t2, &Dgr;x).t) of the vehicle, c3) Conversion of the reverse-calculated coordinate values ​​(k'ij(tn-1,fs,k)) into reverse-calculated distance values ​​(d'pq(tn-1,fs,k)) and pixel coordinates (pq) of a previous acquisition frame (tn-1)., d) Limit comparison of the reverse-calculated distance value d'pq(tn-1,fs,k) with a corresponding raw distance value mpq(tn-1,fs-1) of the same pixel Pixpq according to the rule: |d′pq(tn−1,fs,k)−mpq(tn−1,fs−1)|(modEB(tn−1,fs−1)) <GWe) Ausgabe des im Schritt b) ermittelten Distanzwert dij(tn,fs,k) als valide, wenn ein k-Wert existiert, bei dem der Grenzwert GW unterschritten wird, unter Berücksichtigung des ermittelten k-Werts,f) Wiederholung der vorgenannten Schritte für einen nächsten Erfassungsframe (tn+1) mit einer nächsten Frequenz (fs+1).;

Description

The invention relates to a time-of-flight camera system and a method for operating such a system according to the preamble of the independent claims. Time-of-flight (TOF) camera systems should not only include systems that determine distances from the phase shift of emitted and received radiation. PMD cameras with photomixing detectors (PMDs), such as those described in DE 197 04 496 A1, are particularly suitable as time-of-flight or 3D TOF cameras. To determine a distance or a corresponding phase shift of the reflected light, the reflected light is mixed with the modulating signal in the PMD sensor, as described in detail in DE 197 04 496 A1. This mixture yields an in-phase signal (0°) and a signal shifted by 180°, from which a distance can be determined in a known manner. To improve the quality of the distance measurement, it is possible to selectively shift the transmit modulation by, for example, 90°, 180°, or 270° and preferably to determine a phase angle of the reflected signal relative to the transmitted signal using in-phase quadrature demodulation. This approach is particularly useful for obtaining redundant information to compensate for various parasitic effects such as fixed pattern noise (FPN), background light, or sensor asymmetries. Furthermore, a time-of-flight camera system is known from DE 10 2013 207 654 A1, in which, in one embodiment, each phase position in a detection frame is operated with a different modulation frequency. To determine a distance, the phase positions are converted to a common modulation frequency. A distance is considered valid if the distance differences in several distance measurements remain below a limit value. This limit value can be adjusted, for example, to account for camera movement. German patent DE 10 2013 207 648 A1 discloses a method for a time-of-flight camera system in which a distance is determined using different modulation frequencies. In different measurement cycles, it is intended to select different frequency pairings randomly but according to specific rules in order to, for example, avoid unfavorable frequency pairings. DE 10 2016 219 518 A1 discloses a PMD camera in which the illumination is designed to emit modulated light in the form of a dot pattern, wherein only pixels whose detected useful light amplitude exceeds a predetermined amplitude limit are used for distance determination, and wherein the distance values ​​of immediately adjacent pixels are combined to form a common distance value. The object of the invention is to improve the distance measurement of a time-of-flight camera system. The problem is advantageously solved by the time-of-flight camera system and method according to the inventive clause of the independent claims. Advantageously, a method for operating a time-of-flight camera system with a time-of-flight sensor is provided, consisting of an array of time-of-flight pixels on a vehicle, in which the movement of the vehicle is detected and made available for calculations, comprising the steps: a) Determination of raw distance values ​​(mij(tn,fs)) to a first detection frame (tn) and with a first modulation frequency (fs), b) Conversion of the raw distance values ​​(mij(tn,fs)) into distance values ​​(dij(tn,fs,k)) under the specification with a k-value as a counter from 0 to a maximum k-value and a frequency-dependent uniqueness range (EB), c1) Conversion of the distance values ​​(dij(tn,fs,k)) for a respective pixel coordinate (ij) into coordinate values ​​(kij(tn,fs,k)), c2) Conversion of the coordinate values ​​(kij(tn,fs,k)) back into coordinate values (k'ij(tn-1,fs,k)) to a previous time point (tn-1) taking into account the time difference between the two times (t1, t2,Δx) movement (TΔt) of the vehicle that has taken place, c3) conversion of the recalculated coordinate values ​​(k'ij(tn-1,fs,k)) into recalculated distance values ​​(d'pq(tn-1,fs,k)) and pixel coordinates (pq) of a previous acquisition frame (tn-1), d) limit comparison of the recalculated distance value d'pq(tn-1,fs,k) with a corresponding raw distance value mpq(tn-1,fs-1) of the same pixel Pixpq according to the rule: e) output of the distance value dij(tn,fs,k) determined in step b) as valid if a k-value exists at which the limit value GW is undercut, taking into account the determined k-value, f) repetition of the aforementioned steps for a next acquisition frame (tn+1) with a next frequency (fs+1)., This approach has the advantage that valid and accurate distance data can be output using quick calculations. It is useful to use different modulation frequencies (fn) for directly consecutive acquisition frames (tn, tn+1). Preferably, the modulation frequencies (fs) are taken from a set of predefined modulation frequencies, wherein the set has a predefined maximum number (smax) of modulation frequencies. In a further embodiment, it is advantageous to determine the distances for an output of a valid distance value (dij) over a number of acquisition frames (tn) that is greater than the number (smax) of modulation frequencies specified in the modulation frequency set (fs). This helps to avoid measurement artifacts that can arise from a homogeneous scene. It is useful to provide an incremental counter (Cij) that sums up the capture frames with valid distance values, with the counter (Cij) being set to zero as soon as a capture frame has an invalid distance value. Advantageously, a design is provided in which, starting from a pixel coordinate (pq) which results from the back-calculated distance value (d'pq), neighboring pixels are also subjected to the limit comparison according to step d) of claim 1. In such a configuration, if several pixels fall below the threshold, it is helpful to select the pixel whose incremental counter (Cuv) shows the highest value and to increment the incremental counter (Cuv) of this pixel by one. It is particularly advantageous to provide a time-of-flight camera system (1) with a modulator (30) connected to an illumination (10) and a receiver (20) of the time-of-flight camera system (1), a modulation control unit (38) connected to the modulator (30) and designed such that the modulator (30) can be operated with at least two modulation frequencies (f1, f2, f3), and to configure the time-of-flight camera system (1) to carry out one of the aforementioned methods. Figure 1 schematically shows the basic principle of photomixed detection, Figure 2 a modulated integration of the generated charge carriers, Figure 3 two time courses of charge integration with different phase positions, Figure 4 relation of the integration in an IQ diagram, Figure 5 a distance measurement with one wavelength, Figure 6 a distance measurement with two different wavelengths, Figure 7 a course of the phase shifts with distance, Figure 8 a course of the distance values ​​for different wavelengths, Figure 9 schematically a distance determination for different wavelengths, Figure 10 schematically an object detected by a pixel array, Figure 11 a sensor moved relative to the example according to Figure 10, Figure 12 schematically a back-calculation of the distances of the moving sensor to a previous time point, Figure 13 a flowchart of the procedure according to the invention, Figure 14 different uniqueness ranges, Figure 1515 a detection of objects in different multiples of the uniqueness ranges, Fig. 16 a definition of a measuring range according to the invention, Fig. 17 an output of valid distance values ​​according to the invention. In the following description of preferred embodiments, identical reference numerals denote identical or comparable components. Fig. 1 shows a measurement situation for an optical distance measurement with a time-of-flight camera, as is known, for example, from DE 197 04 496 A1. The time-of-flight camera system 1 comprises a transmitter unit or illumination module 10 with an illumination 12 and an associated beam shaping optics 15, as well as a receiver unit or time-of-flight camera 20 with a receiving optics 25 and a time-of-flight sensor 22. The time-of-flight sensor 22 has at least one time-of-flight pixel, preferably also a pixel array, and is in particular designed as a PMD sensor. The receiving optics 25 typically consist of several optical elements to improve the imaging properties. The beam-shaping optics 15 of the transmitting unit 10 can, for example, be designed as a reflector or lens optics. In a very simple embodiment, optical elements can optionally be omitted on both the receiving and transmitting sides. The measurement principle of this arrangement is essentially based on the fact that, starting from the phase shift of the emitted and received light, the transit time and thus the distance traveled by the received light can be determined. For this purpose, the light source 12 and the light transit time sensor 22 are jointly supplied with a specific modulation signal M0 with a basic phase angle φ0 via a modulator 30. In the illustrated example, a phase shifter 35 is also provided between the modulator 30 and the light source 12, with which the basic phase φ0 of the modulation signal M0 of the light source 12 can be shifted by defined phase angles φvar. For typical phase measurements, phase angles of φvar = 0°, 90°, 180°, 270° are preferably used. Depending on the set modulation signal, the light source 12 emits an intensity-modulated signal Sp1 with the first phase position p1 or p1 = φ0 + φvar. In the illustrated case, this signal Sp1, or the electromagnetic radiation, is reflected by an object 40 and, due to the distance traveled, arrives at the light-time-of-flight sensor 22 with a corresponding phase shift Δφ(tL) and a second phase position p2 = φ0 + φvar + Δφ(tL) as the received signal Sp2. In the light-time-of-flight sensor 22, the modulation signal Mo is mixed with the received signal Sp2, and the phase shift or the object distance d is determined from the resulting signal. To improve measurement accuracy and / or to extend the range of uniqueness, it is advantageous to perform the light transit time measurements with different modulation frequencies. For this purpose, the modulator 30 is connected to a modulation control unit 38, which can preferably specify modulation frequencies within a predetermined frequency spectrum. The modulator 30 could, for example, be configured as a frequency synthesizer, which is controlled via the modulation control unit 38 for the respective measurement task. Switching between crystal oscillators with fixed frequencies is also conceivable. Furthermore, the receiving unit 20 is connected to an evaluation unit 27. The evaluation unit 27 can optionally also be a component of the receiving unit 20 and, in particular, also a component of the light time-of-flight sensor 22. The task of the evaluation unit 27 is to determine and / or evaluate phase shifts based on the received signals in relation to the modulation frequency. The mixing of the received light beams with the modulation frequency preferably takes place in the light time-of-flight sensor 22 or PMD sensor. Furthermore, the modulation control unit 38 can also be a component of the evaluation unit 27. In particular, it can also be provided that the evaluation unit 27 takes over the function of the modulation control unit 38 completely or partially. Infrared LEDs are preferably suitable as the illumination source or light source 12. Of course, other radiation sources in other frequency ranges are also conceivable, in particular light sources in the visible frequency range. The basic principle of phase measurement is shown schematically in Fig. 2. The upper curve shows the time course of the modulation signal M0, which controls the illumination 12 and the time-of-flight sensor 22. The light reflected from the object 40 arrives at the time-of-flight sensor 22 as a received signal Sp2, phase-shifted Δφ(tL) according to its time-of-flight tL. The time-of-flight sensor 22 collects the photonically generated charges q over several modulation periods in the phase of the modulation signal M0 in a first accumulation gate Ga and in a phase shifted by 180° M0+180° in a second accumulation gate Gb. From the ratio of the charges qa, qb collected in the first and second gates Ga, Gb, the phase shift Δφ(tL) and thus a distance d of the object can be determined. Figures 3a and 3b show the charge difference Δq = qa - qb / (qa + qb) as a function of the phase shift Δφ(tL) of the received light signal Sp2 with different phase angles. Figure 3a shows a curve for an unshifted modulation phase M0 with a phase angle φvar = 0°. When the signal Sp2 arrives without phase shift, i.e., Δφ(tL) = 0°, for example, when the transmitted signal Sp1 is directed directly to the sensor, the phases of the modulation M0 and the received signal Sp2 are identical, so that all generated charge carriers are detected phase-synchronously at the first gate Ga, resulting in a maximum difference signal with Δq = 1. With increasing phase shift, the charge at the first accumulation gate Ga decreases and at the second accumulation gate Gb increases. At a phase shift of Δφ(tL) = 90°, the charge carriers qa, qb are equally distributed at both gates Ga, Gb, and the difference is therefore zero, and after a phase shift of 180°, it is -1. With further increasing phase shift, the charge at the first gate Ga increases again, so that the charge difference increases again, reaching a maximum at 360° and 0°, respectively. Mathematically, this is a correlation function of the received signal Sp2 with the modulating signal M0. As previously shown, modulation with a square wave results in a triangular correlation function. Modulation with, for example, a sine wave would result in a cosine function. As Fig. 3a shows, a measurement of the phase with a phase position is only unambiguous up to a phase shift Δφ(tL) ≤ 180°. For maximum detection of the phase shift, the IQ (in-phase quadrature) method is known, in which two measurements are carried out with phase positions shifted by 90°, for example with phase position φvar = 0° and φvar = 90°. The result of a measurement with phase position φvar = 90° is shown in Fig. 3b. The relationship between these two curves can be represented in a known manner, for example for sinusoidal curves in an IQ diagram according to Fig. 4. As a first approximation, this representation is also readily applicable to the triangular functions shown. The phase angle can then be determined in a known manner using an arctan function: To compensate for sensor asymmetry, for example, additional phase measurements shifted by 180° can be performed, so that the phase angle can be determined as follows. From the propagation delay-related phase shift Δφ(tL) shown in Fig. 2, a distance can be determined in a known manner for object distances d that are smaller than half the wavelength λ of the modulation frequency d ≤ λ / 2. For distances d > λ / 2, there is usually no way to measure the phase shift absolutely, so that the determined phase shift can no longer be uniquely assigned to a distance value. Fig. 5 shows an example where the object 40 has a distance d from the transmitter 10, where of course the distance traveled to the receiver 20 is twice as large, namely D = 2d = 4λ + R To increase the uniqueness range, as schematically illustrated in Fig. 6, an object distance d is determined using at least two modulation frequencies or modulation wavelengths. For simplicity, Fig. 6 shows the total distance D between transmitter 10 and receiver 20. Within the uniqueness range of the two wavelengths λ1, λ2, which is typically defined by the least common multiple of the wavelengths λ1, λ2, the following distance equation applies: where the relative phase shift φi(fi,D), which depends on the modulation frequency and the object distance, is given by: The relative phase shift φi(fi,D) is thus a measure of the remaining distance Ri in the distance measurement. For distance determination, a solution to the distance equation shown above can now be found using two phase shifts φ1 / 2(f1 / 2,D) recorded for different modulation frequencies f1 and f2. One possible solution is shown schematically in Fig. 7. Fig. 7 shows two relative phase shifts φ1 / 2(f1 / 2,D) as a function of twice the object distance 2d = total path length D for two different frequencies f1, f2. The solid line represents the phase shift φ1 for f1 = 7.5 MHz corresponding to a wavelength λ1 = 40 m, and the dashed line represents f2 = 5 MHz corresponding to a wavelength λ2 = 60 m. The uniqueness range EB12 for the two frequencies f1, f2 is determined in a known manner by the least common multiple of the two wavelengths λ1, λ2, which in this case is 120 m. For each distance value or total path length D within the common uniqueness domain EB12, there is exactly one phase difference pair (φ1, φ2). For the exemplary distance value D of 23 m, i.e., an object distance d of 11.5 m, the resulting phase difference pair is approximately (1.2 | 0.8). In one possible application, it might be necessary to store a suitable number of phase difference pairs, along with their associated distance value D or object distance d, in a table of values. When measuring distances, it would then be possible to determine which tabulated phase difference pair with a correspondingly assigned distance value is closest to the measured phase difference pair. Alternatively, the object distance could be calculated each time. Fig. 8 corresponds to the representation according to Fig. 7 with the difference that the length of the respective remaining piece is plotted on the y-axis with: In the example shown, up to a total path length D corresponding to the shortest wavelength, in this case 40 m, both remaining segments are the same length. However, for a total distance D of, for example, 70 m, the remaining segments are of different lengths. The relationship between phase and distance shown in Figures 7 and 8 can be advantageously represented in a so-called modulo diagram according to Figure 9. The x- and y-axes represent the phase values ​​φ1 and φ2 for a first and second modulation frequency, and the secondary x- and y-axes represent the distance values ​​or remainder values ​​d1 and d2 corresponding to the phase values. As already mentioned, for a given distance value, only a single pair of phase values ​​exists within the uniqueness range. Fig. 9 shows a modulo diagram for the frequencies f1 = 7.5 MHz corresponding to a wavelength λ1 = 40 m and f2 = 5 MHz corresponding to a wavelength λ2 = 60 m, as already shown in Figs. 7 and 8. The curve begins with the phase difference pair (0 | 0) for D = 0. When the total path length of the light reflected from the object reaches the wavelength λ1 = 40 m of the first modulation frequency, the phase value also reaches its maximum value, namely 2π with the phase value pair (2 | 1.33). With increasing distance, the curve jumps at the points where one of the two phase values ​​passes through a 2π value, until a uniqueness range EB of 120 m is reached. A distance can be determined, for example, by assigning a measured phase value pair to a distance point on the distance curve. Phase value pairs only ideally lie on the distance curve and typically deviate from it, for example, due to noise. In the case shown in Fig. 9, a measured phase value pair with (1.05 | 0.85) is shown as an example. This phase value pair does not lie on the distance curve and must now be assigned to a distance point on one of the two curve segments. The distance dAB between the curve segments is known. For the assignment, it is therefore sufficient to determine the distance dA, dB to one of the two curve segments. The object distance is then determined from the nearest distance point. In the case shown, a distance value D of 23 m, i.e., an actual object distance d of 11.5 m, can be assigned to the phase value pair. If, however, a recorded phase value pair lies, for example, in the middle of the adjacent distance line or curve segments, a distance of 93 m could be assigned to the phase value pair instead of the actual distance of 23 m. Incorrect distance values ​​arise not only from such misclassifications, but also from objects traveling beyond the uniqueness range. In the illustrated case, the uniqueness range EB extends to a total distance D of 120 m, corresponding to a maximum object distance d of 60 m. Typically, a time-of-flight camera system is designed such that objects outside the uniqueness range produce only a weak signal and are disregarded in the evaluation. However, objects with high reflectivity can generate a sufficiently strong signal at the sensor and are thus recognized as objects. If such an object is located at an actual distance of d = 71.5, i.e., a total distance of 143 m, it will be detected with a total distance D of 23 m due to the uniqueness range being limited to 120 m. With an overrange of 213 m, the distance value D is 93 m. In order to detect overreach and misattributions, the invention provides for repeating the distance measurement with further different modulation frequencies and correspondingly different uniqueness ranges, and for allowing distance values ​​only if preferably all or a predetermined number of distance measurements lead to the same result within tolerated limits. A dual- or multi-frequency method works best with static arrangements and / or arrangements moving at low speeds. With moving objects or a moving camera, invalid distance values ​​are to be expected, as the distances within a pixel no longer correspond to each other due to the movement. The core idea of ​​the invention is to detect the intrinsic movement of the time-of-flight camera and to compensate for it using a multi-frequency method. The camera's intrinsic movement can be measured in various ways, for example: using an encoder on the wheels that captures the movement in two-dimensional space, or by integrating the data from a multi-axis gyroscopic sensor. It is also conceivable to use so-called SLAM (Simultaneous Localization and Mapping) methods to determine the intrinsic movement. Furthermore, it is helpful to define the optical system used in an optical model, taking into account factors such as focal length, distortion, etc. The optical model should support both the projection of 3D points onto pixels of the sensor matrix and the back-projection of pixels to 3D directions. Furthermore, the camera system must be calibrated in combination with the transport system. This extrinsic calibration should then match the coordinate system of the ego or vehicle system. Figures 10, 11 to 12 schematically illustrate the basic concept of the invention. At a first time t1, the vehicle is located at a first position, as shown in Figure 10, and the sensor detects an object with pixel Pix_7,2. According to the optics or the optical system upstream of the sensor, each pixel of the sensor sees a different spatial region. For example, pixel Pix_5,3 looks in a different spatial direction than pixel Pix_7,2 and therefore does not detect the object. Distance data is determined for all pixels of the sensor, so that a complete depth image of the detected environment or a distance matrix D can be created from this. As a known procedure, the raw data initially consists of charge differences from which, as already described, phase shifts are determined and, starting from the modulation frequencies f used, a raw distance value m is calculated. Due to the periodicity of the uniqueness domain EB, an actual distance value d results from: Thus, for each pixel Pixijder, the distance value dij(k) is determined as a function of k. Based on a known optical model of the optics or the optical system, the spatial orientation of each pixel is known, so that, together with the determined distance dij(k), the distances of the detected environment for each pixel can be converted into a Cartesian coordinate system of the vehicle. Thus, distance values ​​dij(k) can be transformed into coordinate values ​​kij(k) = (x,y,z), or a distance matrix D(k) into a coordinate matrix K(k). The object captured by pixel Pix7,2 in Fig. 10 is located at the first time t1 with the distance value d7,2(t1,k) at the location k7,2(t1,k) = (x1, y1, z1)7,2 in the ego- / coordinate system of the vehicle. If the vehicle is moved, as shown in Fig. 11, the position of the object also shifts in relation to the vehicle and thus also in relation to the sensor or its pixels, so that at a second time t2 the object is detected at a distance d5,3(t2,k) from another pixel Pix5,3. As already described, these distances d(tn)ij for all pixels Pixij can then also be transformed into a coordinate matrix K for all new times tn. According to the invention, it is provided that different modulation frequencies fs are used in successive acquisition times / acquisition frames t1, t2, i.e. the distances dij(t1,f1,k) at the first time t1 were determined with a first modulation frequency f1 and the distances dij(t2,f2,k) at the second time t2 with a second modulation frequency f2. According to the invention, at least two modulation frequencies are used, which are selected randomly or in a predetermined pattern for a given time. Preferably, a set with a maximum number of modulation frequencies is specified, with which the measurements are then carried out. To verify the distance dij(t2, f2,k) recorded at the second time t2 with a second frequency f2, the invention provides to correct the distances dij(t2,f2) determined at the second time t2 for the vehicle's own motion TΔt, which occurred between the first and second time Δt = t2 - t1, and to calculate back to distances d'pq(t1,f2,k) of the previous time, i.e., here to the first time t1. This procedure is illustrated in Fig. 12. From the known motion TΔt of the vehicle, which generally consists of a translational component x and a rotational component rot, the distance value dij(t2,f2,k) or the coordinate value kij(t2,f2,k) of the object can be calculated back to the previous time tn-1. If the object remains stationary and the motion TΔt of the vehicle is known exactly, the calculated distance d'pq(t1,f2,k) would be essentially identical to the object distance dij(t1,f1,k) actually measured at time t1, provided the appropriate k-values ​​have been found. The steps summarized again: a) dij(t2,f2,k) →(x2,f2,ky2,f2,k, z2,f2,k)ij= kij(t2,f2,k) - Conversion of distance values ​​into coordinate values ​​b) kij(t2,f2,k) _ TΔt→ k'pq(t1,f2,k) - Conversion of the coordinate values ​​back to coordinate values ​​at the previous time c) k'pq(t1,f2,k) → d'pq(t1,f2,k) - Transformation of the converted coordinate values ​​into distance values ​​ed) Comparison of the distance values The steps are preferably performed for all pixels of the sensor. It should be noted once again that due to the movement TΔt of the vehicle, not only do the distances at the pixels change, but also the position of the objects in relation to the sensor or the pixels changes, so that in the subsequent time points / capture frames tn+1 the object is usually captured by different pixels. It should also be noted that when calculating the coordinate matrix back to a previous point in time, in addition to a potentially slightly altered distance value, the point of reference on the sensor may also be different. Therefore, a distance value is not calculated back to the pixel that previously detected the object, but possibly to a neighboring pixel. Such borderline cases can potentially be handled using so-called neighborhood filtering. In cases where the back-calculated pixels Pixpq are outside the visible range of the camera or sensor at time t1, no validation of the distance value dij(t2,f2,k) can take place. To verify the determined distance values, the following method steps are provided according to the invention, as shown in Fig. 13: a) Determination of raw distance values ​​mij(tn,fs) at a first time tn and with a first modulation frequency fs, b) Conversion of the raw distance values ​​mij(tn,fs,k) into distance values ​​dij(tn,fs,k) under the specification dij(tn,fs,k) = mij(fn,fs) + k*EB(fs), c) Conversion of the distance values ​​dij(fn,fs,k) into coordinate values ​​kij(fn,fs,k), d) Back-calculation of the coordinate values ​​kij(tn,fs,k) into coordinate values ​​k'ij(tn-1,fs, k) to a previous time tn-1, taking into account the movement TΔt of the vehicle that took place between the two times t1, t2, e) Conversion of the back-calculated coordinate values k'ij(tn-1,fs,k) into back-calculated distance values ​​d'pq(tn-1,fs,k). It should be taken into account that, in general, the projected pixel coordinates pq for the image at time t0 are not integers.In one possible embodiment of the invention, the projected pq coordinates can be converted to valid pixel coordinates using a nearest-neighbor method. Furthermore, it is advantageous if the new pq coordinates, through the projection of k', also take into account an optical model and an extrinsic calibration. f) Limit comparison of the distance value d'pq(tn-1,fs,k) from step e) with a corresponding raw distance value mpq(tn-1,fs-1) of the same pixel Pixpq. In the most general case, the following formalism is used for the limit consideration: or simplified if the modulo for circular frequencies is taken into account: |d'pq(tn-1,fs,k) - mpq(tn-1, fs-1)|mod EB(tn-1, fs-1) < GW. Here, the k-values ​​are traversed until a maximum specified measurement range d_max is reached or exceeded, with the specification kmax* EB(tn-1,fs-1) >= dmax = maximum specified measurement range MB.g) If a k-value exists below the limit value GW, a distance value dij(tn,fs,k) is calculated and output using this k-value. If no such k-value is found, the distance value for the respective pixel is marked as invalid. h) The aforementioned steps are repeated for the next time point tn+1 with the next frequency fs+1. Here, an outer loop iterates over all k and an inner loop over all pixels. As soon as a match is found for a pixel, this distance dij is recorded in step g) and can be output. As briefly outlined in Fig. 14, different uniqueness ranges E1, E2, ... result in a known manner for different frequencies f1, f2, ... This leads, as well as is known, to misinterpretations, as shown in Fig. 15, when an object, represented here as a circle, is located at a distance multiple of the uniqueness range. While the object represented by a rhombus is always measured with the same raw distance value m in all three distance measurements performed at different frequencies f, the raw distance values ​​vary for objects outside the uniqueness ranges. The distance between these objects can then be determined in a simple manner, as shown in the procedure above, using the modulo limit comparison according to step f). This can be done, for example, as follows. If, for instance, a first uniqueness area EB1 extends by 5 m and a second uniqueness area EB2 by 7 m, and the object is located at a distance of 17 m at time t1, then the measured value m1(t1,f1) at the first time will be 2 m = 17 (mod 5). On the scale of the second uniqueness area EB2, 3 m = 17 (mod 7) would be expected. If, for example, a measurement m2(t2,f2) = 4 m is taken at a second time t2, and a distance value d' of 3.2 m + k*EB2 is obtained when calculating back to the first time t1, the following values ​​result in the above modulo comparison (the modulo comparison uses the circular distance here, see k=1): contemplation In this example, the measuring range MB could be set to 20 m with dmax, so that the k-values ​​are traversed up to a maximum value of 3. If the limit value GW is set to 0.4 m, for example, the distance d'ij with k=2 would be plausible. At time t2, the distance value dij(t2,f2,k) = 4 m + 2*7m = 18 m could then be output as valid. The following examples assume the use of 3 different frequencies. Variations with a different number of frequencies are also possible. Furthermore, the observation of extended planes that are not parallel to the image sensor (e.g. ceiling, floor, walls) using frequencies that are monotonically decreasing or increasing over time can also lead to falsely consistent measurements with certain Ego data. To ensure these cases are also covered and to verify the measurement, the invention provides, as shown in Fig. 16, for a fourth measurement to be taken, which uses either the first or the second frequency. In the example shown, the second frequency f2 was used. This avoids frequency sequences that are monotonically decreasing or increasing over time. If the fourth measurement confirms the distance result of the previous measurements, a valid distance value is obtained. Validation with more than two frequencies can be achieved by repeatedly applying the procedure described above, in which the current frame is compared with the 3 previous frames and pixels are only output as valid if all comparisons with the previous frames are valid. To save computational effort, it is also possible to introduce a counter C_ij(t_n) for each pixel, which counts the number of valid comparisons. When a valid comparison of pixel P_ij(t_n) with pixel P_pq(t_n-1) occurs, the counter of the current frame is then set to the value of the previous frame incremented by one. These counters thus count the length of the "track" in frames minus one. A pixel is valid if the number of valid comparisons is at least the number of frequencies used. In our example with a set of three frequencies fsmit smax= 3, the counter would have to be at least 3, which corresponds to a track length of 4 frames. Fig. 17 schematically illustrates this procedure for the stationary case. The first acquisition frame tn cannot be compared with a predecessor frame; therefore, the counter cij is set to 0. Although a raw distance value m can be determined, no distance value dij can be calculated due to the lack of comparison with the predecessor frame. The subsequent acquisition frames tn+1, tn+2, and tn+3 provide valid distance values ​​when compared with their respective predecessor frames tn, tn+1, and tn+2. Before the fourth time frame, tn+3, two valid distance values ​​had already been determined, so the counter from the previous frame was set to 2. Thus, the counter has reached the number of frequencies minus 1. In the fourth acquisition frame tn+3, a frequency fs from the frequency set (f1, f2, f3) is measured, which differs from the previous frequency fs+2. If a valid distance value d(tn+3) is also found in this fourth acquisition frame tn+3, the current counter is incremented to 3 and the current valid distance value is output as the valid distance value d_out. In the following fifth acquisition frame n+4, a valid distance value is also available and c > smax so that the valid distance value can be output. In the following sixth acquisition frame n+5, no valid distance value is present and the counter is set to zero and no valid distance value is output. In the subsequent acquisition frames n+ ..., valid distance values ​​are again available, so that after three valid distance values ​​a valid distance value can be output again. The procedure can also be carried out using coded or pseudo-noise modulation. So-called maximum sequences are particularly suitable for this purpose. When using such modulation, it is usually sufficient to use two different frequencies. An iteration over the uniqueness ranges can then potentially be omitted. The measurement range would then be the intersection of the two pseudo-noise measurement ranges. Furthermore, it may be possible to stabilize the vehicle's speed using time-of-flight data when integrating a gyroscopic sensor. Furthermore, it should be noted that for the mapping from k' to d', a single pixel is considered with k'ij = x', y', z', taking into account an optical model OM and, if necessary, an extrinsic calibration. Thus, the result OM(x',y',z') is a pixel coordinate r,c. Here, r,c are row and column floating point coordinates on the sensor. The following methods can be used for evaluation: a) Nearest neighbor mapping. The comparison is made using the integer coordinates closest to (r,c). b) Lateral relaxation. The comparison is made using a lateral neighborhood of (r,c) and is valid if at least one pixel in the neighborhood meets the criterion. If multiple pixels meet the threshold criterion, the pixel whose track counter shows the highest value and whose track counter is incremented is used. c) Bilinear interpolation. The comparison is made using bilinear interpolation around (r,c). Furthermore, it would be conceivable to use the transformed distances for temporal filtering (e.g., with a recursive filter). For this, the distances would need to be transformed back into the future. Reference symbol list 10 Lighting module 12 Lighting 22 Light time-of-flight sensor 27 Evaluation unit 30 Modulator 35 Phase shifter, lighting phase shifter 38 Modulation control unit Δφ(tL) time-of-flight phase shift φvar Phase position φ0 Base phase M0 Modulation signal p1 First phase p2 Second phase Sp1 Transmit signal with first phase Sp2 Receive signal with second phase Ga, Gb Accumulation gates Ua, Ub Voltages at the modulation gate f1, f2, f3 First, second, third modulation frequency λ Wavelength PMi Phase measurement cycle Mi Distance measurement cycle D Total path length d Object distance dij Determined object distance

Claims

Method for operating a time-of-flight camera system with a time-of-flight sensor consisting of an array of time-of-flight pixels on a vehicle, in which a movement of the vehicle is detected and made available for calculations, comprising the steps: a) Determining raw distance values ​​(mij(tn,fs)) to a first detection frame (tn) and with a first modulation frequency (fs), b) Converting the raw distance values ​​(mij(tn,fs)) into distance values ​​(dij(tn,fs,k)) under the condition d ij ( tn , fs , k ) = m ij ( tn , fs ) + k * EB ( fs ) with a k-value as a counter from 0 to a maximum k-value and a frequency-dependent uniqueness range (USR), c1) Conversion of the distance values ​​(d ij (t n ,f s ,k)) for a given pixel coordinate (ij) into coordinate values ​​(k ij (t n ,f s ,k)) c2) Back-calculation of the coordinate values ​​(k ij (t n ,f s ,k)) in coordinate values ​​(k' ij (t n-1 ,f s ,k)) to a previous time point (t n-1 ) taking into account the movement (TΔt) of the vehicle that took place between the two times (t1, t2, Δx), c3) Conversion of the back-calculated coordinate values ​​(k' ij (t n-1 ,f s ,k)) into back-calculated distance values ​​(d' pq (t n-1 ,f s ,k)) and pixel coordinates (pq) of a previous capture frame (t n-1 )., d) Limit comparison of the back-calculated distance value d' pq (t n-1 ,f s ,k) with a corresponding raw distance value m pq (t n-1 ,f s-1 ) of the same pixel Pix pq according to the rule: | d ′ pq ( tn − 1 , fs , k ) − m pq ( tn − 1 , fs − 1 ) | ( mod EB ( tn − 1 , fs − 1 ) ) < GW e) Output of the distance value d determined in step b). ij (t n ,f s ,k) as valid if a k-value exists at which the limit value GW is undercut, taking into account the determined k-value, f) Repeat the above steps for the next capture frame (t n+1 ) with a next frequency (f s+1 ). Method according to claim 1, wherein different modulation frequencies (fn) are used in directly successive acquisition frames (tn, tn+1). Method according to one of the preceding claims, wherein the modulation frequencies (fs) are taken from a set of predetermined modulation frequencies and the set has a predetermined maximum number (smax) of modulation frequencies. Method according to one of the preceding claims, wherein for an output of a valid distance value (dij) the distances are determined over a number of acquisition frames (tn) that is greater than the number (smax) of modulation frequencies specified in the modulation frequency set (fs). Method according to one of the preceding claims, wherein an incremental counter (Cij) is provided which sums the acquisition frames with valid distance values, wherein the counter (Cij) is set to zero as soon as an acquisition frame has an invalid distance value. Method according to one of the preceding claims, wherein, starting from a pixel coordinate (pq) which results from the back-calculated distance value (d'pq), neighboring pixels are also subjected to the limit comparison according to step d) of claim 1. Method according to claims 5 and 6, wherein, if several pixels fall below the limit value, the pixel whose incremental counter (Cuv) shows the highest value is used and the incremental counter (Cuv) of this pixel is increased by one. Time-of-flight camera system (1) with a modulator (30) connected to an illumination (10) and a receiver (20) of the time-of-flight camera system (1), a modulation control unit (38) connected to the modulator (30) and configured such that the modulator (30) can be operated with at least two modulation frequencies (f1, f2, f3), characterized in that the time-of-flight camera system (1) is configured to carry out a method according to one of the preceding claims.

Images