Multipath and noise reduction for time-of-flight cameras
Patent Information
- Application Number
- DE102018122686
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2018-09-17
- Publication Date
- 2025-02-06
- Estimated Expiration
- 2038-09-17
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
The invention relates to a time-of-flight, ToF, camera for measuring distance information for objects of a scene and to a corresponding ToF method.BACKGROUND OF THE INVENTIONLight propagates at a speed of approximately 300,000 kilometers per second. By measuring the transit time of the light between two points very accurately, it is possible to calculate the distance between them. This principle forms the basis of the distance measurement methods which are used in so-called time-of-flight, ToF, cameras (sometimes also time-of-flight cameras). A ToF camera includes a light source and an image sensor. The light source emits a modulated light signal which is reflected at the objects located in the scene, and the ToF camera measures the time between the emission and the arrival of the reflected light signal at the image sensor. For each pixel (or for a subset of the pixels) of the image sensor, a distance of the light-reflecting objects calculated from the measured transit time is then output to the ToF camera.In addition to the distance values, some ToF cameras additionally also generate color or brightness values (and optionally further values such as confidence values or the like), which can then likewise be output. ToF cameras, which are often also referred to as "active cameras" because they are equipped with their own light source, are used in a large number of applications, for example in robotics, industrial automation, logistics and medicine, and in various areas of "smart factory".The accuracy of the ToF method depends on how precise the time measurement is. Since the delay time of the light is only about 3.3 ns per meter, the resolution of the time measurement must be correspondingly high. In the so-called pulse modulation, the light source of the ToF camera emits a light pulse at a time t 0 and simultaneously starts a highly precise time measurement. If the light reflected at an object of the scene then arrives in the camera at a time t 1 the distance to the object can be calculated directly from the measured travel time t 1- t 0 as d=c / 2·(t 1- t 0) where c indicates the speed of light. Alternatively, however, a sinusoidal modulation of the light signal can also be used, wherein the distance values are then derived from the phase shift between the outgoing and incoming light signals. This method is also referred to in the literature as a continuous wave (CW) method.In practice, both pulse modulation and CW methods and mixed forms of both are used:For example, in a known method, a rectangular light pulse of duration t p is emitted and at the same time a first electronic shutter ("shutter") is opened for the duration of the light pulse. The reflected light arriving in the ToF camera at this time is stored as a first electric charge S 1. The first shutter is now closed and a second shutter is likewise opened for the duration t p at the time of switching off the light source. The reflected light arriving in the ToF camera at this time is stored as a second electric charge S 2. Since the light pulse is very short, this operation is repeated several thousand times until the set exposure time is over. Subsequently, the integrated electric charges S 1 and S 2 are read out.As a result, two partial measurement images are obtained which each contain the integrated electric charge S 1 or, respectively, for each image point. S 2. In the S 1- partial measurement image, the near objects of the scene are brighter, because with increasing distance, less and less reflected light reaches the ToF camera as long as the first shutter is still open. In the case of the S 2- measurement, on the other hand, it is exactly the reverse. Close objects are dark here, since the second shutter only opens when the light has already traveled a while. The ratio of the integrated electric charges S 1 and S 2 thus changes depending on the distance travelled by the emitted and reflected light. The distance to the object can thus be determined as being for each image point, wherein c in turn specifies the speed of light. In this method, the distance measurement is thus based on the measurement of the phase position of reflected rectangular pulses.The smallest measurable distance is measured when all charge during the early shutter time is stored as S 1 and no charge during the delayed shutter time is stored as S 2 i.e., S 2= 0. Equation (1) then yields d=0. In contrast, the largest measurable distance is measured just when all charge is stored as S 2 and no charge is stored as S 1 i.e., S 1= 0. In this case, equation (1) also yields from this it can be seen that the light pulse width determines the maximum measurable distance in this method. If t p is 47 nanoseconds, for example, distances of 0 to 7 meters can be measured.In addition to the precision of time measurement, there are a plurality of influencing factors which have effects on the distance measurements of a ToF camera and can restrict the measurement accuracy. Thus, in the case of the aforementioned methods, it is generally assumed that the modulated light signals are completely sinusoidal or rectangular. In practice, however, this applies only approximately because of the hardware, wherein the deviation of the real signals from the theoretical assumption leads to a systematic phase error φ w (also referred to in the literature as "wiggling" error). Further sources of error are the so-called phase noise, which is independent of the modulation frequency, and multipath effects. The latter are due to the fact that in reality not only is the once reflected light required for the distance measurement (which travels directly from the camera to an object of the scene and back again) measured, but that the actually measured light is typically adulterated by the superposition of a plurality of scattered light components reflected at further objects. This also leads to a incorrectly calculated phase shift and thus to incorrect distance measurements-for example objects in the scene can be represented in the measured distance image (often also referred to as "depth image") distorted or displaced and planar surfaces can appear curved.One variant of the multipath effects is the so-called scattered light. Incident modulated light is thereby scattered by the objective, primarily by the lens surfaces. Thus, for example, light from very well illuminated objects in the near range can be scattered onto pixels of the image sensor which are actually intended to image other objects at a greater distance. The superposition of the light paths therefore takes place here on the pixels of the image sensor.In view of these problems, it would therefore be desirable to provide a ToF camera and a ToF method in which the influence of one or more of the, preferably all, aforementioned error sources on the distance measurements can be reduced as much as possible.DE 100 39 422 C2 discloses a method for operating a PMD system, in which a PMD is driven by means of at least one modulation signal, a transmitter emits electromagnetic radiation, which is intensity-modulated by means of the at least one modulation signal, wherein the modulation signal is modulated between at least two modulation frequencies.A. D. Payne et al., "Multiple Frequency Range Imaging to Remove Measurement Ambiguity," 9 th Conference on Optical 3-D Measurement Techniques, Vienna, Austria, July 2009, discloses that range imaging systems use a dedicated sensor to capture an image in which the object distance (range) for each pixel is measured using the time of flight. The scene is illuminated with an amplitude modulated light source and the phase of the modulation envelope of the reflected light is measured to determine the time of flight and hence the object distance for each pixel. However, since the modulation waveform is cyclic, there is an ambiguity problem when the phase shift exceeds 2π. To solve this problem, a method is used which superimposes two different modulation frequencies within a single shot.SUMMARY OF THE INVENTIONThe object of the invention is to provide a time-of-flight, ToF, camera which makes it possible to reduce the influence of one or more error sources, e.g. wiggling errors, phase noise or multipath effects (including scattered light), on the distance measurements. Furthermore, the object of the invention is to provide a corresponding ToF method.According to a first aspect of the invention, there is provided a time-of-flight, ToF, camera for measuring range information for objects of a scene, the ToF camera comprising:a light source for emitting modulated light signals for illuminating the objects;an image sensor for detecting light signals reflected at the objects;a processing unit for processing first sections of the reflected light signals modulated with oscillations of a first frequency, for determining first distance measurement values in a first distance measurement value range dependent on the first frequency, and for processing second sections of the reflected light signals modulated with oscillations of a second frequency, for determining second distance measurement values in a second distance measurement value range dependent on the second frequency, wherein the second frequency is higher than the first frequency and the second distance measurement value range is smaller than the first distance measurement value range, wherein the processing unit is adapted to resolve ambiguities of the second distance measurement value range within the first distance measurement value range on the basis of the first distance measurement values and to generate a distance measurement image on the basis of only the second distance measurement values; andan output and / or memory unit for outputting and / or storing the distance measurement image, wherein the oscillations of the first frequency and / or the oscillations of the second frequency are periodic sequences of rectangular pulses, preferably with a pulse-pause ratio of 1 / 2, wherein the ToF camera is adapted to generate a plurality of partial measurement images at different phase positions from the first sections of the reflected light signals and / or from the second sections of the reflected light signals by integrating the reflected light signals over a plurality of integration phases offset in time with respect to one another, wherein the first distance measurement values and the second distance measurement values are determined from the partial measurement images, wherein three partial measurement images are generated from the second sections of the reflected light signals and only one partial measurement image is generated from the first sections of the reflected light signals.The invention is based on the finding by the inventor that, by modulating the light signals used for measurement with a higher frequency, not only does the range of distance measurement values reduce in comparison with a lower modulation frequency, but at the same time this leads to an advantageous reduction of the wiggling error, the phase noise and the multipath effects (including the scattered light), namely in comparison with a lower modulation frequency, by a factor which substantially corresponds to the ratio between the two frequencies. On the other hand, the reduced range of distance measurement values has the disadvantage that it permits unambiguous distance measurement only for scenes with a correspondingly smaller depth. For deeper scenes, ambiguity occurs in distance measurement due to the periodicity of the modulation. The invention solves this conflict by providing range measurements with two different modulation frequencies: a lower first frequency in first portions of the modulated light signals and a higher second frequency in second portions of the modulated light signals. In the measurement at the first frequency, the wiggling error, the phase noise, and the multipath effects (including the stray light) are larger. However, due to the larger range of distance measurement values, unique distance measurement values can be determined for a deeper scene. These first distance measurement values are then used according to the invention in order to resolve the ambiguities (repetitions of the second distance measurement value range) during the measurement with the second frequency on the basis thereof. In this way, unique second range measurement values can be determined with a reduced wiggling error, phase noise, and multipath effects (including stray light). As a result of the ToF distance measurement according to the invention, a distance measurement image can then be output and / or stored, which has been generated on the basis of only the second distance measurement values.The light source advantageously emits the modulated light into the field of view of the ToF camera. Here, for example, LEDs (light emitting diodes) or LDs (laser diodes) can be used. In many applications, infrared light is used as illumination. This has the advantage that it is visually non-noticeable and, in particular, additionally influences color or brightness values to be recorded not at all or only to a slight extent.The advantage of using pulse-modulated light signals (periodic sequences of square-wave pulses) is that a high energy can be emitted in a very short time. Thus, the influence of the backlighting, which can likewise impair the ToF distance measurement, can be reduced and the signal-to-noise ratio can be improved, since the recording time and thus the time range for the integration of the backlighting can be reduced by a high temporal and spatial energy density. However, low average powers are achieved at the same time. This is especially important for eye safety, which is often the limiting criterion for many measurement applications. Moreover, normal CCD or CMOS sensors can be advantageously used, which are more cost-effective in production and have a large light-sensitive area. The use of a pulse-to-pause ratio (also referred to in the literature as duty cycle, duty cycle or duty cycle) of 1 / 2 additionally has the advantage that two or more repetitions of the second range of distance measurement values are directly adjacent to one another over the first range of distance measurement values in each case without an overlap. In this way, neither gaps arise in the depth at which measurement at the higher second frequency is not possible nor do the repetitions of the second distance measurement value range overlap, which would complicate the selection of the "correct" second distance measurement value range on the basis of the determined first distance measurement values.The determination of a distance measurement value based on a modulated light signal can in principle be effected using two fields, as described above. In addition to the distance of an object to be measured, there are also two further unknowns in the ToF method, namely the reflectivity of the object and the background illumination. Since there are thus a total of three degrees of freedom, it is advantageous to use more than two partial measurement images in the distance measurement. For example, three partial measurement images can be generated in each case both from the first sections of the reflected light signals and from the second sections of the reflected light signals. This has a total of six partial measurement images (three for the lower first frequency and three for the higher second frequency), by means of which a direct solution to the problem on the basis of the integrated electrical charges can be obtained. The integration phases offset in time with respect to one another can be realized in this case via three electronic shutters which are opened at corresponding times. The integration phases advantageously correspond in each case to the duration of the square-wave pulses and / or the time offset corresponds in each case to two thirds of the duration of the square-wave pulses.In the present invention, three sub-measurement images are generated from the second portions of the reflected light signals, and only one sub-measurement image is generated from the first portions of the reflected light signals, so that a total of four sub-measurement images (one for the lower first frequency and three for the higher second frequency) is provided. If it is also assumed that the reflectivity of the objects and the background illumination do not change between the measurement with the lower first frequency and the measurement with the higher second frequency, but if one wishes to have an independent distance measurement for each frequency, the number of degrees of freedom for the two frequencies is reduced from two times three equal to six to only four degrees of freedom. The problem on the basis of four partial measurement images can thus be clearly solved. As described, since the measurement at the higher second frequency reduces the wiggling error, the phase noise, and the multipath effects (including the stray light), better results are obtained at less expense if the "unnecessary" partial measurement images are saved at the lower first frequency.According to an advantageous embodiment, the light source is further adapted to emit the second sections to be processed for determining the second distance measurement values temporally before the first sections to be processed for determining the first distance measurement values. In many cases, it is namely desired to synchronize the distance measurement with an external event (for example the supply of a measurement object in applications of industrial automation). For this purpose, it must be able to control the measurement time as accurately as possible, for example by means of a trigger signal. Since, according to the invention, the distance measurement image is generated on the basis of only the second distance measurement values, it is therefore advantageous if the second sections of the modulated light signals to be processed for determining the second distance measurement values are emitted temporally before the first sections of the modulated light signals to be processed for determining the first distance measurement values, since it can thus be ensured that the measurement time point of the second distance measurement values is as close as possible to the desired trigger time point.According to another advantageous embodiment, the light source is further adapted to emit the first sections to be processed for determining the first distance measurement values and the second sections to be processed for determining the second distance measurement values in a temporally interleaved sequence. A problem in distance measurement can arise if an object in the scene moves during the distance measurement, wherein lateral or up or down movements can be problematic in particular. If, for example, a portion of the scene background is uncovered by such a movement of a foreground object during the distance measurement, the resolution of the ambiguities according to the invention may fail if, for example, the determined first distance measurement values still represent the foreground object, but the determined second distance measurement values would already have to indicate the distance of the scene background. This can be avoided by the first sections to be processed for determining the first distance measurement values and the second sections to be processed for determining the second distance measurement values being emitted in a temporally interleaved sequence, so that the measurement time of the first distance measurement values and the measurement time of the second distance measurement values are substantially the same.In this way, greater movement artifacts can be avoided or at least reduced at the expense of a possibly slightly increased movement blur.According to yet another advantageous embodiment, the ToF camera is adapted to operate the light source during the emission of the modulated light signals during the second sections with a longer exposure time and / or with a higher current or a higher voltage than during the first sections. The reason for this is the differences in the operating point of the ToF camera at the frequencies of different magnitude. In particular, due to the finite rise times of drivers and the light source (e.g. LED or LD), the light source can only emit a lower light power at the higher second frequency. This results in higher noise. Since the distance measurement with the higher second frequency carries the detail information, it is advantageous to compensate this effect as much as possible. This can be done by using a longer exposure time and / or by operating the light source with a higher current or a higher voltage during the second sections.According to yet another advantageous embodiment, the processing unit is adapted to perform a correction with respect to a frequency-dependent offset value when determining the first distance measurement values and / or the second distance measurement values. The use of higher frequencies is typically associated with a higher power loss of the electronics. The (briefly) higher temperature of the components ensures other signal transit times, for example in the drivers. These times add up to the light propagation times to be measured and distort the result. This effect can be eliminated or at least reduced by a suitable calibration of the propagation time difference and a correction by means of a frequency-dependent offset value.According to an advantageous embodiment, when resolving the ambiguities, in each case that repetition of the second distance measurement value range within the first distance measurement value range is selected for which the second distance measurement value lying therein comes closest to the associated first distance measurement value. In this way, the "correct" second distance measurement value can also be determined very robust under the influence of wiggling errors, phase noise and multipath effects (including stray light).According to an advantageous embodiment, the second frequency is an integer multiple of the first frequency. In this case, the larger first distance measurement value range can just correspond to the sum of two or more repetitions of the smaller second distance measurement value range. The first distance measurement value range can thus be covered by an integer number of repetitions of the second distance measurement value range, so that the further end of the first distance measurement value range coincides with the further end of one of the repetitions of the second distance measurement value range. This simplifies processing in selecting the "correct" second range of distance measurement values in resolving the ambiguities.According to an advantageous embodiment, the processing unit is further adapted to process third sections of the reflected light signals, which are modulated with oscillations of a third frequency, for determining third distance measurement values in a third distance measurement value range dependent on the third frequency, wherein the third frequency is higher than the first frequency and lower than the second frequency and the third distance measurement value range is smaller than the first distance measurement value range and larger than the second distance measurement value range, wherein the processing unit is adapted to resolve ambiguities of the third distance measurement value range within the first distance measurement value range on the basis of the first distance measurement values when determining the third distance measurement values and to resolve ambiguities of the second distance measurement value range within the third distance measurement value range on the basis of the third distance measurement values when determining the second distance measurement values.By using such "cascading" of different high frequencies, there is an advantageous robustness against pronounced phase noise, multipath effects (including stray light) and motion artifacts. In addition to the additional third frequency, one or more further frequencies can of course also be provided which lie between the first and the second frequency (for example below the third frequency or above the same or also partly below and partly above). The selection of the "correct" range of distance measurement values dependent on a respective frequency when resolving the ambiguities then takes place in each case on the basis of distance measurement values determined for a smaller frequency, preferably the next lower frequency.According to an advantageous embodiment, the ToF camera furthermore comprises an interface for parameterizing the ToF camera, wherein a number of frequencies used for the distance measurement and / or a ratio of at least two of the first frequency, the second frequency and, if provided, the third frequency can be set by the parameterization. In this way, the ToF camera can be very flexibly adapted to the prevailing circumstances, e.g. the expected motion artifacts, etc.According to another aspect of the invention, there is provided a time-of-flight, ToF, method for measuring range information for objects of a scene, the ToF method comprising:emitting modulated light signals for illuminating the objects;detecting light signals reflected on the objects;processing first portions of the reflected light signals modulated with oscillations of a first frequency for determining first distance measurement values in a first distance measurement value range dependent on the first frequency, and processing second portions of the reflected light signals modulated with oscillations of a second frequency for determining second distance measurement values in a second distance measurement value range dependent on the second frequency, wherein the second frequency is higher than the first frequency and the second distance measurement value range is smaller than the first distance measurement value range, wherein the processing, when determining the second distance measurement values, resolves ambiguities of the second distance measurement value range within the first distance measurement value range on the basis of the first distance measurement values and generates a distance measurement image on the basis of only the second distance measurement values; andoutputting and / or storing the distance measurement image, wherein the oscillations of the first frequency and / or the oscillations of the second frequency are periodic sequences of rectangular pulses, preferably with a pulse-pause ratio of 1 / 2, wherein the ToF method generates a plurality of partial measurement images at different phase positions from the first sections of the reflected light signals and / or from the second sections of the reflected light signals by integrating the reflected light signals over a plurality of integration phases offset in time with respect to one another, wherein the first distance measurement values and the second distance measurement values are determined from the partial measurement images, wherein three partial measurement images are generated from the second sections of the reflected light signals and only one partial measurement image is generated from the first sections of the reflected light signals.It is understood that the ToF camera according to claim 1 and the ToF method according to claim 11 have similar and / or identical preferred embodiments, in particular as defined in the dependent claims.It is to be understood that a preferred embodiment of the invention may also be any combination of the dependent claims with the corresponding independent claim.BRIEF DESCRIPTION OF THE DRAWINGSPreferred embodiments of the invention are described in more detail below with reference to the attached figures, in which FIG. 1 shows schematically and by way of example the structure of a ToF camera, FIG. 2 shows schematically and by way of example how the modulation of the light signals can preferably be realized, FIG. 3 shows, schematically and by way of example, phase diagrams illustrating the multipath effects at different frequencies, FIG. 4 shows schematically and by way of example how an infinite number of indirect paths are generated, which differ only slightly, by an area parallel or approximately parallel to the direct path, FIG. 5 shows schematically and exemplarily the multipath effects in the situation shown in FIG. 4, and FIG. 6 shows schematically and by way of example how the resolution of the ambiguities of the second distance measurement value range within the first distance measurement value range can be realized in a preferred manner on the basis of the first distance measurement values.In the figures, the same or corresponding elements or units are each provided with the same or corresponding reference numerals. If an element or a unit has already been described in connection with a figure, a detailed illustration is omitted, if appropriate in connection with another figure.FIG. 1 shows schematically and by way of example the structure of a ToF camera 10 for measuring distance information for objects 14 of a scene, as can be used, for example, in robotics, industrial automation, logistics and medicine, or in various areas of "smart factory". The ToF camera 10 comprises a light source 11 for emitting modulated light signals 15 for illuminating the objects 14, and an image sensor 12 for detecting light signals 16 reflected at the objects 14.The light source 11 advantageously emits the light into the field of view of the ToF camera 10. Here, for example, LEDs (light emitting diodes) or LDs (laser diodes) can be used. In this embodiment, infrared light is used as illumination. This has the advantage that it is visually non-noticeable and, in particular, additionally influences color or brightness values to be recorded not at all or only to a slight extent.How the modulation of the light signals 15 can preferably be realized will now be described with reference to FIG. 2. In this figure, the modulated light signals 15 emitted by the light source 11 for illuminating the objects 14 of the scene are shown in the uppermost row. It can be seen from the illustration that the modulated light signals 15 comprise first sections A 1 which are modulated with oscillations of a first frequency f 1 and second sections A 2, which are modulated with oscillations of a second frequency f 2. The three points between the respective first and second portions A 1 and B. A 2 indicates that the sections repeat many times, for example several thousand times, until the respective set exposure time is over.In this embodiment, the oscillations are square pulses with a pulse-to-pause ratio of 1 / 2 and the second frequency f 2 is higher than the first frequency f 1. In particular, the second frequency f 2 in this embodiment is a integer multiple N, here four times, of the first frequency f 1. The first sections A 1 each comprise only one square-wave pulse and the associated pause, while the second sections A 2 each comprise a periodic sequence consisting of N square-wave pulses and the associated pauses. The modulated light signals 15 with the first and second sections A 1 and A 2 are reflected by the objects 14 of the scene and the reflected light signals 16 are detected by the image sensor 12.The ToF camera 10 shown in FIG. 1 further comprises a processing unit 13 for processing the first and second sections A 1 and A 2 of the reflected light signals 16 for determining first and second distance measurement values d 1 and d 2. In this case, the first distance measurement values d 1 lie in a first distance measurement value range D 1 and the second distance measurement values d 2 lie in a second distance measurement value range D 2. This is shown in the lowermost row of Fig. 2. As can be seen, the two distance measurement value ranges D are 1 or dependent. D 2 in each case depends on the frequency f 1 or f 2 respectively, wherein, on account of the second frequency f 2 which is higher in comparison with the first frequency f 1 the second distance measurement value range D 2 is smaller--namely likewise by a factor of four--than the first distance measurement value range D 1. As an example, assume here that the duration t p1 of the square pulses in the first sections A is 147 nanoseconds, and that the duration t p2 of the square pulses in the second sections A is 2 a quarter thereof, i.e. 11.75 nanoseconds. Then, for example, 7 meters can be measured with the first range of distance measurement values D 1 while only 1.75 meters can be measured with the second range of distance measurement values D 2.The use of the second frequency f 2 in the second sections A 2 which is higher by a factor N=4 compared to the first frequency f 1 in the first sections A 1 has the result that the second distance measurement value range D 2 is ambiguous within the first distance measurement value range D 1. In other words: the second distance measurement value range D 2 repeats within the first distance measurement value range D 1, and it is not possible to unambiguously determine from the phase position of the rectangular pulses in the second sections A 2 of the light signals 16 reflected by the objects 14 of the scene that repetition of the second distance measurement value range D 2 in which a respective object 14 is located. To remain in the above example, it cannot be unambiguously determined whether the object 14 is in a range of ranging values from 0 to 1.75 meters (first repetition), or in a range of 1.75 to 3.5 meters (second repetition), or in a range of 3.5 to 5.25 meters (third repetition), or in a range of 5.25 to 7 meters (fourth repetition).The processing unit 13 of the ToF camera 10 is now adapted according to the invention to resolve the ambiguities of the second distance measurement value range D 2 within the first distance measurement value range D 1 on the basis of the first distance measurement values d 1 when determining the second distance measurement values d 2 and to generate a distance measurement image 18 (see FIG. 1 ) on the basis of only the second distance measurement values d 2. The ToF camera 10 further comprises an output and / or storage unit 17 for outputting and / or storing the distance measurement image 18.Before we explain in more detail with reference to FIG. 6 how the resolution of the ambiguities can be realized in a preferred manner, the advantages of the described procedure will first be explained in more detail. As mentioned, the invention is based on the discovery by the inventor that, although the second range of distance measurement values D 2 is reduced on the one hand by the use of the higher second frequency f 2 in the second sections A 2 of the modulated light signals 15, this simultaneously leads to an advantageous reduction of the wiggling error, the phase noise and the multipath effects (including the scattered light), namely by a factor N which substantially corresponds to the ratio between the two frequencies f 1 and f 2( i.e. four here), in comparison with the lower first frequency f 1 in the first sections A 1 of the modulated light signals 15.To make this matter relevant, we once again refer to Figure 1, in which a wall or other larger object 19 is shown in the scene next to the objects 14. As described, the light source 11 emits the modulated light signals 15 for illuminating the objects 14, and the image sensor 12 detects the light signals 16 reflected on the objects 14, while the shown light signals 15 travel directly from the light source 11 to the objects 14 and from there to the image sensor 12, it is also possible that at least a part 20 of the modulated light signals 15 emitted by the light source 11 is reflected on the wall or the larger object 19 and from there travel indirectly to the objects 14 and back to the image sensor 12. This light portion 20 reflected indirectly is superimposed with the light 15 reflected directly, i.e. the image sensor 12 detects a superposition of light that has passed on different paths ("multipath") from the light source 11 to the image sensor 12. As explained, this leads to a incorrectly calculated phase shift and thus to incorrect distance measurements ("multipath effects").This effect can also be described for the light signals 15 modulated with square-wave pulses approximately on the basis of the phase diagrams illustrated in FIG. 3. It is assumed here that the phase shift φ of the reflected light signals 16 impinging on the image sensor 12 is proportional to the distance travelled, i.e. the distance to the respective object 14 is c again indicating the speed of light and f the modulation frequency. FIG. 3(a) shows the signal S= Ξe iφ, generated by the image sensor 12, wherein the amplitude Ξcorresponds to the brightness (generated charge) and φ is a measure of the object distance d. If the image sensor 12 also receives light from the object 14 via an indirect path, a direct signal S D and an indirect signal S I are superimposed to form a total signal S=S D+ S I. This is shown in Fig. 3(b). If all possible phase positions are now allowed for the indirect signal S I (it is therefore no matter how long the indirect path is), but if a maximum amplitude ≅ I is assumed, then the total signal S lies within the circle shown in FIG. 3(c). The angle φ err then describes the maximum measurement error of S vs. S D, which is caused by the indirect path.If the modulation frequency f is now N-fold, the phase angle φ of the direct signal S D, is also N-fold, since the causal signal transit times remain unchanged. However, the area of the circle for the measurement error caused by the indirect path does not change, since it is defined only by the maximum amplitude ≅ I of the indirect signal S I. This is shown in Fig. 3(d). If, for comparison of the two cases, the phase angle φ of the direct signal S D at the higher frequency is now divided by the ratio N of the two frequencies, the case from FIG. 3(d) is clearly mapped onto the phase angle φ of the direct signal S D at the lower frequency (the case from FIG. 3(c)). As a result, as shown in Fig. 3(e), the error circle becomes similar to an ellipse with its area decreasing by a factor of 1 / N. However, the maximum measurement error caused by the indirect path is then also reduced accordingly if the distance measurement is falsified by light received on a plurality of indirect paths.A particularly favorable reduction of the multipath effects is also obtained in the described procedure if an infinite number of indirect paths are generated by a surface parallel or approximately parallel to the direct path, which differ only slightly. This is shown schematically and exemplarily in FIG. 4, wherein the direct path 21 runs directly between the ToF camera 10 and the objects 14 of the scene and the indirect paths run over the wall 23.In such a situation, the image sensor 12 thus receives, in addition to the light signal 16 reflected directly at the respective object 14, a superposition of the infinite number of light signals 22 reflected via the wall 23 on the indirect paths. If this is now compared with the case in which the same amount of light is reflected via a single indirect path (not shown in FIG. 4 ) via the center of the wall 23, it is recognized from FIG. 5 ( a) that the amplitude ≅ IS of the total interference vector, which results from the superposition of the light signals 22, is reduced in comparison with the amplitude ≅ I of the light signal received via the single indirect path. This result is clearly obtained if it is considered that the resulting phase angle φ IS of the superposition corresponds to the phase angle φ I of the individual indirect signal, and that the amplitude ≅ IS is obtained as the average value of the indirect signals, in each case mapped to the resulting phase angle φ IS= φ I. Thus, ≅ IS is always smaller than ≅ I, the reduction being the larger the phase range φ F is (cf. FIG. 5( b)).As mentioned, the phase range φ F is initially dependent on the extent of the wall 23. An increase in the modulation frequency f in the case described, in which an infinite number of indirect paths are generated by a surface parallel or approximately parallel to the direct path, thus leads to a particularly favorable reduction of the multipath effects.In addition to the reduction of multipath artifacts explained above, the described approach correspondingly reduces the wiggling error and the phase noise. As already stated in the introduction to the description, in practice the oscillations of the modulated light signals are only approximately sinusoidal or rectangular due to hardware, wherein the deviation of the real signals from the theoretical assumption leads to a systematic phase error φ w( "wiggling"). This error does not depend on the modulation frequency f, but is formed on the range of distance measurement values D dependent on the frequency. The standard deviation of the Wiggling error at a N-fold of the modulation frequency f thus results in where σ w indicates the standard deviation of the Wiggling error at the lower frequency. The same also applies to the phase noise, which is also independent of the modulation frequency f, but which maps itself, as it were, to the range of distance measurement values D dependent on the frequency. Here too, the N-fold of the modulation frequency f thus results in a reduction of the standard deviation of the phase noise of where σ P indicates the standard deviation of the phase noise at the lower frequency.In the following, FIG. 6 schematically and exemplarily shows how the resolution of the ambiguities of the second distance measurement value range D 2 within the first distance measurement value range D 1 can be realized in a preferred manner on the basis of the first distance measurement values d 1. For this purpose, the first range of distance measurement values D 1 already shown in the lowermost row of FIG. 2 and--in the illustration below--the ambiguous second range of distance measurement values D 2 within the first range of distance measurement values D 1 are shown once again in the figure. As explained, the second distance measurement value range D 2 repeats within the first distance measurement value range D 1, and it is not possible to unambiguously determine from the phase position of the rectangular pulses in the second sections A 2 of the light signals 16 reflected by the objects 14 of the scene that repetition of the second distance measurement value range D 2 in which a respective object 14 is located. This does not only perform the ambiguity resolution according to the invention.Since the signals generated by the image sensor 12 are distorted by wiggling errors, phase noise and multipath effects (including scattered light), according to the invention, the repetition of the second distance measurement value range D 2 within the first distance measurement value range D 1 is selected for which the second distance measurement value d 2 lying therein comes closest to the associated first distance measurement value d 1 in each case. In FIG. 6, for example, the first distance measurement value d 1 is identified in the larger first distance measurement value range D 1 and the four different "possible" second distance measurement values are identified in the respective repetitions of the smaller second distance measurement value range D 2. As can be seen, these differ in each case by an offset which results from the periodicity of the oscillations of the second frequency f 2 in the second sections A 2 and corresponds to the size of the second range of distance measurement values D 2. If the distance measurement value d 1 is now mapped from the upper part of the figure to the lower part of the figure, it can be seen that it comes closest to the "possible" second distance measurement value in the third repetition of the second distance measurement value range D 2. Therefore, the third repetition of the second distance measurement value range D 2 is selected, and the "correct" second distance measurement value d 2 can be determined to be by adding twice the offset.The above-described selection of the "correct" second distance measurement value range D 2 can be realized mathematically in different ways.In a first variant, the following mapping is first calculated, the second distance measurement value of which it is not yet known in which repetition of the second distance measurement value range D 2 it is correctly located and div is the division without remainder.However, this mapping does not yet guarantee that the second distance measurement value d 2 determined in this way comes closest to the first distance measurement value d 1. Therefore, correction is made according to the two rulesFinally, overflow beyond the farther end of the first distance measurement value range D 1 can be avoided as followsAn alternative variant is based on the modulo function mod and determines the "correct" second distance measurement value d 2 as follows:In this embodiment, the ToF camera 10 is adapted to generate a plurality of partial measurement images at different phase positions from the first sections A 1 of the reflected light signals 16 and from the second sections A 2 of the reflected light signals by integrating the reflected light signals 16 over a plurality of integration phases offset in time with respect to one another, wherein the first distance measurement values d 1 and the second distance measurement values d 2 are determined from the partial measurement images. In particular, three partial measurement images are generated here in each case both from the first sections A 1 of the reflected light signals 16 and from the second sections A 2 of the reflected light signals 16. The integration phases offset in time with respect to one another can in this case be realized via three electronic shutters SH 1, SH 2 and SH 3 which are opened at corresponding times. The integration phases correspond in each case to the duration of the square-wave pulses and the time offset corresponds in each case to two thirds of the duration of the square-wave pulses (cf. FIG. 2 - there second to fourth rows).Further variations of the disclosed embodiments may be understood and carried out by one skilled in the art practicing the claimed invention from a consideration of the drawings, specification and appended claims.In the above embodiments, the oscillations of the first frequency f 1 and the oscillations of the second frequency f 2 are respectively periodic sequences of rectangular pulses having a pulse-to-pause ratio of 1 / 2. In particular, in alternative embodiments, periodic sequences of rectangular pulses with other pulse pause ratios (e.g. 1 / 3) can also be used. It is likewise possible to use sinusoidal or other types of oscillations instead of square-wave pulses.In the above embodiments, the distance measurements at the first and second frequencies were made on the basis of a total of six partial measurement images. In other embodiments, it is also possible for the measurements to be carried out on the basis of only four partial measurement images. For example, in an advantageous embodiment, three partial measurement images are generated from the second sections of the reflected light signals and only one partial measurement image is generated from the first sections of the reflected light signals, so that a total of four partial measurement images (one for the lower first frequency and three for the higher second frequency) is provided.In the above embodiments, the invention has been explained only for the use of a first frequency f 1 and a second frequency f 2. However, it is also possible to use "cascading" of frequencies of different magnitude, in order to further increase the robustness with respect to pronounced phase noise, multipath effects (including scattered light) and movement artifacts. It is thus possible, for example, for the processing unit 13 of the ToF camera 10 to be further adapted to process third sections A 3 of the reflected light signals 16, which are modulated with oscillations of a third frequency f 3 for determining third distance measurement values d 3 in a third distance measurement value range D 3 dependent on the third frequency f 3, wherein the third frequency f 3 is higher than the first frequency f 1 and lower than the second frequency f 2 and the third distance measurement value range D 3 is smaller than the first distance measurement value range D 1 and larger than the second distance measurement value range D 2, wherein the processing unit 13 is adapted, in determining the third distance measurement values d 3 to resolve ambiguities of the third distance measurement value range D 3 within the first distance measurement value range D 1 on the basis of the first distance measurement values d 1 and in determining the second distance measurement values d 2 to resolve ambiguities of the second distance measurement value range D 2 within the third distance measurement value range D 3 on the basis of the third distance measurement values d 3.The ToF camera 10 can furthermore comprise an interface 24 for parameterizing the ToF camera 10, wherein a number of frequencies used for the distance measurement (e.g. three) and / or a ratio of at least two of the first frequency f 1, the second frequency f 2 and, if provided, the third frequency f 3 can be set by the parameterization.The processing unit 13 of the ToF camera 10 may be adapted to perform further processing steps, such as filtering of the ranging image (e.g. an outlier filtering that detects individual pixels in the ranging image whose ranging values deviate greatly from the ranging values in the immediate environment), the generation of false color representations, etc.In the claims, the words "comprise" and "comprise" do not exclude other elements or steps, and the indefinite article "a" does not exclude a plurality.A single unit or device may perform the functions of multiple elements recited in the claims. The fact that individual functions and / or elements are recited in different dependent claims does not mean that a combination of these functions and / or elements could not also be used advantageously.The reference numerals in the claims should not be understood to limit the subject matter and scope of the claims by such reference numerals.In summary, a time-of-flight, ToF, camera for measuring range information for objects of a scene has been described. The ToF camera comprises a light source for emitting modulated light signals for illuminating the objects, an image sensor for detecting light signals reflected at the objects, a processing unit for processing first sections of the reflected light signals modulated with oscillations of a first frequency for determining first distance measurement values in a first distance measurement value range dependent on the first frequency, and for processing second sections of the reflected light signals modulated with oscillations of a second frequency for determining second distance measurement values in a second distance measurement value range dependent on the second frequency, wherein the second frequency is higher than the first frequency and the second distance measurement value range is smaller than the first distance measurement value range, wherein the processing unit is adapted to:, determining the second distance measurement values to resolve ambiguities in the second distance measurement value range within the first distance measurement value range on the basis of the first distance measurement values and to generate a distance measurement image on the basis of only the second distance measurement values, and an output and / or storage unit for outputting and / or storing the distance measurement image.
Claims
A time-of-flight, ToF, camera (10) for measuring distance information for objects (14) of a scene, the ToF camera (10) comprising: - a light source (11) for emitting modulated light signals (15) for illuminating the objects (14); - an image sensor (12) for detecting light signals (16) reflected at the objects (14); - a processing unit (13) for processing first portions (A 1) of the reflected light signals (16) modulated with oscillations of a first frequency (f 1), determining first distance measurement values (d 1) in a first distance measurement value range (D 1), dependent on the first frequency (f 1) and processing second sections (A 2) of the reflected light signals (16) modulated with oscillations of a second frequency (f 2) for determining second distance measurement values (d 2) in a second distance measurement value range (D 2), dependent on the second frequency (f 2), wherein the second frequency (f 2) is higher than the first frequency (f 1) and the second distance measurement value range (D 2) is smaller than the first distance measurement value range (D 1), wherein the processing unit (13) is adapted, when determining the second distance measurement values (d 2) to resolve ambiguities of the second distance measurement value range (D 2) within the first distance measurement value range (D 1) on the basis of the first distance measurement values (d 1) and to generate a distance measurement image (18) on the basis of only the second distance measurement values (d 2) ; an output and / or memory unit (17) for outputting and / or storing the distance measurement image (18), wherein the oscillations of the first frequency (f 1) and / or the oscillations of the second frequency (f 2) are periodic sequences of rectangular pulses, preferably with a pulse-to-pause ratio of 1 / 2, wherein the ToF camera (10) is adapted to generate a plurality of partial measurement images at different phase positions from the first sections (A 1) of the reflected light signals (16) and / or from the second sections (A 2) of the reflected light signals (16) by integrating the reflected light signals (16) over a plurality of integration phases offset in time with respect to one another, wherein the first distance measurement values (d 1) and the second distance measurement values (d 2) are determined from the partial measurement images, wherein three partial measurement images are generated from the second sections (A 2) of the reflected light signals (16) and only one partial measurement image is generated from the first sections (A 1) of the reflected light signals (16).The ToF camera (10) according to claim 1, wherein the integration phases each correspond to the duration of the square pulses and / or the time offset each corresponds to two thirds of the duration of the square pulses.The ToF camera (10) according to claim 1 or 2, wherein the light source (11) is further adapted to emit the second sections (A 2) to be processed for determining the second distance measurement values (d 2) prior in time to the first sections (A 1) to be processed for determining the first distance measurement values (d 1).The ToF camera (10) according to any one of claims 1 to 3, wherein the light source (11) is further adapted to emit the first portions (A 1) to be processed for determining the first distance measurement values (d 1) and the second portions (A 2) to be processed for determining the second distance measurement values (d 2) in a time-interleaved order.The ToF camera (10) according to any one of claims 1 to 4, wherein the ToF camera (10) is adapted to operate the light source (11) with a longer exposure time and / or with a higher current or a higher voltage than during the first sections (A 1). when emitting the modulated light signals (15).The ToF camera (10) according to any one of claims 1 to 5, wherein the processing unit (13) is adapted to perform a correction with respect to a frequency dependent offset value when determining the first distance measurement values (d 1) and / or the second distance measurement values (d 2).ToF camera (10) according to one of Claims 1 to 6, wherein, when the ambiguities are resolved, in each case that repetition of the second distance measurement value range (D 2) within the first distance measurement value range (D 1) is selected for which the second distance measurement value (d 2) lying therein comes closest to the associated first distance measurement value (d 1).The ToF camera (10) according to any one of claims 1 to 7, wherein the second frequency (f 2) is an integer multiple (N) of the first frequency (f 1).The ToF camera (10) according to any one of claims 1 to 8, wherein the processing unit (13) is further adapted to process third portions (A 3) of the reflected light signals (16) modulated with oscillations of a third frequency (f 3) for determining third distance measurement values (d 3) in a third distance measurement value range (D 3) dependent on the third frequency (f 3), wherein the third frequency (f 3) is higher than the first frequency (f 1) and lower than the second frequency (f 2) and the third distance measurement value range (D 3) is smaller than the first distance measurement value range (D 1) and larger than the second distance measurement value range (D 2), wherein the processing unit (13) is adapted, in determining the third distance measurement values (d 3) to resolve ambiguities of the third distance measurement value range (D 3) within the first distance measurement value range (D 1) on the basis of the first distance measurement values (d 1) and in determining the second distance measurement values (d 2) to resolve ambiguities of the second distance measurement value range (D 2) within the third distance measurement value range (D 3) on the basis of the third distance measurement values (d 3).The ToF camera (10) according to any one of claims 1 to 9, wherein the ToF camera (10) further comprises an interface (24) for parameterizing the ToF camera (10), wherein a number of frequencies used for distance measurement and / or a ratio of at least two of the first frequency (f 1), the second frequency (f 2) and, if provided, the third frequency (f 3) can be set by the parameterization.Time-of-flight, ToF, method for measuring distance information for objects (14) of a scene, wherein the ToF method comprises: - emitting modulated light signals (15) for illuminating the objects (14); - detecting light signals (16) reflected at the objects (14); - processing first sections (A 1) of the reflected light signals (16), which are modulated with oscillations of a first frequency (f 1) for determining first distance measurement values (d 1) in a first distance measurement value range (D 1), dependent on the first frequency (f 1), and processing second sections (A 2) of the reflected light signals (16), which are modulated with oscillations of a second frequency (f 2) for determining second distance measurement values (d 2) in a second distance measurement value range (D 2), which depends on the second frequency (f 2) wherein the second frequency (f 2) is higher than the first frequency (f 1) and the second distance measurement value range (D 2) is smaller than the first distance measurement value range (D 1), wherein the processing, when determining the second distance measurement values (d 2) resolves ambiguities of the second distance measurement value range (D 2) within the first distance measurement value range (D 1) on the basis of the first distance measurement values (d 1) and generates a distance measurement image (18) on the basis of only the second distance measurement values (d 2) ; and - outputting and / or storing the distance measurement image (18), wherein the oscillations of the first frequency (f 1) and / or the oscillations of the second frequency (f 2) are periodic sequences of square pulses, preferably with a pulse-to-pause ratio of 1 / 2, wherein the ToF method generates a plurality of partial measurement images at different phase positions from the first sections (A 1) of the reflected light signals (16) and / or from the second sections (A 2) of the reflected light signals (16) by integrating the reflected light signals (16) over a plurality of integration phases offset in time with respect to one another, wherein the first distance measurement values (d 1) and the second distance measurement values (d 2) are determined from the partial measurement images, wherein three partial measurement images are generated from the second sections (A 2) of the reflected light signals (16) and only one partial measurement image is generated from the first sections (A 1) of the reflected light signals (16).
Citation Information
Patent Citations
Methods and devices for operating a PMD system
DE10039422C2