Method and apparatus for dynamic expansion of time-of-flight camera system

Abstract

The invention relates to a method for dynamic extension of a raw phase image of a time-of-flight camera or a time-of-flight camera system, in which method at least two depth images are taken using different exposure times.

Description

Technical Field

[0001] The present invention relates to a method in which multiple raw data measurements with different exposure times are combined with each other in order to comprehensively extend the dynamic range of an image sensor and in this way prevent saturation effects as much as possible. Background Technology

[0002] The term time-of-flight camera or time-of-flight camera system is intended here to specifically cover systems that obtain distance from the phase shift of emitted and received radiation. Specifically, a PMD camera, including a photomixing detector (PMD) (e.g., the PMD described in DE 197 04496A1), is suitable as a time-of-flight or TOF camera.

[0003] Furthermore, the determination of the distance or corresponding phase shift of light reflected from an object is known from DE 197 04 496A1. Specifically, it is disclosed to selectively shift the transmitter modulation by 90°, 180°, or 270° in order to determine the phase shift and thus the distance from these four phase measurements or four phase images via an arctangent function. Summary of the Invention

[0004] The purpose of this invention is to improve the dynamic range of a time-of-flight camera.

[0005] Advantageously, a method is provided for dynamic range extension of the raw phase image (M) of a time-of-flight camera system, wherein the time-of-flight camera system obtains the phase shift of emitted and received radiation. To determine the distance (d), where, for distance determination, different phase positions between the lighting device and the receiver are considered. Multiple raw phase images (M) are acquired at different exposure times (t). exp1 t exp2 To capture these raw phase images, the method includes the following steps:

[0006] a) Identify invalid (i.e. saturated) pixels and valid (i.e. unsaturated) pixels in the original phase image (M).

[0007] b) Calculate the normalized original phase image using the following method. Subtract the fixed pixel offset (fixed pattern noise (FPN)) from each original phase image (M) according to the following formula, and with the corresponding exposure time (t) exp1 t exp2 Normalize the image with reduced fixed pixel offset (FPN).

[0008]

[0009] c) By normalizing the original phase image according to the following formula The original phase image is calculated by multiplying it by the corresponding weighting matrix (G).

[0010]

[0011] d) Calculate the weighted normalized HDR image for the corresponding phase position as follows: The original phase image for the corresponding phase position is weighted and normalized according to the following formula. Add and normalize to the sum of weighted matrices

[0012]

[0013] Among them, operators This represents element-wise multiplication, such as the so-called Hadamard operator.

[0014] e) Calculate the HDR original phase image (M) in the following way HDR The weighted normalized HDR raw phase image is calculated according to the following formula: Multiply by exposure time (t) exp1 t exp2 One of them, plus a fixed pixel offset.

[0015]

[0016] f) Provides HDR original phase image (M HDR (This is for subsequent processing.)

[0017] The advantage of this method is that the original phase image of HDR (M HDR This can be used for further processing that is no different from regular processing. Therefore, HDR processing can be considered a completely independent preprocessing step and performed as such.

[0018] Furthermore, if the weighted normalized HDR raw phase image is used in step e), Multiply by the longest exposure time (t) exp1 t exp2 ), which is advantageous.

[0019] Preferably, the value of the weighted matrix (G) for each unsaturated pixel in step c) corresponds to the corresponding base exposure time (t). exp1 t exp2 Furthermore, for saturated pixels, these pixels are set to zero in the weighting matrix G. Attached Figure Description

[0020] The invention will now be explained in more detail with reference to the accompanying drawings and exemplary embodiments.

[0021] The attached diagram schematically illustrates:

[0022] Figure 1 A time-of-flight camera system is shown; and

[0023] Figure 2 The HDR weighting scheme is shown. Detailed Implementation

[0024] Figure 1 The measurement is shown using an optical distance measurement with a time-of-flight camera, as is known, for example, from DE197 04496A1.

[0025] The time-of-flight camera system 1 includes: a transmitter unit or illumination module 10, which includes an illumination device 12 and an associated beamforming optics 15; and a receiver unit or time-of-flight camera 20, which includes a receiving optics 25 and a time-of-flight sensor 22.

[0026] The time-of-flight sensor 22 includes at least one time-of-flight pixel, preferably also including a pixel array, and is specifically designed as a PMD sensor. The receiving optics 25 typically consists of multiple optical elements to improve imaging characteristics. The beamforming optics 15 of the transmitter unit 10 can be designed, for example, as a reflector or lens optics. In a very simple construction, optical elements on both the receiving and transmitting sides can be omitted.

[0027] The measurement principle of this arrangement is basically based on the fact that the propagation time of the received light can be determined from the phase shift of the emitted and received light, and thus the distance traveled by the received light can be determined. For this purpose, the light source 12 and the time-of-flight sensor 22 are jointly provided with a base phase position via the modulator 30. The specific modulation signal M0. In the example shown, a phase shifter 35 is also provided between the modulator 30 and the light source 12, by means of which the base phase of the modulation signal M0 of the light source 12 is... It can be shifted according to a defined phase position. For typical phase measurements, it is preferable to use... And a phase position of 270°.

[0028] According to the set modulation signal, the light source 12 emits light with a first phase position p1 or Intensity modulation signal S p1 In the case shown, the signal S p1 Or the electromagnetic radiation is reflected by object 40, and changes phase shift accordingly due to the distance traveled. Impact object 40, where the second phase position As the received signal S on the time-of-flight sensor 22p2 In the time-of-flight sensor 22, the modulation signal M0 and the received signal S p2 Mix the signals and determine the phase shift or object distance d from the resulting signal.

[0029] In addition, the system includes a modulation controller 27, which changes the phase position of the modulation signal M0 according to the actual measurement task. And / or adjust the modulation frequency via frequency oscillator 38.

[0030] Preferably, the infrared light-emitting diode is suitable as an illumination source or light source 12. Of course, other radiation sources in other frequency ranges are conceivable, and light sources in the visible frequency range are particularly worth considering.

[0031] High dynamic range imaging has been well established in the 2D domain. However, this mode is more important in the case of 3D-ToF measurements because it is an active measurement method that allows for large intensity differences. Therefore, saturation effects often occur at close ranges, while at long ranges, distant noise is preferentially encountered due to the low signal intensity of reflected light.

[0032] In the method described herein, multiple raw data measurements obtained using at least two different exposure times are combined with each other and weighted according to their exposure times, so that the full information content of all pixels, in addition to saturated pixels, is utilized.

[0033] Another key aspect of this invention is that HDR calculation can be performed as a completely independent preprocessing step, and subsequent processing of the measurement data can be performed without alteration and completely independently of the HDR calculation.

[0034] To determine the distance based on the phase measurement principle, the distance is first measured at different phase positions between the transmitter and receiver. The phase difference / phase shift between the transmitted and received modulated signals is detected. For each pixel ij of the time-of-flight optical sensor, these phase shifts m are detected. ij These phase shifts form the original phase image M. To calculate the actual phase shifts... At different phase positions At least two, preferably three or four, and possibly more, raw phase images M are acquired.

[0035] Based on the original phase image M, the distance d or depth image D can then be calculated by considering the known modulation frequency and arctangent relationship.

[0036] According to the present invention, it is now intended to utilize at least two different exposure times t exp1 and t exp2To capture the raw phase image M. For example, for four different phase positions. With two exposure times, eight raw phase images M can be obtained.

[0037] For further processing, such as using a saturation threshold to identify saturated pixels in all the original phase images M.

[0038] In order to combine the measurements from two exposure times, these measurements must then be normalized to the corresponding exposure times after subtracting the so-called fixed pattern noise (FPN):

[0039]

[0040] For these normalized raw phase images For each of the elements, an additional weighting matrix G of the corresponding size is now created. The values ​​of this weighting matrix G correspond to the base exposure time t for each unsaturated pixel. exp1 or t exp2 .

[0041] On the other hand, for saturated pixels, the corresponding entries in the weighting matrix G always take zero values. In this case, the saturated pixels of each independent original phase image M can be considered independently and thus stored as zero weights in the corresponding weighting matrix G. However, alternatively, an OR operation can be performed on the saturated pixels of all four original phase images at the exposure time, such that the weighting matrix G of all four original phase images M at the exposure time is the same.

[0042] For simplicity, the following examples assume that two different exposure times t have been used with a sensor having a resolution of 2×2 pixels. exp1 and t exp2 To film the scene, where: t exp1 =2t exp2 Of course, in principle, exposure time can have any other relationship.

[0043] Furthermore, in this example, it is assumed that for the longer of the two exposure times, the top two pixels are classified as saturated. For each exposure time, a corresponding weighting matrix G is created. Then, all pixels are multiplied by this matrix, where saturated pixels are ignored due to their zero weight, and only unsaturated pixels contribute to the measurement.

[0044] This example will be based on the following normalized raw phase image:

[0045]

[0046] Saturated pixels have now been identified and classified using zero weights in their corresponding weighting matrices. This is achieved by multiplying the normalized original phase image by the weighting matrix. and Then the matrix The calculation yields the following intermediate result:

[0047]

[0048]

[0049] The calculation for the second exposure time is similar:

[0050]

[0051]

[0052] Now, the weighting matrix and subsequent multiplication are calculated individually for each of the original phase images in the same manner.

[0053] Then, the normalized original phase image is weighted according to the corresponding weighting matrix G. The sums are added and then normalized again to obtain the sum of the weighted matrices. The operators here... Operators that represent element-wise multiplication, such as the Hadamard operator.

[0054]

[0055]

[0056] Assuming t exp1 =2t exp2 In this case, it can be simplified to:

[0057]

[0058] By multiplying by the longest exposure time, the original resolution is maintained. On the other hand, multiplying with a shorter exposure time will result in a decrease in resolution, but this is theoretically possible if necessary. Specifically, in cases with more than two exposure times, a particularly advantageous exposure time can be selected for further processing. However, the longest exposure time is generally preferred.

[0059] Therefore, all pixel information from all captured raw phase images M is utilized in the best way. Finally, the combined HDR raw phase image is multiplied by the longest exposure time t. exp1 And add FPN to convert them into output versions.

[0060]

[0061] Therefore, conventional processing can then be applied in an unmodified manner to the combined, synthesized original phase image M. HDR HDR processing expands the original range of values, allowing negative pixel values ​​to appear after combination.

[0062] Figure 2 The diagram schematically illustrates an HDR weighting scheme for four captured raw phase images, each with two exemplary exposure times t. exp1 =1500μs and t exp2 =500μs.

[0063] This process is essentially used to generate a dataset consisting of synthesized original phase images that have the same structure as the original original phase images, so that computation can be performed in the same way as the original original phase images.

[0064] Therefore, there is no difference in data processing, whether it is a real raw phase image or a synthetically generated HDR raw phase image, and the processing can be considered a completely autonomous and independent preprocessing step.

Claims

1. A method for dynamically expanding a raw phase image M for a time-of-flight camera system, wherein, The time-of-flight camera system determines the distance d from the phase shift of the emitted and received radiation ​ In order to determine the distance, different phase positions between the lighting device and the receiver are used. Multiple raw phase images M are acquired at different exposure times t. exp1 t exp2 To capture the plurality of raw phase images, the method includes the following steps: a) Identify invalid (i.e., saturated) pixels and valid (i.e., unsaturated) pixels in the original phase image M. b) The normalized original phase image is calculated using the following method. The fixed pixel offset, i.e., the fixed pattern noise FPN, is subtracted from each original phase image M according to the following formula, and the corresponding exposure time t is used. exp1 t exp2 The original phase image with the fixed pixel offset FPN reduced is normalized. c) By normalizing the original phase image according to the following formula Multiply by the corresponding weighting matrix To compute the weighted normalized original phase image ， d) Calculate the weighted, normalized original phase image for the corresponding phase position using the following method. HDR The weighted normalized original phase image at the corresponding phase position is calculated according to the following formula. Add them together and normalize them to the sum of the weighted matrices. ， Among them, operators Represents element-wise multiplication. e) Calculate the original phase image M using the following method HDR The weighted normalized original phase image is calculated according to the following formula. HDR Multiply by the exposure time t exp1 t exp2 One of them plus the fixed pixel offset M HDR = HDR ·t exp +FPN, and t exp It is t exp1 or t exp2 , f) providing said original phase image M HDR for subsequent processing.

2. The method according to claim 1, wherein, In step e), the weighted normalized original phase image HDR Multiply by the exposure time t exp1 t exp2 The longest exposure time in the process.

3. The method according to any one of the preceding claims, wherein, The weighting matrix of each unsaturated pixel in step c) The value corresponds to the corresponding exposure time t exp1 t exp2 And for saturated pixels, these pixels are in the weighting matrix Zero is used in the middle.

4. A time-of-flight camera or time-of-flight camera system for performing the method according to any one of claims 1 to 3.

Images