Light propagation time camera

Abstract

The invention relates to a light propagation time camera comprising an illumination means for emitting a structured light in the form of a plurality of light spots or light structures - and comprising a light propagation time sensor for detecting the light emitted and reflected by a scene, - comprising an evaluation unit which is designed such that the following steps are provided for determining an exposure time and / or light intensity: - detecting a brightness pattern by means of the light propagation time sensor, - dividing the brightness pattern into tile-shaped pixel groups, wherein a dimension of the pixel groups is defined such that the centre points of the pixel groups have a distance which is equal to or greater than a minimum distance of the received light structures, - ascertaining a brightness maximum value within the pixel groups, - forming a brightness sequence or determining a brightness distribution starting from the ascertained brightness maximum values - determining an exposure time and / or light intensity of the emitted light on the basis of the brightness sequence or brightness distribution.

Description

[0001] Time-of-flight camera

[0002] The invention relates to a time-of-flight camera according to claim 1.

[0003] If you want to use spot lighting to specifically adjust the exposure time to darker areas of a scene (such as with HDR), you always have the difficulty with standard methods (e.g. quantile over all pixels) that you cannot distinguish the spot edges of the bright areas from the desired spot centers of the dark areas.

[0004] The object of the invention is to design a time-of-flight camera with a spot illumination in such a way that an exposure can be suitably adjusted.

[0005] The problem is solved by the time-of-flight camera according to the invention.

[0006] Advantageously, a time-of-flight camera is provided, with an illumination for emitting a structured light in the form of several light spots or light structures,

[0007] - and with a time-of-flight sensor to detect the light emitted and reflected by a scene,

[0008] - with an evaluation unit designed in such a way that the following steps are provided for determining an exposure time:

[0009] - Capturing a brightness image using the time-of-flight sensor,

[0010] - dividing the brightness image into tile-shaped pixel groups, wherein a dimension of the pixel groups is determined such that the centers of the pixel groups have a distance that is equal to or greater than a minimum distance of the received light structures,

[0011] - Determination of a maximum brightness value within the pixel groups,

[0012] - Forming a brightness series or determining a brightness distribution based on the determined maximum brightness values

[0013] - Determination of an exposure time based on the brightness series or brightness distribution.

[0014] Dividing the brightness image into pixel groups of appropriate size ensures that at least one light spot or light structure can be detected within each pixel group. After determining a maximum brightness value for each pixel group, a suitable exposure time for the respective application can be determined based on a brightness series or a histogram created from it.

[0015] The approach considers a combination of a time-of-flight (ToF) camera (e.g., dToF, iToF, LiDAR) and structured lighting. This could be, for example, spot lighting. With this approach, the scene is illuminated only at specific points, and distances are only evaluated at these points.

[0016] They show schematically

[0017] Figure 1 shows a shot of a scene with spot lighting,

[0018] Figure 2 Histograms of bright and dark objects,

[0019] Figure 3 Top: Original brightness image. Middle / bottom: Brightness image after applying the Max-Bin method for a tile size of 7 pixels (middle) and 9 pixels (bottom).

[0020] Figure 4 a brightness histogram over all pixels,

[0021] Figure 5 shows a histogram related to pixel groups,

[0022] Figure 6 shows a histogram of the maximum brightness values ​​with an exposure time optimized for dark objects.

[0023] Figure 1 shows a shot of a scene with spot lighting. A potentially interesting dark area of ​​the image is indicated by a rectangle.

[0024] Other structured lighting such as line or checkerboard patterns are also conceivable, but for the sake of simplicity we will focus on spot lighting here.

[0025] A high dynamic range of scenes often presents a challenge, particularly in recording techniques using active lighting, where objects in the foreground are usually significantly brighter. In this case, it may be that primarily darker areas in the image are relevant, for example, objects further away. This is particularly the case with high-dynamic-range (HDR) recordings, which typically require both a recording in which the bright objects are well exposed and one in which the dark objects are well exposed. This invention describes a method for determining an optimal exposure time with structured lighting, particularly for dark areas.

[0026] In spot systems, a spot usually covers multiple pixels. A spot is characterized on the sensor by a spot center (e.g., the center, centroid, or brightest pixel of the spot) and a spot neighborhood around this center (e.g., a circular area with a radius of 3.5 pixels or a square with 9x9 pixels). For evaluation, the data required for output (e.g., phases, amplitudes, distances, raw data) are weighted averaged for each spot neighborhood before being further processed in the usual way for the method.

[0027] In the following, brightness refers to quantities that reflect the number of registered photons. This can be the actual number of photons, but also related quantities, such as - in the context of indirect ToF cameras - the maximum of the real (Re) and imaginary parts (Im), (max(| / ?e|, | / m|)), or the amplitude (A = / ?e 2 + In 2 ).

[0028] For measurement methods without spot illumination, i.e., homogeneous illumination, a suitable exposure can be determined by determining a quantile based on the brightness of all pixels. For example, if you consider the brightness distribution of the pixels in a scene and set the exposure time so that the 40% quantile is just below the saturation value, the brightest 60% will be saturated, but the darkest 40% will have good illumination and can be further processed.

[0029] This approach fails for spot lighting, as spots typically consist of bright centers and dark edges. Choosing the 40% quantile across all pixels might reveal some of the edges of the bright spots and some of the centers of the dark spots (see Fig. 2).

[0030] One would therefore have to apply the quantile to the pixels of the spot centers, which are not necessarily available.

[0031] According to the invention, the image is now divided into tiles (or so-called bins) such that the tile size roughly corresponds to the spot spacing. Each tile now contains a spot center.

[0032] Alternatively, the tiles can be selected to be larger than the spot spacing, so that some tiles contain several spot centers.

[0033] Now, the brightest pixel in each tile is selected, which should thus correspond to a spot center (Fig. 3). We therefore refer to this as the "max-bin" method. The distribution of these brightest pixels therefore approximately corresponds to the distribution of the brightnesses of the spot centers.

[0034] Figure 3 shows the original brightness image (top). Middle / bottom: The brightness image after applying the max-bin method for a tile size of 7 pixels (center) and 9 pixels (bottom).

[0035] When choosing the tile size, it is important not to select it too small. If this is the case, tiles will be created that lack a spot center, resulting in too many dark values ​​in the distribution. Choosing the tile size too large is less critical, as this allows two spot centers to be located in one tile. However, as long as this applies equally to both light and dark areas, it does not lead to any significant systematic distortion of the distribution.

[0036] Using the distribution thus obtained, the exposure time can now be determined, as would be usual with homogeneous lighting. For example, the exposure time can be chosen so that the 40% quantile of the brightness lies just below the saturation value (e.g., 90% of maximum brightness).

[0037] Other methods are also conceivable, e.g. the mean of the distribution is 50% of the maximum brightness.

[0038] When choosing the tile sizes, analogous to the spot spacing, the line spacing can be used for line lighting and the spacing of the squares of the chessboard can be used for chessboard lighting.

[0039] It's worth noting that the tiles can also have variable sizes. As can be seen in Fig. 3 (above), the spot spacing can change depending on the position in the image, making this approach advantageous.

[0040] It should also be noted that the tiles can also overlap, e.g. if different tile sizes make overlap-free coverage impossible, or the tiles only cover part of the entire pixel array.

[0041] It may also be advantageous to use the second, third, etc. brightest value, or a quantile, instead of the true maximum value in a tile to ignore possible outliers or defect pixels. It is also conceivable to determine a weighted value within a tile, where the weights can be based, for example, on the squares of the brightnesses, to primarily obtain the brightest value. For simplicity, we will nevertheless refer to this value as the maximum value.

[0042] Figure 4 shows an example of a brightness histogram across all pixels, and Figure 5, based on the example shown in Figure 4, shows a histogram of the maximum brightness values ​​of the pixel groups. Different exposures can be set depending on the application. If the bright areas are important for the application, the exposure can be left as it is in the example shown in Figure 5. If, on the other hand, the dark areas or dark objects are of interest, it is helpful to extend the exposure time in the example shown, accepting that the brighter areas become saturated, as shown in Figure 6.

Claims

Patent claims 1 . Time-of-flight camera with an illumination for emitting structured light in the form of several light spots or light structures, - and with a time-of-flight sensor to detect the light emitted and reflected by a scene, - with an evaluation unit designed in such a way that the following steps are provided for determining an exposure time and / or a light intensity: - Capture of a brightness image using the time-of-flight sensor, - dividing the brightness image into tile-shaped pixel groups, wherein a dimension of the pixel groups is determined such that the centers of the pixel groups have a distance equal to or greater than a minimum distance of the received light structures, - Determination of a maximum brightness value within the pixel groups, - Forming a brightness series or determining a brightness distribution based on the determined maximum brightness values - Determination of an exposure time and / or a light intensity of the emitted light based on the brightness series or brightness distribution.

2. Time-of-flight camera according to claim 1, wherein, in the case of an illumination which emits structured light in the form of light spots, the pixels of the time-of-flight sensor are smaller than a typical light spot imaged on the time-of-flight sensor.

3. Time-of-flight camera according to one of the preceding claims, in which the time-of-flight camera operates according to the phase measurement principle.

4. Time-of-flight camera according to one of the preceding claims, in which the exposure time is determined based on a quantile or mean value of the maximum values ​​of the brightness is determined.

5. A time-of-flight camera according to any one of the preceding claims, wherein the tiles have variable sizes.

6. A time-of-flight camera according to any one of the preceding claims, wherein the tiles do not cover the entire pixel array and / or the tiles overlap.

7. Time-of-flight camera according to one of the preceding claims, in which the second, third, or fourth brightest value is used as the maximum value in a tile.

8. Time-of-flight camera according to one of the preceding claims, in which a quantile or a weighted mean is used as the maximum value in a tile.

Images