Camera with spot illumination
The integration of time-of-flight cameras with spot lighting and advanced statistical analysis enables accurate distance evaluation by separating signals from multiple objects, addressing the issue of erroneous measurements in spot lighting scenarios.
Patent Information
- Application Number
- PCT/EP2024/061524
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-26
- Filing Date
- 2024-04-26
- Publication Date
- 2024-12-05
AI Technical Summary
Time-of-flight cameras with spot lighting face challenges in accurately evaluating distances due to spots overlapping object edges, leading to incorrect assignments of pixels and erroneous results from naive averaging.
A method combining time-of-flight cameras with spot lighting, where data is weighted and averaged within spot environments, and advanced statistical analysis is applied to identify and separate signals from multiple objects by analyzing distribution characteristics and pixel positions, allowing for accurate object property separation.
This approach effectively identifies and corrects erroneous distance measurements by distinguishing between single-object and multi-object scenarios, providing accurate and noise-reduced output for distance and amplitude properties.
Smart Images

Figure EP2024061524_05122024_PF_FP_ABST
Abstract
Description
[0001] Camera with spot lighting
[0002] The invention relates to a camera, in particular a time-of-flight camera with a spot illumination according to the class of the independent claim.
[0003] The term "time-of-flight camera" encompasses all time-of-flight or 3D-TOF camera systems that determine distances directly from the time-of-flight (dTOF) or from the phase shift of emitted and received radiation (iTOF). PMD cameras with photonic mixer detectors (PMD), as described in DE 197 04 496 A1, among others, or cameras that use a SPAD array as the receiving sensor, are particularly suitable as time-of-flight or 3D-TOF cameras. Lorenzo Rimoldini, "Weighted skewness and kurtosis unbiased by sample size and Gaussian uncertainties; Astronomy and Computing, Volume 5, July 2014, Pages 1-8," describes, for example, one way of determining the moments described here in a weighted manner and for small sample sizes.
[0004] The object of the invention is to improve the evaluation of a camera with spot lighting.
[0005] The problem is solved by the inventive procedure.
[0006] It shows
[0007] Figure 1 shows a shot of a scene with conventional lighting,
[0008] Figure 2 shows a shot of a scene with spot lighting,
[0009] Figures 3 and 4 show an amplitude image for a shot with spot lighting,
[0010] Figure 3: all found spots are marked as ▲,
[0011] Figure 4: four spot splits (spots lying on an edge) identified in the sense of the invention are marked with ▲,
[0012] Figures 5 and 6 show a distribution of phases in a spot environment,
[0013] Figure 5 for a spot on an edge and
[0014] Figure 6 for a spot on a single object.
[0015] With spot cameras, the emitted spots are not point-like, but rather have a certain extension. If a spot falls on an object edge, this can lead to the pixels belonging to this spot being assigned to different objects, and therefore a report via the spot leads to erroneous values. The described invention detects these cases and, if necessary, makes it possible to separate the signals from the different objects.
[0016] This study examines a combination of a time-of-flight (ToF) camera and spot lighting. Spot lighting illuminates the scene only at specific points, and distances are only evaluated at these points.
[0017] Figure 1 shows an example of a scene that is homogeneously illuminated, and Figure 2 shows a scene with spot lighting.
[0018] A spot typically covers multiple pixels. A spot is characterized on the sensor, for example, by a spot center (e.g., the center, center of gravity, or the brightest pixel of the spot) and a spot neighborhood around this center (e.g., a circular area with a radius of 3.5 pixels or a square with 9x9 pixels). For evaluation, the data required for output (e.g., phases, amplitudes, distances, raw data) are weighted averaged for each spot neighborhood before being further processed in the usual manner for the method.
[0019] For finding spots, for example, the amplitude image (= Re 2 + In 2 (as is common with ToF) is used (see Figure 2). This can be done, for example, by determining the maxima in the image or by binarizing the image using a threshold value and then finding connected areas.
[0020] Occasionally, a spot falls on an object edge, which results in the pixels of the spot environment belonging to this spot being assigned to different objects.
[0021] Therefore, naive averaging leads to erroneous results. In most cases, the resulting distance will be somewhere between the two object distances, which is problematic for many applications because there is no object at that distance.
[0022] This patent describes methods for detecting these cases, which allows for their removal (see Figures 3 and 4). If necessary, and with low-noise measurements, the signals can also be separated. In such a case, the user can decide whether to output the property (distance / amplitude) of the closer, farther, brighter, darker, etc., or of all objects. Figures 3 and 4 show an amplitude image for a spot-illuminated image. Figure 3: All detected spots are marked with ▲.
[0023] Figure 4: Four spot splits (spots that lie on an edge) detected in accordance with the invention are marked with ▲.
[0024] In principle, it is also possible for a spot to hit multiple consecutive object edges, thus covering more than two objects. For simplicity, however, we limit the description to the case of two objects, noting that the methods can also be applied or extended to the more general case—under favorable measurement conditions.
[0025] To detect whether the spot lies on an edge, suitable pixel variables (F) in the spot's vicinity are considered. These variables can be, for example, phase, distance, amplitude-normalized real / imaginary component, amplitude normalized by a known amplitude profile, or combinations thereof. These variables are advantageously chosen so that they change little or not in the spot's vicinity, provided the spot lies on a single object.
[0026] Consideration of the distribution
[0027] In the simplest version, we now consider the distribution of these quantities (see Figures 5 and 6). If the spot is on a single object (Figure 6), the distribution is less broad and ideally consists of only a single fixed value. Furthermore, the width of the distribution is usually due to noise, which means that the distribution in this case approximately corresponds to a Gaussian distribution. If the spot hits an edge, a bimodal or multimodal distribution and a broader distribution can be expected. This can be seen in Figure 5.
[0028] Figures 5 and 6 show a distribution of phases in a spot environment Figure 5 for a spot on an edge and
[0029] Figure 6 for a spot on a single object.
[0030] To test whether the spot lies on an edge, the moments of the distribution or combinations thereof can be used. For example, if an inhomogeneity and / or the expected noise (R) are known, the standard deviation of the distribution (o) and the ratio o / R can be determined.
[0031] This should be close to 1 for individual objects. If it is not within a specified range, the spot can be identified as "lying on an edge."
[0032] Advantageously, higher moments such as skewness and kurtosis can also be used. These are zero for Gaussian distributions, so deviations from zero can be due to multiple objects in the spot.
[0033] Inclusion of pixel positions
[0034] Since an object edge divides the spot into two areas, corresponding to one object and the other, it is advantageous to also consider the position within the spot's surroundings. One simple possibility is, for example, to perform a plane fit in the space spanned by the phase and the x and y directions of the spot's surroundings. If the resulting plane has a pronounced component in the x and / or y direction, the phase changes along this direction and the object edge is perpendicular to this direction. A spot can now be identified as "lying on an edge" if the magnitude of the normal in the x and y directions exceeds a limit.
[0035] Similarly, from the phase F[x,y], for example, the quantities which are also a measure of the change in phase in the x and y directions. The sum runs over all pixels of the spot surroundings and (x z ,y z) is the spot center. If the amount of C exceeds x or C y a threshold, the spot can be identified as “lying on an edge”.
[0036] To achieve higher confidence, distributions of different sizes can be considered and different criteria (e.g. kurtosis and level fit) can be combined.
[0037] Separating object properties
[0038] The method can be elegantly extended to separate properties (e.g., distances) of the two objects. If one considers only the distribution of the sizes (F), one can use, for example, "Otsu's Thresholding Method" to find the boundary of the bimodal distribution. Now, all pixels in the spot's vicinity above this boundary can be assigned to one object and below the boundary to the other. Advantageously, pixels close to the boundary can be discarded to achieve better results.
[0039] When including the pixel positions, it can be exploited that the plane normal, as well as the vector perpendicular to the object edge, which enables or facilitates separation. Here, too, pixels close to the object edge can be discarded to achieve better results.
[0040] The properties (e.g., distance, amplitude) of the objects can now be averaged separately, as before, using a weighted average. Likewise, the user can decide whether to output the properties of the closer, farther, brighter, darker, etc., or of all objects. Alternatively, spots that lie on an edge can be discarded.
Claims
Patent claims 1 . Camera with an illumination which emits light in the form of a dot pattern or a plurality of spots, with a receiving sensor with a plurality of pixels for receiving the emitted dot pattern or the emitted spots, wherein the size of the pixels is smaller than the extent of a received spot, and wherein the extent of the spot is set such that the received spot illuminates a plurality of pixels, in which at least one measured value is output for at least some of the spots, with an evaluation unit which is designed such that a decision can be made on the basis of the pixels in the area of the received spots as to whether a spot lies on at least one object edge.
2. Camera according to claim 1, wherein the decision is made based on at least one variable (F) which changes as little as possible for individual objects within a spot.
3. Camera according to claim 2, wherein the distribution of the quantity(s) (F) and / or their position relative to the spot position is taken into account.
4. Camera according to one of the preceding claims, wherein spots lying on an edge are discarded.
5. Camera according to claims 1-3, wherein measured values (distance / amplitude) of one of the objects separated by an edge are output, or the separated measured values of several objects.
6. Time-of-flight camera according to one of the preceding claims, in which the light is emitted in a modulated manner, and the time-of-flight camera is designed such that distance values are determined from a phase shift between the emitted and received light.
7. Time-of-flight camera according to one of claims 1-5, wherein the light is emitted in pulses and the receiving sensor has a plurality of pixels designed as SPAD pixels.
Citation Information
Patent Citations
Method and device for determining the phase and / or amplitude information of an electromagnetic wave
DE19704496A1
Object detection device, object detection method and program
JP2013114611A
Distance measuring device, control method thereof, and distance measuring system
WO2022168500A1
