Camera with light spot illumination
By performing brightness, shape, gradient and filter consistency checks on ghosted light spots in the time of flight camera, the errors at the periphery of the light spot are removed, the accuracy of distance measurement under light spot illumination is improved, and the solution is to solve the problem of light spot illumination. The problem of ghost light spots achieves a more accurate distance assessment.
Patent Information
- Application Number
- CN202410664215.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-26
- Filing Date
- 2024-05-27
- Publication Date
- 2024-11-26
AI Technical Summary
When existing time-of-flight cameras use spot lighting, they are prone to ghosting spot problems, resulting in measurement errors and image noise, affecting the accuracy of distance evaluation.
By identifying and removing ghosted light spots, including brightness, shape, gradient and filter consistency checks at the periphery of the light spot, ensuring that the brightness distribution and shape of the center and outer light spots conform to the characteristics of the actual light spot, and then the distance calculation is carried out .
Effectively remove ghosted light spots, improve distance measurement accuracy and image quality under spot illumination, and reduce the impact of measurement errors and noise.
Smart Images

Figure CN119024358A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a camera, in particular to a time-of-flight camera with light spot illumination. Background Art
[0002] "Time of flight camera" shall include all time of flight camera systems or 3D-TOF camera systems that obtain distance directly from the time of flight (dTOF) or from the phase shift of the emitted and received radiation (iTOF). PMD cameras with photonic mixing detectors (PMD) as described in DE19704496A1 or cameras with SPAD (Single Photon Avalanche Diode) arrays as receiving sensors are particularly suitable as time of flight cameras or 3D-TOF cameras. Summary of the invention
[0003] The object of the invention is to improve the evaluation of cameras with spot illumination.
[0004] This object is achieved by the method according to the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] In the attached picture:
[0006] Figure 1 A photograph of a scene with conventional lighting is shown;
[0007] Figure 2 A photograph of a scene with spot lighting is shown;
[0008] Figure 3 shows an amplitude image recorded using spot illumination without ghost removal, where the output spots are marked with triangles;
[0009] Figure 4 The figure shows the method according to which the ghost image is removed. Figure 3 Amplitude image of
[0010] Figure 5 The periphery of the light spot whose center is located at (4,4) is shown;
[0011] Figure 6 Shown according to Figure 5 Histogram of the periphery of the light spot;
[0012] Figure 7 The periphery (F[x,y]) of the light spot whose center is located at (4,4) is shown;
[0013] Figure 8 Shown according to Figure 7The gradient of the light spot periphery in the x direction ((Δ x F)[x,y]); and
[0014] Fig. 9 A flow chart of the method according to the invention is shown by way of example. DETAILED DESCRIPTION
[0015] For a spot camera, it is important that the emitted distance corresponds to the spot actually emitted by the spot illumination. The proposed method describes a way to identify and discard falsely found so-called ghost spots.
[0016] Consider combining a Time of Flight (ToF) camera with spot lighting, in which the scene is illuminated only at individual points, and distances are evaluated only at these points.
[0017] Figure 1 An example of a uniformly illuminated scene is shown. Figure 2 A scene with spot lighting is shown.
[0018] Here, a light spot usually covers several pixels. The light spot is characterized on the sensor, for example, by the center of the light spot (e.g., the center point of the light spot, the center of gravity or the brightest pixel) and the periphery of the light spot surrounding the center (e.g., a circular area with a radius of 3.5 pixels or a square area of 9 pixels×9 pixels). For evaluation, for example, a weighted average is performed on each light spot periphery before the desired data (e.g., phase, amplitude, distance, raw data) are further processed in a conventional manner to output.
[0019] To find the light spot, use, for example, the amplitude image ( As is common in ToF) (see Figure 1 ). This can be achieved, for example, by determining the maximum value in the image or by binarizing the image based on a threshold and then finding the connected surfaces. Here, all light points emitted by the illumination source should be identified as far as possible. However, measurement errors and image noise often lead to so-called ghost light points, which are emitted as detected light points but do not correspond to real light points emitted by the light point illumination (see Figure 3 ).
[0020] This patent describes methods for finding these ghost light spots and removing them. These methods rely to some extent on the brightness of the periphery of the light spot, such as Figure 4 For example, amplitude (or intensity, etc.) can be used directly for brightness. For example, better results can be obtained if the amplitude background, i.e. the amplitude values far from the center of the light spot, is subtracted from the amplitude.
[0021] Figure 3 and Figure 4Amplitude images recorded using spot illumination are shown. The emitted spots are marked with ▲. Figure 3 : Ghost is not removed. Some ghost light spots are emitted in the lower left area of the image. Figure 4 : Ghost removal in the sense of the present invention.
[0022] Light spot distribution inspection
[0023] The points of light in spot lighting usually have a similar brightness distribution. This means that they are brighter in the center of the point and darker at the edges of the point, see Figure 5 .
[0024] To ensure that the outer periphery of the light spot is from the actual light spot, the distribution of brightness can be used ( Figure 6 ). This can be used to check whether the distribution corresponds to the distribution of real light spots. This can be done, for example, by dividing the standard deviation by the mean of the distribution and checking whether the result is in a range corresponding to real light spots. The mean absolute deviation or other (normalized or not) distribution moments normalized by the brightness of the light spot center are possible.
[0025] Spot shape inspection
[0026] The spot of light for spot illumination also usually has a given shape. Therefore, it is helpful to relate the measured shape to the expected shape (see, for example, Figure 7 ). The desired shape can be obtained from reference measurements or by a suitable parameterization (e.g. a Gaussian function) whose parameters are set so that the algorithm provides the desired result. Typically, a cross-correlation is used between the measured shape F[x,y] and the reference shape Ref[x,y]. However, if the spot center is known, the calculation can be simplified to:
[0027]
[0028] The sum is calculated for all pixels x, y around the light spot. Of course, for normalization ((∑ x,y F[x,y]), there may be other ways, such as the brightness at the center of the light spot.
[0029] Now you can check whether C is in the range that provides the expected result.
[0030] Figure 5 The periphery of the spot (F[x,y]) with the spot center at (4,4) is shown.
[0031] Spot gradient inspection
[0032] The light spot of the spot illumination usually starts brighter from the left, reaches a maximum value, and then gets dimmer. It is equal from top to bottom. This is reflected in the measured gradient (see Figure 7 and Figure 8 ). While unfavorable clustering can lead to ghost spots when checking spot shape, this is less likely to happen when checking gradients. Therefore, it makes sense to check this behavior. A simple approach is to determine the size.
[0033]
[0034] Among them, Δ x is the discrete gradient in the x direction, that is, (Δ x F)[x,y]=F[x+1,y]-F[x,y]. Z is the x value of the center of the light spot. For the actual light spot, to the left of the center of the light spot, (Δ x F)[x,y]>0 and (x Z -x)>0, and to the right of the spot center, both of these terms are <0. Therefore, the sum of the actual spots gives an apparent positive value. Now we can check C x Is it within the expected range?
[0035] Figure 6 Shows the Figure 5 The gradient of the light spot periphery in the x direction ((Δ x F)[x,y]).
[0036] The check of the gradient in the y direction is equivalent.
[0037] Other approaches are also conceivable, for example correlation with the gradient of the reference light point Ref[x,y] (which can be determined as above).
[0038] Filter consistency check
[0039] In difficult cases, filtering the intensity image (or the image that produces it) in a preprocessing step (e.g., temporally or spatially) can help. For more complex filters, each pixel will have a separate filter strength (α), which can prevent blurring or abrupt changes at the edges. If the filter strength varies within the periphery of the light spot, this usually leads to artifacts such as ghost light spots.
[0040] Therefore, it makes sense to determine a measure of the fluctuations of this filter strength in the periphery of the light spot, and accept the light spot only if these fluctuations are within a certain range. An example of such a measure is the standard deviation. It is also possible to compare the filter strength at the center of the light spot with the pixels in the rest of the periphery.
[0041] Single outlier check
[0042] In some cases, for example due to image noise, individual pixels erroneously have too high a brightness, which can result in ghost spots. This can be solved by comparing the brightness of the spot center (Fz) with the brightness of the spot periphery that is smaller than the typical spot diameter. (For example, ) to check. In this case, the ratio yields a value close to zero, while for real spots the ratio is much larger since by definition they do not fall into the amplitude background within the spot periphery. Now, if If it is within the expected range, the light spot is allowed.
[0043] In each of the approaches mentioned, it is checked whether the inspection result is within a certain range. The range can of course also be limited to a maximum or minimum value. Furthermore, the desired range can also depend on the position on the image sensor and the (possibly already determined) distance of the light spot, since the light spot shape can vary with the image range and the distance from the camera. Furthermore, the light spot periphery in the approaches mentioned does not necessarily have to be the same.
[0044] Fig. 9 A possible flow chart in the sense of the invention is shown. First, image data are acquired from the sensor and potential light spots are searched in preprocessing. In an iterative process, it is checked whether a valid light spot is involved or whether the light spot should be discarded. In a first check, it is checked whether the light spot shape is within an acceptable shape. In subsequent checks, it is checked whether the light spot has an allowed gradient, an allowed consistency and an acceptable brightness value (single outlier). If the light spot exceeds the accepted range even in only one check, the light spot is discarded as invalid. The allowed range for the light spot check / test can be specified individually by the user as required, in particular, the allowed / accepted range can be specified by the user.
[0045] The distance calculation is then performed only for the accepted light points.
Claims
1. A camera having an illumination source designed for spot illumination and a pixel array having pixels smaller than a typical spot, in, The camera is designed as follows: outputs at least one measurement value for each potential light spot found, Wherein, the potential light spot is characterized by a light spot center and at least one light spot periphery, wherein at least one characteristic value as a potential light spot is calculated based on the brightness or intermediate results of previous processing steps in the periphery of the light spot and / or the pixel position relative to the center of the light spot, and Therein, whether to accept or discard the corresponding potential light spot is determined based on at least one characteristic value.
2. The camera according to claim 1, which is designed as a time-of-flight camera.
