Camera with temporal filter

By introducing time filter technology and spot illumination into time-of-flight cameras, the filter intensity is dynamically adjusted to identify spots and reduce noise interference, the problem of insufficient measurement accuracy in the prior art is solved, and higher spot recognition accuracy is achieved and distance measurement reliability.

CN119024357APending Publication Date: 2024-11-26IFM ELECTRONIC GMBH +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410664038.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2024-05-27
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing time-of-flight cameras have shortcomings in measurement accuracy, especially when dealing with noise and identifying light spots, it is difficult to effectively distinguish between real light spots and noise-induced ‘ghosts’, resulting in a decrease in measurement accuracy.

Method used

Using time filter technology, by calculating the difference function and weighted average between the current image and the previous image, identifying the light spots and preventing noise interference, dynamically adjusting the filter intensity α to optimize the light spot recognition and denoising process, combined with light spot illumination technology to improve the accuracy of distance measurement.

Benefits of technology

It significantly improves the measurement accuracy of the time-of-flight camera, reduces noise interference, ensures the accuracy and consistency of spot recognition, and enhances the reliability of distance measurement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119024357A_ABST
    Figure CN119024357A_ABST
Patent Text Reader

Abstract

The invention provides a camera with a temporal filter, the camera having an array of pixels and an illumination source, the illumination source emitting light in the form of a dot pattern or a plurality of light spots, at least one measurement being output for at least a portion of the light spots, the temporal filter being applied, the temporal filter being applied to the illumination source, and the temporal filter being applied to the illumination source. The temporal filter includes at least one previous frame in the evaluation of the current frame, where the temporal filter identifies changes in spot position and / or characteristics such as amplitude, phase, distance, raw data and correspondingly attenuates the filter effect (alpha).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a camera, in particular to a time-of-flight camera with a time filter. Background Art

[0002] From US5442462A, a device for unidirectional time-domain smoothing of a current image having a previous image sequence is known, wherein the device comprises: a device for calculating a difference function between an original value of a single pixel of the current image and a smoothed previous image value corresponding to at least one pixel of at least one previous image; and a device for generating a smoothed pixel value of a single pixel of the current image by calculating a weighted sum of the smoothed previous image value and the original value of a single pixel of the current image, wherein the weight is a function of the difference function and at least one feature of at least a part of the current image, and the at least one feature is different from the difference between the single pixel value and at least one pixel of at least one previous image.

[0003] "Time of flight camera" shall include all time of flight camera systems or 3D-TOF camera systems that obtain distance directly from the time of flight (dTOF) or from the phase shift of the emitted and received radiation (iTOF). PMD cameras with photonic mixing detectors (PMD) as described in DE19704496A1 or cameras with SPAD (Single Photon Avalanche Diode) arrays as receiving sensors are particularly suitable as time of flight cameras or 3D-TOF cameras. Summary of the invention

[0004] The object of the present invention is to improve the measurement accuracy of a time-of-flight camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the attached picture:

[0006] Figure 1 A photograph of a scene with conventional lighting is shown;

[0007] Figure 2 A photograph of a scene with spot lighting is shown;

[0008] Figure 3 The correlation between the filter effect and the previous image changes is shown;

[0009] Figure 4 An amplitude image showing a point illumination of a hand in a video sequence; and

[0010] Figure 5 Shown with Figure 4The image that corresponds to the filter effect to be used (alpha image) and the point centers found. DETAILED DESCRIPTION

[0011] In videos, consecutive images are often highly correlated and can therefore be used for averaging and noise reduction. By combining the Time of Flight (ToF) camera considered here with spot illumination, special features can be exploited specifically.

[0012] With point lighting, the scene is illuminated only at single points, and distances are evaluated only at these points.

[0013] Figure 1 An example of a uniformly illuminated scene is shown. Figure 2 A scene with spot lighting is shown.

[0014] Here, a light spot usually covers several pixels. The light spot is characterized on the sensor, for example, by the center of the light spot (e.g., the center point of the light spot, the center of gravity or the brightest pixel) and the periphery of the light spot surrounding the center (e.g., a circular area with a radius of 3.5 pixels or a square area of ​​9 pixels×9 pixels). For evaluation, for example, a weighted average is performed on each light spot periphery before the desired data (e.g., phase, amplitude, distance, raw data) are further processed in a conventional manner to output.

[0015] To find the light spots, use e.g. the amplitude image As is common in ToF. Here, all light points emitted by the illumination source should be identified as far as possible, without identifying any non-existent "ghost" light points caused by noise. For this purpose, the image must be sharp and noise-free.

[0016] To this end, we assume a temporal filter that is filtered based on the current (unfiltered) image (I n ) and the previous filtered image Calculate the current filtered image

[0017]

[0018] The variable α or filter effect α (0<α≤1) characterizes the strength of the temporal filter, wherein the smaller the filter effect α, the stronger the filtering, and α=1 deactivates the temporal filter or performs a reset operation. In addition,

[0019] Although amplitude images are often used to find light spots, temporal filters can also be used on other images, such as raw data or Re / Im.

[0020] Adaptation α

[0021] In the sense of the present invention, α is pixel-individual, i.e. the temporal filter selects the best α for each pixel. This means, for example, that the user specifies a desired base α (α0), but the algorithm will choose a larger α (e.g., α=1), i.e. a reset is performed whenever: the light spot on the sensor has moved; or the light spot has changed its characteristics (e.g., brightness / distance).

[0022] For example, Figure 3 As shown in n ) image and the previous (I n-1 or ) images and a threshold value (b) to identify both. For example, the threshold value may also include expected image noise.

[0023] Correlation between filter strength (α) and change (Δ) from previous image.

[0024] exist Figure 4 and Figure 5 As can be seen in , moving edges are thereby identified and filtering is prevented (α=1).

[0025] Figure 4 Amplitude images of the spot illumination of the hand in the video sequence are shown. The arrows indicate the direction of the hand movement. Figure 5 The associated alpha image is shown. The black dots indicate the found centers of the light spots.

[0026] Consistency of the light spot periphery

[0027] Since the data are usually averaged over the periphery of the spot, the data must be as consistent as possible.

[0028] Therefore, for all pixels around the light spot, α should be chosen to be as identical as possible. The method for this is:

[0029] If the light spot position is already known (approximately), the average / maximum value of these pixels can be used for each light spot periphery of all containing pixels. If the light spot position is not yet known, the approximate position from the previous image can be used.

[0030] The previously determined alpha image can be blurred using a low pass.

[0031] The previously determined alpha image may be fixed, ie alpha is constant in predetermined tiles within the image.

[0032] The center of a spot is usually brighter than the edges of the spot. These differences within the periphery of the spot can cause inconsistencies in the alpha image. This can be avoided by allowing larger or smaller changes (Δ) between the current image and the previous image, depending on the brightness of the corresponding pixels.

[0033] The temporal filter identifies motion within a predefined region (e.g., a tile or the entire image) in the image (e.g., based on using the number or proportion of reset pixels) and increases the cardinality α (α0) in this region according to the strength of the motion in the corresponding region.

[0034] Preventing outliers

[0035] To prevent the above-mentioned "ghost" spots, it is important that there are no outliers in areas of weak signal, i.e. no pixels whose brightness significantly exceeds that of their surroundings simply due to image noise. This can be prevented by choosing the threshold (b) according to whether the change (Δ) increases or decreases the amplitude. If the reset is prevented when the amplitude jumps upward once, the temporal filter remains active and the outliers are hidden.

[0036] Behavior after reset: If the user chooses a smaller cardinality α (e.g., α0 = 0.1), the algorithm can optimize the behavior after reset. If the reset occurs in the nth frame (α = 1), it can be seen in equation (1) If it is assumed that no reset occurs in the subsequent frames (n+1, n+2, ...), then it is as follows:

[0037]

[0038]

[0039]

[0040]

[0041] Resetting the nth frame gives a very large weight to some frames after the reset, which may have an adverse effect on noise and motion artifacts. If you want to ensure that all frames after the reset are equally weighted, the best choice for the mth frame after the reset is α = 1 / (m+1). Therefore, if you choose the maximum value from α0 and 1 / (m+1) for α, you can get a noise-free result.

[0042] In principle, the process can also be applied to flood lighting with no, few or only weakly distinct light points.

Claims

1. A camera having a pixel array and an illumination source, The illumination source emits light in a dot pattern or multiple light points. in, outputting at least one measurement value for at least a portion of the light spot, wherein a temporal filter is applied, the temporal filter including at least one previous frame in the evaluation of the current frame, Therein, the temporal filter detects changes in the light spot position and / or in characteristics such as amplitude, phase, distance, raw data and attenuates the filter effect accordingly.

2. A camera according to claim 1, wherein the pixels of the camera are smaller than a typical light spot and the temporal filter of the camera provides a mechanism to ensure the consistency of the light spot periphery.

3. The camera according to claim 1 or 2, wherein: The attenuation of the filter effect at each pixel depends on the comparison with the previous frame. Is the brightness / amplitude of the comparison larger or smaller.

4. A camera according to any one of the preceding claims, wherein: The filter effect (α) is adjusted according to the number of frames after the reset (m) to prevent over-weighting of specific frames.

5. The camera according to any of the preceding claims, designed as a time-of-flight camera with an illumination source for emitting pulsed and / or modulated light, The camera includes a receiving sensor having a pixel array, in, The pixels are designed as SPAD pixels or as PMD pixels.