Method and device for cleaning a measurement of depth information

Abstract

The present invention relates to a method and a device for carrying out a method for cleaning depth information of a three-dimensional scene from interference signals, wherein a three-dimensional scene is irradiated by means of a micromechanical mirror that can be tilted about two axes with an irradiation pattern of emitted electromagnetic radiation in a spatially and time-dependent manner, wherein electromagnetic radiation reflected at the three-dimensional scene is detected, so that a reflection image of the three-dimensional scene is captured, from which the depth information of the scene is evaluated based on a comparison of the captured reflection image with a reference image.In order to remove interference signals from the measurement, the method provides parameters of the irradiation pattern at the time of acquisition of the reflection image in a cleaning step and compares a cleaning reference image, which is calculated from a feedback signal of the mirror, with the reflection image in order to identify and eliminate sections in the reflection image that contain interference signals.

Description

[0001] The present invention relates to a method for cleaning depth information of a three-dimensional scene from interference signals, comprising the following steps: a) emitting electromagnetic radiation by means of a radiation source, b) spatially illuminating at least one area of ​​the three-dimensional scene with the emitted electromagnetic radiation such that the at least one area of ​​the three-dimensional scene is irradiated with an irradiation pattern, c) spatially detecting electromagnetic radiation reflected from the three-dimensional scene by means of a detector such that a reflection image of at least one area of ​​the three-dimensional scene is recorded, which reflection image is generated at least partially by a reflection of the irradiation pattern on the three-dimensional scene, d) evaluating the depth information of the at least one area of ​​the three-dimensional scene based on a comparison of the image obtained in step c.captured reflection image with a reference image.

[0002] The present invention further relates to a device for carrying out the method.

[0003] Various methods are known in the art for measuring depth information in a three-dimensional scene. Among the most common methods is structured light projection, in which a predefined irradiation pattern consisting of points and / or stripes is projected onto a three-dimensional scene. A three-dimensional scene is defined as a three-dimensional object or an arrangement of several objects, for example, in a room. Such a three-dimensional scene could be observed, for example, with 3D glasses.

[0004] To evaluate the depth information of the three-dimensional scene, electromagnetic radiation reflected from the three-dimensional scene is spatially detected. Due to the varying distances between the surface of the three-dimensional scene and the radiation source, distortions arise, which are evaluated by comparing them with a reference image that would be created on a flat surface. Based on the known positions of the detector and the radiation source, as well as the captured distorted reflection image of the three-dimensional scene, the distance from the radiation source is evaluated as depth information for individual areas or points on the surface of the three-dimensional scene using the principle of triangulation.

[0005] The quality of the evaluated depth information depends on various factors. On the one hand, the quality is determined by system-specific properties such as the pixel density of the projected pattern; on the other hand, the quality also depends on external factors.

[0006] External factors that affect the measurement of depth information include, for example, lighting conditions influenced by scattered light sources such as lamps or sunlight or other devices for measuring depth information, which in turn project irradiation patterns onto the three-dimensional scene and overlap with the projected irradiation pattern of the actual measurement.

[0007] These external factors can result in the detection of reflected radiation from the three-dimensional scene that does not originate from a reflection of the projected radiation pattern of the actual device. Individual areas of the captured reflection image thus contain information originating from a radiation source with an unknown position. When evaluating the reflection image, this leads to incorrect evaluation of the sections affected by the interference signals, and the depth information is not accurately reproduced.

[0008] Various methods are known from the state of the art with which the measurement of depth information can be cleaned of such interference signals.

[0009] One approach involves identifying the frequency of the interfering radiation and adjusting the active illumination of the three-dimensional scene during the actual measurement so that it occurs at a time offset from the frequency of the interfering signals, or by detecting the reflected radiation over time and filtering out the interfering signal based on its known frequency. However, this approach has the disadvantage that it can only eliminate periodic interfering signals emitted at a specific frequency. Furthermore, the frequency of the interfering signals must be recorded, which impairs real-time evaluation of the depth information.

[0010] An alternative solution is to provide an additional unit for detecting interference signals in the measuring device for the depth information, which can, for example, exchange data with another measuring device that also detects depth information, thus aligning the timing of the two measuring devices.

[0011] It is also known to detect and filter out interfering signals with a specific wavelength, to discard abnormal deviations in the evaluated depth information as erroneous data, or to delay the active illumination accordingly when an interfering signal is detected. However, the delay in the active illumination is disadvantageous if the depth information is to be acquired and evaluated at a specific time interval. Furthermore, this approach also involves additional data acquisition and thus impairs the real-time capability of the measuring device.

[0012] The present invention is therefore based on the object of providing a method and a device with which the measurement of depth information can be cleaned of external interference signals without significantly impairing the real-time capability of the device or requiring additional components such as sensors for detecting the interference signals.

[0013] To solve this problem, the spatial illumination in step b) of the method mentioned above is carried out in such a way that the emitted electromagnetic radiation is deflected in a time-dependent manner to different locations in the three-dimensional scene by means of a micromechanical mirror that can be tilted about at least two axes, so that the irradiation pattern is composed of a time-dependent trajectory of the electromagnetic radiation, wherein the method further comprises a cleaning step comprising the following sub-steps: B1. Providing parameters of the irradiation pattern at the time of acquiring the reflection image in step c., B2. Comparing a cleaning reference image with the reflection image acquired in step c., wherein the cleaning reference image is calculated using a feedback signal from the micromechanical mirror, B3. Determining from the comparison in step B2 whether a section of the reflection image can be assigned to a reflection of the emitted electromagnetic radiation on the surface of the three-dimensional scene or not, B4. Identifying one or more sections of the reflection image that cannot be attributed to a reflection of the emitted electromagnetic radiation as an interference signal, B5. Cleaning the reflection image acquired in step c. or the depth information evaluated in step d. from the sections identified as interference signals.

[0014] The irradiation pattern is described, for example, by the parameters amplitude, phase, and frequency, which are determined by a drive of the micromechanical mirror. Thus, the parameters of the irradiation pattern for a specific time point can be calculated from the feedback signal of the micromechanical mirror for that specific time point, and the parameters can be determined from the time-dependent irradiation pattern. This enables a comparison between theoretically expected parameters for a specific time point and actually measured parameters.

[0015] A captured section of the reflection image, which was irradiated at a specific time with an irradiation pattern comprising specific parameters, can thus be assigned the operating parameters of the micromechanical mirror that were used to illuminate that section. The comparison reveals whether the captured parameters of the irradiation pattern match the parameters expected based on the mirror's deflection, and thus whether the captured section represents radiation from the irradiation pattern reflected from the scene.

[0016] The feedback signal can, for example, be an electrical signal that is used by the drive of the micromechanical mirror to drive the mirror.

[0017] By illuminating the three-dimensional scene over time using a micromechanical mirror, the operating parameters of the micromechanical mirror determine the spatial area of ​​the three-dimensional scene illuminated at any given time. A corresponding feedback signal from the micromechanical mirror is used to calculate a cleanup reference image. This reference image corresponds to the reflection image that would be expected if the irradiation pattern were radiated onto a flat surface without any interference signals.

[0018] It is therefore essential for the method according to the invention that the time of acquisition of a section of the reflection image is known so that a comparison can be made with the theoretically expected operating parameters of the micromechanical mirror at the time of acquisition.

[0019] The cleanup reference image is then compared with the reference image acquired in step c., whereby the comparison determines whether or not a portion of the acquired reflection image corresponds to a reflection of the emitted electromagnetic radiation on the surface of the three-dimensional scene.

[0020] During the comparison in step B2, either individual pixels of the reflection image and the cleanup reference image can be compared, or trajectory segments can be compared. The latter offers the advantage of saving computing power and thus speeding up the process. A segment within the meaning of step B3 is therefore either a single pixel of the reflection image, a plurality of pixels, or a trajectory segment.

[0021] If it was determined in step B3 that a section of the reflection image was not created by a reflection of the emitted electromagnetic radiation on the surface of the three-dimensional scene, this section is identified as an interference signal. This identification is repeated for all sections determined in step B3 as not related to the reflection of the emitted electromagnetic radiation, so that in a final step B5, either the reflection image acquired in step c. or the depth information evaluated in step d. is cleaned of the section identified as an interference signal.

[0022] In one embodiment, the depth information is cleaned according to step B5, i.e., such that the reflection image acquired in step c is cleaned of the interference signals identified in step B4 before the depth information is evaluated in step d. In other words, the depth information is evaluated based on a reflection image that has already been cleaned of interference signals.

[0023] Alternatively, in one embodiment, the cleaning of the depth information in step B5 is carried out such that the cleaning takes place after the depth information has been evaluated in step d.

[0024] The method according to the invention thus provides depth information of a three-dimensional scene that has been cleaned of interference signals by using existing system signals, i.e. the feedback signal of the micromechanical mirror, for the cleaning. An additional component, for example for detecting the frequencies of interference signals, is not required. Furthermore, the timing of the method does not have to be interrupted or adjusted for the cleaning. In other words, in the method according to the invention, the depth information is obtained as in a method known from the prior art, but is subjected to a cleaning process that is carried out exclusively with parameters already present in the system. Both periodic interference signals and aperiodic interference signals can be eliminated.Since no additional components are required for the cleaning, energy can also be saved, so that the method according to the invention can operate more energy-efficiently than methods known from the prior art.

[0025] The time-dependent illumination of the three-dimensional scene also has the advantage that other devices that simultaneously generate irradiation patterns on the three-dimensional scene are highly likely to illuminate a different area on the surface of the three-dimensional scene at a given time than the device whose measurement of the depth information is to be evaluated.

[0026] Various radiation sources are used to emit the electromagnetic radiation. It is advantageous for the emitted radiation to have the smallest possible divergence. In one embodiment, a laser is used as the radiation source. Alternatively, a laser diode with a collimator can be used. In another embodiment, multiple radiation sources are used, each emitting electromagnetic radiation. This allows the three-dimensional scene to be evaluated from multiple directions for its depth information, thus allowing even shadowed areas, for example, to be fully captured.

[0027] The emitted electromagnetic radiation can have a wavelength in the visible or non-visible range. Preferably, however, the wavelength of the emitted electromagnetic radiation is in the non-visible range, particularly preferably in the ultraviolet, infrared, or near-infrared range. It is understood that a detector with appropriate sensitivity in the specified frequency range is used as the detector.

[0028] In a further embodiment, bandpass filters are also used whose central wavelength lies in the range of the emitted electromagnetic radiation, so that radiation with wavelengths in the non-emitted range is filtered out.

[0029] In one embodiment, the cleanup reference image corresponds to the reference image in step d. Therefore, it is not necessary to calculate a new cleanup reference image in step B2. Instead, the existing reference image, which is required anyway for evaluating the depth information, can be used. This has the advantage that no additional computing power, which would require additional time, is required.

[0030] In a further embodiment, the detector is synchronized with the micromechanical mirror in a sub-step B0. This simplifies the correlation between the operating parameters of the micromechanical mirror and the parameters of the irradiation pattern at a specific time of illumination.

[0031] In one embodiment, the micromechanical mirror is periodically tilted about the first axis at a frequency f1 and periodically tilted about the second axis at a frequency f2, wherein a trajectory segment used for the comparison in step B2 has a length of at least a quarter period of the frequency f1 or f2, preferably a length of at least half a period of the frequency f1 or f2, and particularly preferably a length of a whole period of the frequency f1 or f2. This ensures that during the comparison in step B3, a distortion of the reflection image is not incorrectly identified as an interference signal that has only arisen due to the three-dimensional shape of the scene. Interference signals generally only impair the recorded reflection pattern in very limited local areas. This means that only individual or a few connected pixels are affected.By capturing and evaluating a correspondingly longer continuous section of the irradiation pattern, it is possible to determine whether a deviation from the reference image exists for a large part of the section or only for a few pixels, so that it can be assumed that this is an interference signal and not a distortion due to the three-dimensional scene.

[0032] In one embodiment, the frequency with which the micromechanical mirror is periodically tilted about the two axes is at least 1 kHz, wherein the frequency for the tilting movement about the two axes preferably differs.

[0033] In a further embodiment, the comparison in step B2 is carried out by means of cross-correlation and / or based on a neural network. For this purpose, the acquired reflection image is divided into N sections, for example, and a cross-correlation with the cleanup reference image is formed for each of the N sections. If a high level of similarity is determined for a specific section, the section is evaluated as a point on the surface of the three-dimensional scene and not as an interference signal. If, on the other hand, a low level of similarity is determined, the section is eliminated as an interference signal. Alternatively, in one embodiment, the x- and y-coordinates of pixels of the acquired reflection image that have a value other than 0 are determined.These determined x and y coordinates are then converted into a one-dimensional vector using a pairing function, with each component of the vector then being compared with a corresponding component of a one-dimensional vector using cross-correlation, which was also generated using a pairing function for the cleanup reference image. For example, a Cantor pairing function or a Szudzik pairing function can be used. In image comparison methods based on neural networks, a neural network is used to determine the similarity between the reflection image and the cleanup reference image. In particular, convolutional neural networks or neural networks with image encoders can be used. In one embodiment, a neural network with integrated cross-correlation layers is used.

[0034] In one embodiment, steps a. to d. are repeated periodically, wherein the cleaning step is not carried out with each run through steps a. to d., but after a number N of runs through steps a. to d., where N is a natural number, preferably 2 ≤ N ≤ 30. The repetition rate of the cleaning can thus be adapted to the respective application situation of the method according to the invention. If, for example, a three-dimensional scene is observed that changes only slightly over the observation period, for example when the detector is not moved or is moved only very slightly, cleaning with a low repetition rate is possible, whereas the cleaning can be repeated more often in the case of three-dimensional scenes that change significantly over time, for example when the detector is moving, for example in applications in the automotive industry. By adapting the repetition rate of the cleaning, computing power and thus energy can be saved.

[0035] In a further embodiment, steps a. to d. are repeated periodically, wherein the cleaning step is only performed if the depth information evaluated in step d. changes abruptly compared to a previous run of steps a. to d. In particular, an abrupt change is determined by comparing the evaluated depth information of one or more previous runs with the evaluated depth information of a current run in a further step e. and determining whether a change in the depth information exceeds a certain threshold value. Which change is classified as abrupt according to the inventive method is defined depending on the respective application.For extended reality applications, for example, the typical translation and rotation speed of the human head can be taken into account, thus defining a threshold for the change in depth information, above which a cleaning step is triggered. The threshold is preferably determined from an average over several pixels. This exploits the fact that the points on the surface of the three-dimensional scene, in contrast to noise signals, are subject to a different temporal change than the noise signals. The temporal change expected for the pixels on the surface of the three-dimensional scene can be estimated for the respective application situation as described above. If pixels in a temporal average exhibit a different behavior than the expected change, it can be assumed that a noise signal is present.

[0036] In the event that the interference signal exhibits a very similar change to the pixels on the surface of the scene resulting from the movement of the system, in one embodiment the deviation between the reflection image and the cleaning reference image is compared for several cleaning steps and the cleaning is stopped if it is determined that the deviation in a cleaning step varies little or not at all compared to the deviations determined in previous cleaning steps.

[0037] In a further embodiment, the measurement of the depth information according to steps a. to d. is stopped if it is determined in the cleaning step that a number A of sections identified as interference signals exceeds a certain limit, wherein measures are preferably taken to reduce the interference signals in a subsequent run through steps a. to d. In other words, the measurement of the depth information is stopped if a massive impairment of the measurement by external factors is determined. Such massive impairment occurs, for example, if the acquired reflection image is saturated or the irradiation pattern cannot be reconstructed. As a countermeasure against the impairments, the sensitivity or exposure of the detector can be reduced, for example.

[0038] In a further embodiment, a phase of the micromechanical mirror is modulated. The modulation can follow a random or non-random pattern. A single pattern or a specific sequence of patterns can be used for the modulation. With a sequence of patterns, it must be ensured that gaps in one pattern are covered by a subsequent pattern. Phase modulation of the micromechanical mirror can also ensure that an irradiation pattern emitted by another device at a specific time is not identified as the irradiation pattern to be evaluated.This could be the case, for example, if two depth measurement devices, each with a micromechanical mirror, happened to have identical or highly similar phases for the mirror's movement axes, thus generating identical or similar time-dependent irradiation patterns. However, phase modulation results in a deviation in the phase of the irradiation pattern, allowing an irradiation pattern from another device to be reliably eliminated.

[0039] In one embodiment, the time-dependent deflection of the electromagnetic radiation in step b. is carried out such that the trajectory, at least in sections, corresponds to a Lissajous figure or a spiral figure. For this purpose, the micromechanical mirror is tilted about at least two axes. Preferably, the time-dependent deflection of the electromagnetic radiation in step b. is carried out such that the electromagnetic radiation essentially follows a sinusoidal shape on at least two axes.

[0040] In one embodiment, the micromechanical mirror is driven by a piezoelectric drive, an electrostatic drive, or an electromagnetic drive.

[0041] In particular, in one embodiment, the feedback signal of the micromechanical mirror describes an angle or a parameter representing the angle in which the micromechanical mirror is deflected at a specific time relative to two axes of movement of the micromechanical mirror.

[0042] In a further embodiment, the feedback signal of the micromechanical mirror is generated by a piezo sensor, an electrostatic sensor, or an optical sensor, e.g., a photodiode. However, the feedback signal preferably does not need to be generated continuously. Rather, the feedback signal can be acquired once, for example, at the beginning of the observation period, and the further course of the trajectory of the irradiation pattern can be calculated based on known operating parameters of the micromechanical mirror. This also saves computing power and energy.

[0043] A CMOS sensor, a CCD sensor, or an event-based camera can preferably be used as the detector. The alignment of the detector does not necessarily have to be parallel to the emitted electromagnetic radiation, but can also be at an angle to the emitted electromagnetic radiation if this is expedient for the application, for example if a particularly space-saving device is required. Furthermore, in one embodiment, several detectors are provided for the spatial detection of the reflection image in step c., with each detector detecting a reflection image that is evaluated according to step d. and cleaned according to steps B1. to B5. In particular, when using several detectors, beam splitters can also be used to direct the reflected radiation onto the individual detectors in order to enable the most compact design possible despite multiple detectors.

[0044] In the case that the detector is an event-based camera, the event-based camera captures the irradiation pattern for a plurality N of sections on the three-dimensional scene as a reflection image in a time-dependent manner, the method comprising the further steps: E1. Providing parameters of the irradiation pattern based on the feedback signal of the micromechanical mirror for the respective time of acquisition of the irradiation pattern for the plurality N of the acquired sections, E2. Calculating at least one parameter of the irradiation pattern that is to be expected for an N+1 th section following the plurality N of detected sections based on the parameters of the irradiation pattern provided in step E1 for the plurality N of detected sections, E3. Comparing the parameter calculated in step E2 with parameters of the irradiation pattern for the N+1st section captured by the event-based camera, wherein the N+1st section is identified as a noise signal if the parameter of the captured irradiation pattern for the N+1st section deviates from the calculated parameter by a certain limit value.

[0045] In contrast to a detector in the form of a CMOS or CCD sensor, an event-based camera records each section or pixel in the reflection image aperiodically and individually. With the event-based camera, there is therefore no exposure time in which all pixels are recorded together. The event-based camera tracks the emitted electromagnetic radiation, which is deflected by the micromechanical mirror onto the three-dimensional scene in a time-dependent manner, and can thus track the course of the emitted electromagnetic radiation on the surface of the three-dimensional scene. This property is used to make a prediction based on the calculated parameters, for example, for a position range of the pixel that the event-based camera will next detect.If the position of the pixel detected by the camera does not match the prediction, the detected pixel is discarded as a noise signal. This embodiment is based on the assumption that a three-dimensional scene changes only slightly between two pixels. Large deviations, however, indicate a noise signal.

[0046] The parameters that are compared in step E3 can therefore be the parameters amplitude, phase or frequency already mentioned, but it can also be a specific position that a pixel occupies in the irradiation pattern and that can be described by corresponding coordinates.

[0047] The object underlying the invention is further achieved by a device for measuring depth information of a three-dimensional scene, wherein the device comprises a radiation source for emitting electromagnetic radiation, an illumination device for irradiating at least one region of the three-dimensional scene with an irradiation pattern, wherein the illumination unit is connected to the radiation source in such a way that, during operation of the device, the irradiation pattern is generated from the electromagnetic radiation emitted by the radiation source, a detector for spatially detecting electromagnetic radiation reflected from the three-dimensional scene, so that, during operation of the device, a reflection image of at least one region of the three-dimensional scene is captured, which reflection image is generated at least partially by a reflection of the irradiation pattern from the three-dimensional scene,and an evaluation unit for evaluating the depth information of at least one region of the three-dimensional scene, wherein the evaluation unit is configured and connected to the detector such that, during operation of the device, the depth information is evaluated based on a comparison of the captured reflection image with a reference image, wherein the illumination unit has a micromechanical mirror tiltable about at least two axes, with which, during operation of the device, the emitted electromagnetic radiation is deflected to different locations of the three-dimensional scene in a time-dependent manner, so that the irradiation pattern is composed of a time-dependent trajectory of the emitted electromagnetic radiation, wherein the evaluation unit is further configured such that, during operation of the device, a cleanup method is carried out with the evaluation unit, comprising the following steps: B1. Providing parameters of the irradiation pattern at the time of acquiring the reflection image, B2. Comparing a cleaning reference image with the reflection image captured by the detector, wherein the cleaning reference image is calculated using a feedback signal from the micromechanical mirror, B3. Determining from the comparison in step B2 whether a section of the reflection image can be assigned to a reflection of the emitted electromagnetic radiation on the surface of the three-dimensional scene or not, B4. Identifying one or more sections of the reflection image that cannot be attributed to a reflection of the emitted electromagnetic radiation as an interference signal, B5. Cleaning the captured reflection image or the evaluated depth information from the sections identified as noise.

[0048] It is understood that embodiments described for the method according to the invention are also transferable to the device according to the invention and vice versa.

[0049] The device according to the invention is characterized by a compact design that can be integrated, for example, into mobile devices such as smartphones or 3D glasses. The use of system-inherent signals to eliminate interference also saves energy, as less computing power or additional sensors are required, thus further facilitating mobile use.

[0050] Further advantages, features and possible applications of the present invention will become clear from the following description of various embodiments and the associated figures. Fig. Figure 1 shows an exemplary representation of a three-dimensional scene whose measurement of depth information is impaired by external noise. Fig. 2 shows an exemplary reflection image, which was taken with a detector of the type shown in Fig. 1 scene was captured. Fig. 3 shows a schematic representation of an embodiment of the device according to the invention. Fig. 4 shows a schematic three-dimensional representation of the Fig. 3 shown embodiment of the device during detection of a three-dimensional object. Fig. 5a shows an exemplary reference image of an embodiment of the method according to the invention. Fig. 5b shows an exemplary captured reflection image of an embodiment of the method according to the invention before the removal of interference signals. Fig. 5c shows an exemplary reflection image of an embodiment of the method according to the invention after the removal of interference signals. Fig. 6 shows a flow chart of a first embodiment of the method according to the invention. Fig. 7 shows a flow chart of a second embodiment of the method according to the invention. Fig. 8 shows a flow chart of a third embodiment of the method according to the invention. Fig. Figure 9 shows a schematic reflection image captured with an event-based camera. Fig. 10 shows a flow chart of a fourth embodiment of the method according to the invention.

[0051] To illustrate possible external interference factors, Fig. 1 shows an example of a three-dimensional scene 100 in which irradiation patterns 102, 102' are projected onto the three-dimensional scene 100 by two different devices 1 and 1'. The devices 1, 1' can be, for example, as in Fig. 1, 3D glasses are used. Additionally, the three-dimensional scene 100 is illuminated by a lamp 10, which projects a scattered light 110 onto the three-dimensional scene 100.

[0052] In Fig. 2 shows the reflection image 103, which is generated by a detector of the device 1 from the Fig. 1. It can be seen that the pattern 102 emitted by the device 1 overlaps in a region 103a of the reflection image 103 with a pattern 102' emitted by the further device 1' and in a region 103b with the scattered light 110. In the regions 103a and 103b, this results in an incorrect evaluation of the depth information in devices and methods known from the prior art for measuring depth information, since the captured reflection image 103 is not composed exclusively of the emitted electromagnetic radiation 101.

[0053] A detailed representation of the structure of the device 1, with which depth information is acquired free of interference signals, is shown in the Fig. 3 and Fig. 4. The device 1 according to the invention according to the embodiment shown comprises an irradiation unit 20 with a radiation source 2 and a micromechanical mirror 3, wherein the radiation source 2 emits the electromagnetic radiation 101 in the direction of the micromechanical mirror 3, so that the electromagnetic radiation 101 is imaged into the irradiation pattern 102 in the form of a time-dependent trajectory 300 with the aid of the micromechanical mirror 3. Appropriate optical components 5 ensure that the emitted electromagnetic radiation has a low divergence and strikes the three-dimensional scene 100 in a correspondingly focused manner.

[0054] The micromechanical mirror 3 is a mirror that can be tilted in two axes with a nearly sinusoidal motion, so that the irradiation pattern 102 is a Lissajous figure whose segments can be described by an amplitude, a frequency, and a phase. The frequency at which the micromechanical mirror 3 is periodically tilted about the two axes is at least 1 kHz, preferably at least 4 kHz, and can vary, with a frequency f1 of the tilting movement about one axis being, for example, 3 kHz and a frequency f2 of the tilting movement about the other axis being 15 kHz.

[0055] The micromechanical mirror 3 is driven by a drive 6, for example, a piezoelectric drive, which sends a feedback signal 105 to an evaluation unit 7. The feedback signal 105 is based, for example, on a measurement of the tilt angles of the micromechanical mirror 3, which are detected by a piezoelectric sensor provided in the irradiation unit 20.

[0056] The evaluation unit 7 also receives the reflection image 103 captured by the detector 4, for example, a CMOS or CCD sensor, which is composed of the radiation 106 reflected from the three-dimensional scene 100. The evaluation unit 7 thus receives all the information required for the cleaning step according to the inventive method. The evaluation unit 7 is therefore configured to perform the cleaning of the depth information according to the inventive method.

[0057] Out of Fig. 3 further shows that the evaluation unit 7 can also send signals to the drive 6 of the irradiation unit 20, so that if, for example, a massive impairment of the acquired data is detected by the evaluation unit 7, a corresponding signal can be sent to the drive 6 of the irradiation unit 20, whereby the irradiation of the three-dimensional scene 100 is interrupted or adjusted accordingly.

[0058] In Fig. Figure 4 illustrates the operation of the device 1. In the embodiment shown, the electromagnetic radiation 101 emitted by the radiation source 2 is deflected in a time-dependent manner by the micromechanical mirror 3, which can be tilted about two axes, such that an irradiation pattern 102 in the form of a Lissajou figure impinges on the three-dimensional scene 100. The detector 4 captures a reflection image 103 from the three-dimensional scene 100, in which a cup is arranged. In addition, the three-dimensional scene 100 is irradiated with an interference signal, which is emitted, for example, by a lamp 10.

[0059] In Fig. 5b shows a top view of the reflection image 103, which is used for the Fig. 1. From the illustration, it can be seen that the actually projected Lissajous pattern 102 contains not only distortions caused by a book, but also a distortion caused by the radiation of the lamp 10. By comparing it with a cleanup reference image 104, which is shown in Fig. 5a, a cleaned reflection image 103' can be calculated by the evaluation unit 7, which in Fig. 5c and only shows the distortions caused by the book in the three-dimensional scene 100.

[0060] The exact sequence of the method according to the invention for cleaning the measurement of the depth information according to a first embodiment is shown in Fig. 6. Accordingly, in a first step 601, the detector 4 is synchronized with the micromechanical mirror 3. Subsequently, the electromagnetic radiation 101 is emitted by the radiation source 2 in a step 602. In a subsequent step 603, the emitted electromagnetic radiation 101 is deflected by the micromechanical mirror 3 in a time-dependent manner such that a Lissajous figure is projected onto the three-dimensional scene 100. This electromagnetic radiation 101, projected in the form of the Lissajous figure onto the three-dimensional scene 100, is reflected off the surface of the three-dimensional scene and spatially detected by a detector 4 as a reflection image 103. At the same time, the current parameters of the Lissajous projection are also detected in step 604. In a further step 605, a cleanup reference image 104 is calculated based on the feedback signal 105 of the micromechanical mirror 3.This cleanup reference image 104 is compared with the acquired reflection image 103 in a subsequent step 606 to identify and remove interference signals in the reflection image 103, resulting in a cleaned reflection image 103'. In a final step 607, the depth information of the three-dimensional scene 100 is then evaluated using the cleanup reference image 104 and the cleaned reflection image 103'.

[0061] The Fig. The sequence of a second embodiment of the method according to the invention shown in Figure 7 differs from that shown in Fig. 6 is that, in a step 606', the depth information is first evaluated using the cleanup reference image 104 and the captured reflection image 103 with interference signals, before the interference signals in the depth information are identified and removed in a step 607'. This is done by comparing the cleanup reference image 104 with a trajectory of the evaluated depth information.

[0062] The Fig. The sequence of a third embodiment of the method according to the invention shown in Figure 8 essentially corresponds to the sequence of the method shown in Fig. 7. In contrast to the Fig. 7, the procedure according to Fig. 8, however, has a further step 606a', in which it is first checked whether the depth information evaluated in step 606' contains interference signals and the cleaning of the depth information after step 607' is only initiated if the check was affirmed.

[0063] In Fig. 9 further shows a schematic reflection image 103', which was recorded by a device 1 having an event-based camera as detector 4. In contrast to a detector 4 in the form of a CMOS or CCD sensor, an event-based camera records each pixel in the reflection image 103' aperiodically and individually. Therefore, with the event-based camera, there is no exposure time in which all pixels are recorded together. The event-based camera tracks the emitted electromagnetic radiation 101, which is deflected by the micromechanical mirror 3 onto the three-dimensional scene 100 in a time-dependent manner, likewise in a time-dependent manner and can thus track the course of the emitted electromagnetic radiation 101 on the surface of the three-dimensional scene 100.

[0064] Due to this property of an event-based camera, the measurement of the depth information can be cleaned of interference signals in such a way that the image points captured by the camera 4 are compared with the feedback signal 105 of the micromechanical mirror 3, which represents the direction of the Lissajous figure at a specific time, and it is determined whether the properties of the captured image point match the expected property according to the feedback signal 105.

[0065] A sequence of an embodiment of the method with an event-based camera 4 is shown in Fig. 10. Assuming that the event-based camera 4 has recorded N events in a time-dependent manner in a step 604', a prediction for the properties of the N+1st event can be made in a subsequent step 605' based on the feedback signal 105 of the micromechanical mirror 3 and the camera properties, so that a region of interest 301 can be defined. The region of interest 301 is calculated based on the parameters of the micromechanical mirror 3, i.e. the frequency and the tilt angles around the two axes and the camera parameters, such as the frequency with which pixels are recorded. If it is determined from the comparison of the parameters in step 606' that, as in Fig. 9, the N+1st event is located outside this region of interest 301, this pixel is discarded as an interference signal according to step 607". Reference symbol 1 device 1' foreign device 2 Radiation source 3 micromechanical mirror 4 Detector 5 optical components 6 Drive 7 Evaluation unit 10 Lamp 100 three-dimensional scenes 101 emitted electromagnetic radiation 102 irradiation patterns 102' Irradiation pattern of a foreign device 103 Reflection image 103a, 103b sections with interference signal 104 Reference image, cleanup reference image 105 Feedback signal 106 reflected electromagnetic radiation 300 time-dependent trajectory 601 first step 602 second step 603 third step 604, 604' fourth step 605, 605' fifth step 606, 606', 606" sixth step 606a' Verification step 607, 607', 607" seventh step

Claims

[1] Method for cleaning up a measurement of depth information of a three-dimensional scene (100) from interference signals by the following steps: a) Emitting electromagnetic radiation (101) by means of a radiation source (2), b) Spatial illumination of at least one area of ​​the three-dimensional scene (100) with the emitted electromagnetic radiation (101) such that at least one area of ​​the three-dimensional scene is irradiated with an irradiation pattern (102), c) Spatial detection of electromagnetic radiation (106) reflected from the three-dimensional scene (100) by means of a detector (4) such that a reflection image (103) of at least one area of ​​the three-dimensional scene (100) is captured, which is generated at least partially by a reflection of the irradiation pattern (102) from the three-dimensional scene (100), d) Evaluating the depth information of at least one area of ​​the three-dimensional scene (100) by comparing the reflection image (103) captured in step c. with a reference image (104), characterized by , that The spatial illumination in step b) is carried out such that the emitted electromagnetic radiation (101) is deflected time-dependently to different locations of the three-dimensional scene (100) by means of a micromechanical mirror (3) that can be tilted about at least two axes, so that the irradiation pattern (102) is composed of a time-dependent trajectory (300) of the emitted electromagnetic radiation (101), and The process includes a cleanup step comprising the following substeps: B1. Providing parameters of the irradiation pattern (102) at the time of acquisition of the reflection image (103) in step c., B2. Comparing a cleanup reference image with the reflection image (103) acquired in step c., wherein the cleanup reference image is calculated using a feedback signal (105) from the micromechanical mirror (3), B3. Determine from the comparison in step B2 whether a section (103a, 103b) of the reflection image (103) can be attributed to a reflection of the emitted electromagnetic radiation (101) on the surface of the three-dimensional scene (100) or not, B4. Identifying one or more sections (103a, 103b) of the reflection pattern (103) that cannot be attributed to a reflection of the emitted electromagnetic radiation (101) as interference signals, B5. Clean up the reflection image captured in step c. or the depth information evaluated in step d. from the sections identified as interference (103a, 103b). [2] Method according to the preceding claim, wherein the cleanup reference image corresponds to the reference image from step d. [3] Method according to one of the preceding claims, wherein the cleaning step comprises a substep B0. in which the detector (4) is synchronized with the micromechanical mirror (3). [4] Method according to one of the preceding claims, wherein the micromechanical mirror is periodically tilted about the first axis at a frequency f1 and periodically tilted about the second axis at a frequency f2, wherein a trajectory segment used for comparison in step B2 has a length of at least one quarter of a period of the frequency f1 or f2, preferably a length of at least one half a period of the frequency f1 or f2 and particularly preferably a length of one whole period of the frequency f1 or f2. [5] Method according to any of the preceding claims, wherein the comparison in step B2. is carried out by means of a cross-correlation and / or on the basis of a neural network. [6] A method according to any of the preceding claims, wherein steps a. to d. are repeated periodically, wherein the cleaning step is not performed at each iteration of steps a. to d., but after a number N of iterations of steps a. to d., wherein N is a natural number, wherein preferably 2 ≤ N ≤ 30. [7] Method according to any of the preceding claims, wherein steps a. to d. are repeated periodically, wherein the cleanup step is only performed if the depth information evaluated in step d. changes abruptly compared to a previous iteration of steps a. to d. [8] Method according to the preceding claim, wherein the abrupt change is detected by comparing, in step e., the evaluated depth information of one or more previous runs with the evaluated depth information of a current run and determining whether a change in the depth information exceeds a certain threshold. [9] Method according to any of the preceding claims, wherein the measurement of the depth information according to steps a. to d. is stopped when, in the cleanup step, it is determined that a number A of sections identified as interference signals exceed a certain limit, preferably taking measures to reduce the interference signals in a subsequent iteration of steps a. to d. [10] Method according to any of the preceding claims, wherein a phase of the micromechanical mirror is modulated. [11] Method according to one of the preceding claims, wherein the time-dependent deflection of the electromagnetic radiation in step b. is carried out such that the trajectory corresponds at least section by section to a Lissajous figure or a spiral figure. [12] Method according to the preceding claim, wherein the time-dependent deflection of the electromagnetic radiation in step b. is carried out such that the electromagnetic radiation follows a sinusoidal shape on at least two axes. [13] Method according to the preceding claim, wherein a drive (6) of the micromechanical mirror is a piezoelectric drive, an electrostatic drive or an electromagnetic drive. [14] Method according to one of the preceding claims, wherein the feedback signal of the micromechanical mirror describes an angle or a parameter representing the angle in which the micromechanical mirror is deflected at a certain time relative to two axes of motion of the micromechanical mirror. [15] Method according to any of the preceding claims, wherein the feedback signal of the micromechanical mirror is generated by a piezoelectric sensor, an electrostatic sensor or an optical sensor. [16] Method according to any of the preceding claims, wherein the detector is a CMOS sensor, a CCD sensor or an event-based camera. [17] Method according to one of the preceding claims, wherein the detector is an event-based camera, wherein the event-based camera captures the irradiation pattern for a plurality N of sections on the three-dimensional scene as a reflection image in a time-dependent manner, wherein the method comprises the further steps: E1. Providing parameters of the irradiation pattern based on the feedback signal of the micromechanical mirror for the respective time of acquisition of the irradiation pattern for the majority N of the acquired sections, E2. Calculate at least one parameter of the irradiation pattern that is to be expected for an N+1th section following the plurality N of recorded sections, based on the irradiation pattern parameters provided in step E1 for the plurality N of recorded sections. E3. Comparing the parameter calculated in step E2 with parameters of the irradiation pattern captured by the event-based camera for the N+1th section, the N+1th section being identified as a noise signal if the parameter of the detected irradiation pattern for the N+1st section deviates from the calculated parameter by a certain threshold. [18] Method according to one of the preceding claims, wherein the spatial detection in step c. is carried out with multiple detectors, each detector capturing a reflection image which is evaluated according to step d. and cleaned according to steps B1. to B5. [19] A device (1) for measuring depth information of a three-dimensional scene, wherein the device comprises a radiation source for emitting electromagnetic radiation, an illumination device (3) for irradiating at least one area of ​​the three-dimensional scene with an irradiation pattern, wherein the illumination unit is connected to the radiation source in such a way that, during operation of the device, the irradiation pattern is generated from the electromagnetic radiation emitted by the radiation source, and a detector for spatial detection of electromagnetic radiation reflected from the three-dimensional scene, such that, during operation of the device, a reflection image of at least one area of ​​the three-dimensional scene is captured, which is generated at least partially by a reflection of the irradiation pattern from the three-dimensional scene.and an evaluation unit (7) for evaluating the depth information of at least one area of ​​the three-dimensional scene, wherein the evaluation unit is configured and connected to the detector such that, during operation of the device, the depth information is evaluated by comparing the captured reflection image with a reference image, wherein the illumination unit has a micromechanical mirror that can be tilted about at least two axes, with which, during operation of the device, the emitted electromagnetic radiation is deflected to different locations of the three-dimensional scene in a time-dependent manner, such that the irradiation pattern consists of a time-dependent trajectory of the emitted electromagnetic radiation, wherein the evaluation unit is further configured such that, during operation of the device, a cleanup procedure is carried out with the evaluation unit comprising the following steps: B1. Providing parameters of the irradiation pattern at the time of recording the reflection image, B2. Comparing a cleanup reference image with the reflection image captured by the detector, wherein the cleanup reference image is calculated using a feedback signal from the micromechanical mirror, B3. Determine from the comparison in step B2 whether a section of the reflection image can be attributed to a reflection of the emitted electromagnetic radiation on the surface of the three-dimensional scene or not. B4. Identifying one or more sections of the reflection pattern that cannot be attributed to a reflection of the emitted electromagnetic radiation as interference, B5. Clean up the captured reflection image or the evaluated depth information from the sections identified as interference signals.

Images