Depth measurement via display

Abstract

To provide a device and a method that allow depth measurement with high reliability through a display, with low technical efforts and low requirements from the viewpoint of technical resources and cost.SOLUTION: A display device comprises: an irradiation source 114 configured to project an irradiation pattern including a plurality of irradiation characteristics; an optical sensor 120 configured of an optical sensor 118 having a photosensitive area 120; a light-transmissive display 112 configured to display information, the light-transmissive display 112 having the irradiation source 114 and the optical sensor 118 arranged in a propagation direction of the irradiation pattern in front of the display 112; and an evaluation device 124 configured to evaluate a first image 122. The evaluation device 124 is configured to generate a depth map for true characteristics.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to a display device and a method for depth measurement using a translucent display, as well as various uses of the display device. The device, method, and use according to the present invention can be specifically employed in various fields such as daily life, security technology, gaming, transportation technology, production technology, art, photography including digital photography or videography for documentary or technical purposes, information technology, agriculture, crop protection, maintenance, cosmetics, medical technology, or science. However, other applications are also possible. [Background technology]

[0002] Several display devices are known. Recent developments concerning devices with displays have shown that the display area must cover the entire available space, and the frame surrounding the display must be as small as possible. As a result, electronic components and sensors (e.g., front cameras, flashlights, proximity sensors, and even 3D image sensors) can no longer be placed within the frame and must be placed beneath the display. However, the most common 3D imaging techniques and systems (e.g., 3D imaging systems based on structured light or 3D time-of-flight (ToF)) cannot be placed beneath the display without difficulty.

[0003] Until now, it has not been known that 3D imaging systems based on structured light or 3D-ToF operate under a display, that is, without creating an empty window that does not contain microcircuits and / or microwirings for positioning the components or devices of the 3D imaging system to be "viewed" through the window.

[0004] In the case of structured light, the main problem lies in the microstructure of the microcircuits and / or microwirings of the transparent display, and consequently, the low transmittance of light passing through the display. This microstructure is due to the electrode matrix used to address individual pixels. Furthermore, since the metal cathode of a single pixel is not transparent, the pixel itself exhibits a reciprocal lattice. In principle, a display structure can be made transparent or translucent as a whole, including the electrodes, by using specific materials, but to date, no transparent or translucent display exists that does not have a lattice-like microstructure.

[0005] Structured light-based 3D imaging devices are based on projecting a point cloud containing thousands of points and known patterns into a scene. The microstructure of a transparent or translucent display acts like a diffraction grating structure for laser light. Many projectors in structured light imaging devices are based on a laser source that projects a clearly defined dot pattern, so that this pattern is affected by the diffraction grating effect of the display, causing all spots in the dot pattern to exhibit a high order of diffraction. This has a fatal impact on structured light imaging devices because the extra unwanted points caused by the diffraction grating structure make the algorithm for finding the intended pattern extremely complex.

[0006] Furthermore, conventional structured light imaging devices use a considerably large number of projection points. Because transparent displays have very low light transmittance, even in infrared (IR) wavelengths of 850 nm and 940 nm, which are typical wavelengths for 3D imaging devices, very high output power is required for the structured light projector to gain enough power to pass through the display so that it can be detected by the imaging device. In addition, the structured light projector must be placed below the display, which results in additional light absorption. The combination of a large number of points and low light transmittance can result in poor ambient light robustness.

[0007] In the case of 3D-ToF sensors, different display structures have different refractive indices due to reflections on the display surface that cause multi-reflections, as well as differences in delay when light passes through the display, which hinders robust functionality when used behind the display. Furthermore, 3D-ToF sensors also require a large amount of light to illuminate the scene. Moreover, the illumination needs to be uniform. When the light transmittance of the display is low, it is difficult to provide sufficient light, and the grid structure affects the uniformity of illumination.

[0008] General 3D sensing systems have problems measuring through a transparent display. Current devices use cutouts provided in the display. By doing so, the sensor is not obstructed by diffraction optical effects.

[0009] DE202018003644U1 describes a portable electronic device, which includes a bottom wall and side walls that cooperate to define a cavity, the side walls having edges that define an opening leading to the cavity; a protective layer covering the opening and surrounding the cavity; and a vision subsystem disposed within the cavity and between the protective layer and the bottom wall, serving to provide a depth map of an object outside the protective layer, the vision subsystem comprising a clip assembly for carrying optical components that cooperate to generate information for the depth map, the clip assembly including a first bracket arranged to support and hold the optical components at a certain distance from each other and a second bracket having a body fixed to the first bracket (the second bracket having a protrusion extending away from the body).

[0010] US9,870,024B2 describes an electronic display including a plurality of layers such as a cover layer, a color filter layer, a display layer including light-emitting diodes or organic light-emitting diodes, and a thin-film transistor layer. In one embodiment, the layer includes a substantially transparent region disposed above the camera. The substantially transparent region allows light from the outside to reach the camera and enables the camera to record an image.

[0011] US10,057,541B2 describes an imaging device and an imaging method. This imaging device comprises a transparent display panel and a camera facing the bottom surface of the transparent display panel for acquiring an image located in front of the transparent display panel, with the shutter time synchronized with the period during which the transparent display panel displays a black image.

[0012] US10,215,988B2 describes an optical system for displaying light from a scene, comprising an active optical component including a first plurality of optical apertures, a photodetector, a processor, a display, and a second plurality of optical apertures. The first plurality of optical apertures are arranged to provide an optical input to the photodetector. The photodetector is arranged to receive the optical input and convert the optical input into electrical signals corresponding to intensity and position data. The processor is connected to receive data from the optical detector and process the data for the display. The second plurality of optical apertures are arranged to provide an optical output from the display.

[0013] WO2019 / 042956A1 describes a detector for determining the position of at least one object. The detector comprises: - at least one sensor element having a matrix of photosensors, each photosensor having a photosensitive area, each photosensor being designed to generate at least one sensor signal in response to illumination of its respective photosensitive area by a reflected light beam propagating from an object to the detector, and the sensor element being adapted to determine at least one reflected image; and - at least one evaluation device being adapted to select at least one reflected feature of the reflected image, and configured to determine at least one longitudinal region of the selected reflected feature of the reflected image by evaluating a combined signal Q from the sensor signal, and being adapted to determine at least one displacement region in at least one reference image corresponding to the longitudinal region, and being adapted to match at least one reference feature in the displacement region with the selected reflected feature. [Overview of the project] [Problems that the invention aims to solve]

[0014] Therefore, an object of the present invention is to provide an apparatus and method that addresses the aforementioned technical problems of known apparatuses and methods. Specifically, an object of the present invention is to provide an apparatus and method that enables reliable depth measurement through a display with low technical effort and low requirements in terms of technical resources and cost. [Means for solving the problem]

[0015] This problem is solved by the present invention, which has the features of an independent patent claim. Advantageous developments of the present invention, which can be realized individually or in combination, are shown in the dependent claims and / or the following specification and detailed embodiments.

[0016] Where used below, the terms “have,” “equip,” or “include,” or any grammatical variations thereof, are used in a non-exclusive manner. Thus, these terms can refer to both situations in which the entity described in this context has no further features other than those introduced by these terms, and situations in which one or more further features exist. For example, the expressions “A has B,” “A equips B,” and “A includes B” can refer to both situations in which A has no other elements other than B (i.e., A exclusively constitutes B), and situations in which, in addition to B, entity A has one or more elements, such as element C, elements C and D, or even more elements.

[0017] Furthermore, note that the terms “at least one,” “one or more,” or similar expressions indicating that a feature or element may appear more than once, are typically used only once when introducing each feature or element. In most cases below, the expressions “at least one” or “one or more” will not be repeated when referring to each feature or element, despite the fact that those features or elements may appear more than once.

[0018] Furthermore, where used below, the terms “preferably,” “more preferably,” “particularly,” “more especially,” “specifically,” “more specifically,” or similar terms are used in relation to any feature without limiting the possibility of alternatives. Thus, the features introduced by these terms are arbitrary features and are not intended to limit the scope of the claims in any sense. The present invention can be carried out using alternative features as will be recognized by those skilled in the art. Similarly, features introduced by “in one embodiment of the present invention” or similar expressions are intended to be arbitrary features without any limitations on alternative embodiments of the present invention, without any limitations on the scope of the present invention, and without any limitations on the possibility of combining such features with any or no other features of the present invention.

[0019] In a first aspect of the present invention, a display device is disclosed. As used herein, the term “display” can mean any device of any shape configured to display an item of information, such as at least one image, at least one figure, at least one histogram, at least one piece of text, or at least one symbol. The display may be at least one monitor or at least one screen. The display may have any shape, preferably rectangular. As used herein, the term “display device” can generally mean at least one electronic device including at least one display. For example, the display device may be at least one device selected from the group consisting of a television, a smartphone, a game console, a personal computer, a laptop, a tablet, at least one virtual reality device, or a combination thereof.

[0020] The display device is - At least one illumination source configured to project at least one illumination pattern containing multiple illumination features onto at least one scene; - At least one photosensitive sensor having at least one photosensitive area, wherein the photosensitive sensor is configured to determine at least one first image including a plurality of reflection features generated by the scene in response to illumination by the illumination feature; - At least one translucent display configured to display information, wherein the illumination source and the light sensor are positioned in front of the display in the direction of propagation of the illumination pattern; - At least one evaluation device, the evaluation device is configured to evaluate the first image, the evaluation of the first image includes identifying the reflection features of the first image, sorting the identified reflection features with respect to brightness, each of the reflection features includes at least one beam profile, and the evaluation device, by analysis of those beam profiles, determines at least one vertical coordinate z of each of the reflection features DPR At least one evaluation device configured to determine, It is equipped with, The evaluation device, the vertical coordinate z DPR By using, the system is configured to clearly match the reflection features to the corresponding illumination features, and the matching is performed by decreasing the brightness of the reflection features, starting with the brightest reflection feature, and the evaluation device is configured to classify reflection features that match the illumination features as true features and reflection features that do not match the illumination features as false features, and the evaluation device is configured to exclude the false features, and the vertical coordinate z DPR It is configured to generate a depth map of true features using [this method].

[0021] As used herein, the term “scene” may refer to at least one arbitrary object or spatial area. A scene may include at least one object and its surrounding environment.

[0022] The illumination source is configured to project at least one illumination pattern containing multiple illumination features onto the scene. As used herein, the term “illumination source” may generally refer to at least one arbitrary device adapted to provide at least one illumination light beam for illuminating a scene. The illumination source may be adapted to illuminate the scene directly or indirectly, and the illumination pattern is reflected or scattered by the surface of the scene, thereby directed at least partially to the light sensor. The illumination source may be adapted to illuminate the scene, for example, by directing a light beam to a scene that reflects light beams. The illumination source may be configured to generate an illumination light beam for illuminating a scene.

[0023] The illumination source may include at least one light source. The illumination source may include multiple light sources. The illumination source may include an artificial illumination source, in particular at least one laser source, and / or at least one incandescent lamp, and / or at least one semiconductor light source, such as at least one light-emitting diode, in particular organic and / or inorganic light-emitting diodes. As an example, the light emitted by the illumination source may have wavelengths of 300 to 1100 nm, in particular 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range, such as in the range of 780 nm to 3.0 μm, may be used. Specifically, light in the near-infrared region, particularly in the range of 700 nm to 1100 nm, to which silicon photodiodes are applicable, can be used. The illumination source may be configured to produce at least one illumination pattern in the infrared region. Using light in the near-infrared region allows the light to be undetectable or only slightly detected by the human eye, but detectable by silicon sensors, in particular standard silicon sensors.

[0024] As used herein, the term “ray” generally refers to a line perpendicular to the wavefront of light that indicates the direction of energy flow. As used herein, the term “beam” generally refers to a collection of rays. Hereafter, the terms “ray” and “beam” are used synonymously. Where further used herein, the term “light beam” generally refers to a quantity of light, specifically a quantity of light traveling essentially in the same direction, including the possibility that the light beam has an expansion angle or divergence angle. A light beam can have spatial extent. Specifically, a light beam can have a non-Gaussian beam profile. The beam profile may be selected from the group consisting of trapezoidal beam profiles; triangular beam profiles; and conical beam transverse intensity profiles. A trapezoidal beam profile may have a plateau region and at least one edge region. A light beam may be a Gaussian light beam or a linear combination of Gaussian light beams, as outlined in more detail below. However, other embodiments are also possible.

[0025] The irradiation source can be configured to emit light at a single wavelength. Specifically, the wavelength may be in the near-infrared region. In other embodiments, the irradiation may be adapted to emit light having multiple wavelengths, enabling additional measurements in other wavelength channels.

[0026] The illumination source may be at least one multiple beam light source, or may include multiple beam light sources. For example, the illumination source may include at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the illumination source may comprise at least one laser and / or laser source. Various types of lasers may be employed, such as semiconductor lasers, double heterostructure lasers, external cavity lasers, separated and contained heterostructure lasers, quantum cascade lasers, dispersed Bragg reflector lasers, polariton lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, indium arsenide lasers, transistor lasers, diode-pumped lasers, dispersed feedback lasers, quantum well lasers, interband cascade lasers, gallium arsenide lasers, semiconductor ring lasers, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alternatively, non-laser light sources such as LEDs and / or light bulbs may be used. The illumination source may include one or more diffractive optical elements (DOEs) adapted to generate an illumination pattern. For example, the illumination source may be adapted to generate and / or project a point cloud, and may include one or more of the following: at least one digital photoprocessing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one array of light-emitting diodes; at least one array of laser light sources. Given their generally defined beam profiles and other characteristics of handling, the use of at least one laser source as the illumination source is particularly preferred. The illumination source may be integrated into the housing of the display device.

[0027] In one embodiment, the illumination source may be a single or multiple beam source and may be configured to project at least one illumination pattern, such as at least one dot pattern. The illumination pattern may be generated as follows: The illumination source may be configured to generate at least one light beam. The illumination source may be positioned in front of the display in the direction of propagation of the illumination pattern. Thus, the beam path of the light beam can pass from the illumination source through the display to the scene. While passing through the display, the light beam may undergo diffraction by the display, resulting in a characteristic illumination pattern, such as a dot pattern. In this embodiment, the display can function as a grating. The wiring of the display, in particular the wiring of the screen, may be configured to form gaps and / or slits and ridges of the grating.

[0028] Furthermore, the light source may be configured to emit modulated or unmodulated light. When multiple light sources are used, different light sources may have different modulation frequencies, which can later be used to distinguish between light beams, as outlined in more detail below.

[0029] One or more light beams generated by an illumination source may generally propagate parallel to the optical axis or at an angle to the optical axis, for example, at an angle to the optical axis. A display device may be configured such that one or more light beams propagate from the display device toward the scene along the optical axis of the display device. For this purpose, the display device may include at least one reflective element, preferably at least one prism, for deflecting the illumination light beam onto the optical axis. As an example, one or more light beams, such as a laser light beam, and the optical axis may have an angle of less than 10°, preferably less than 5°, and even less than 2°. However, other embodiments are also possible. Furthermore, one or more light beams may be on the optical axis or off the optical axis. As an example, one or more light beams may be parallel to the optical axis, or even coincide with the optical axis, at a distance of less than 10 mm, preferably less than 5 mm, and even less than 1 mm from the optical axis.

[0030] As used herein, the term “at least one illumination pattern” refers to at least one arbitrary pattern that includes at least one illumination feature adapted to illuminate at least a portion of a scene. As used herein, the term “illumination feature” refers to at least one feature that extends at least partially from the pattern. An illumination pattern may include a single illumination feature. An illumination pattern may include multiple illumination features. An illumination pattern may be selected from the group consisting of at least one dot pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; and at least one pattern including an arrangement of periodic or aperiodic features. An illumination pattern may include regular and / or constant and / or periodic patterns such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or even a convex tile pattern. An illumination pattern may show at least one illumination feature selected from the group consisting of at least one point; at least one line; at least two lines such as parallel or intersecting lines; at least one point and one line; at least one arrangement of periodic or aperiodic features; and at least one feature of any shape. The illumination pattern may include at least one pattern selected from the group consisting of: at least one point pattern, in particular a pseudo-random point pattern; a random point pattern or a quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern containing at least one known feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern; at least one pattern containing a convex, uniform tiling; at least one line pattern containing at least one line; and at least one line pattern containing at least two lines, such as parallel or intersecting lines. For example, the illumination source may be adapted to generate and / or project a point cloud. The illumination source may include at least one light projector adapted to generate a point cloud such that the illumination pattern contains multiple point patterns.The illumination source may include at least one mask adapted to generate an illumination pattern from at least one light beam produced by the illumination source.

[0031] The distance between two features of the illumination pattern and / or the area of ​​at least one illumination feature may depend on the circle of confusion in the image. As outlined above, the illumination source may include at least one light source configured to produce at least one illumination pattern. Specifically, the illumination source includes at least one laser source and / or at least one laser diode assigned to produce laser radiation. The illumination source may include at least one diffractive optical element (DOE). The display device may include at least one point projector, such as at least one laser source and DOE, adapted to project at least one periodic point pattern.

[0032] Where used further herein, the term "projects at least one illumination pattern" means providing at least one illumination pattern for illuminating at least one scene.

[0033] For example, the projected illumination pattern may be a periodic dot pattern. The projected illumination pattern may have a low dot density. For example, the illumination pattern may include at least one periodic dot pattern having a low dot density, and the illumination pattern having 2500 or fewer dots per field of view. Compared to structured light, which typically has a dot density of 10k to 30k in a 55 × 38° field of view, the illumination pattern according to the present invention may have a lower density. This allows the proposed technique to increase the power per dot so that it is less dependent on ambient light compared to structured light.

[0034] The display device may include one camera equipped with a light sensor. The display device may also include multiple cameras, each equipped with one or more light sensors.

[0035] A photosensor has at least one photosensitive area. As used herein, “photosensor” generally refers to a photosensitive device for detecting a light beam, such as for detecting an illumination and / or light spot generated by at least one light beam. As further used herein, “photosensitive area” generally refers to an area of ​​a photosensor that is illuminated externally by at least one light beam and generates at least one sensor signal in response to said illumination. Specifically, the photosensitive area may be located on the surface of each photosensor. However, other embodiments are also possible. A display device may include a plurality of photosensors, each having a photosensitive area. As used herein, the term “photosensor, each having at least one photosensitive area” refers to a configuration comprising a plurality of single photosensors, each having one photosensitive area, and a configuration comprising a single coupled photosensor having a plurality of photosensitive areas. The term “photosensor” further refers to a photosensitive device configured to generate one output signal. If the display device includes multiple light sensors, each light sensor may be embodied by providing exactly one photosensitive area that can be illuminated, for example, so that exactly one photosensitive area exists within each light sensor, and by generating exactly one uniform sensor signal for the entire light sensor in response to illumination of the photosensitive area. Thus, each light sensor may be a single-area light sensor. The use of single-area light sensors, however, makes the configuration of the display device particularly simple and efficient. Thus, as an example, commercially available light sensors, such as commercially available silicon photodiodes, each having exactly one photosensitive area, may be used in the configuration. However, other embodiments are also possible.

[0036] Preferably, the photosensitive area can be oriented substantially perpendicular to the optical axis of the display device. The optical axis may be a straight optical axis, or it may be refracted or even split, for example, by using one or more deflection elements and / or one or more beam splitters, in which case substantially perpendicular orientation refers with respect to the local optical axis of each branch or beam path of the optical configuration.

[0037] The light sensor may specifically consist of at least one photodetector, preferably an inorganic photodetector, more preferably an inorganic semiconductor photodetector, most preferably a silicon photodetector, or may include them. Specifically, the light sensor may be sensitive in the infrared spectral range. All pixels of the matrix, or at least one group of light sensors in the matrix, may specifically be identical. A group of identical pixels in the matrix may specifically be provided for different spectral ranges, or all pixels may be identical with respect to spectral sensitivity. Furthermore, pixels may be identical in size and / or with respect to their electronic or optoelectronic properties. Specifically, the light sensor may be at least an inorganic photodiode sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers, or may include them. Specifically, the light sensor may be sensitive in the near-infrared region, particularly in the range of 700 nm to 1100 nm, where silicon photodiodes are applicable. The infrared light sensor that can be used in the light sensor may be a commercially available infrared light sensor, such as the infrared light sensor sold under the brand name Hertzstueck® by trinamX GmbH at Ludwigshafen am Rhein, Germany, D-67056. Thus, as an example, the light sensor may include at least one intrinsic photovoltaic light sensor, more preferably at least one semiconductor photodiode selected from the group consisting of Ge photodiodes, InGaAs photodiodes, extended InGaAs photodiodes, InAs photodiodes, InSb photodiodes, and HgCdTe photodiodes. Additionally or alternatively, the light sensor may include at least one exogenous photovoltaic light sensor, more preferably at least one semiconductor photodiode selected from the group consisting of Ge:Au photodiodes, Ge:Hg photodiodes, Ge:Cu photodiodes, Ge:Zn photodiodes, Si:Ga photodiodes, and Si:As photodiodes.Additionally or alternatively, the optical sensor may include at least one photoconductive sensor, such as a PbS or PbSe sensor, a bolometer, preferably selected from the group consisting of a VO bolometer and an amorphous Si bolometer.

[0038] The photosensor may have sensitivity in one or more of the ultraviolet, visible, or infrared spectral ranges. Specifically, the photosensor may have sensitivity in the visible spectral range of 500 nm to 780 nm, most preferably 650 nm to 750 nm, or 690 nm to 700 nm. Specifically, the photosensor may have sensitivity in the near-infrared region. Specifically, the photosensor may have sensitivity in the near-infrared region, particularly in the range of 700 nm to 1000 nm, where silicon photodiodes are applicable. Specifically, the photosensor may have sensitivity in the infrared spectral range, specifically in the range of 780 nm to 3.0 μm. For example, the photosensor may be at least one element selected from the group consisting of photodiodes, photocells, photoconductors, phototransistors, or any combination thereof, or may include them. For example, the light sensor may be at least one element selected from the group consisting of CCD sensor elements, CMOS sensor elements, photodiodes, photocells, photoconductors, phototransistors, or any combination thereof, or may include such elements. Any other type of photosensitive element may be used. The photosensitive element can generally be made entirely or partially from inorganic materials and / or entirely or partially from organic materials. Most commonly, one or more commercially available photodiodes, such as inorganic semiconductor photodiodes, may be used.

[0039] An optical sensor may include at least one sensor element that includes a matrix of pixels. Therefore, as an example, an optical sensor may be part of or constitute a pixelated optical device. For example, an optical sensor may be at least one CCD device and / or CMOS device, and / or include them. As an example, an optical sensor may be part of or constitute a CCD device and / or CMOS device having a matrix of pixels, where each pixel forms a photosensitive area.

[0040] As used herein, the term “sensor element” generally refers to a device or combination of devices configured to sense at least one parameter. In this case, the parameter may specifically be an optical parameter, and the sensor element may specifically be an optical sensor element. The sensor element may be formed as a single, unified device or as a combination of several devices. The sensor element includes a matrix of optical sensors. The sensor element may include at least one CMOS sensor. The matrix may consist of independent pixels, such as independent optical sensors. Thus, it can constitute a matrix of inorganic photodiodes. However, alternatively, a commercially available matrix, such as one or more CCD detectors, such as a CCD detector chip, and / or CMOS detectors, such as a CMOS detector chip, may be used. Thus, generally, the sensor element may be at least one CCD device and / or CMOS device, and / or may include it, and / or the optical sensors may form a sensor array or be part of a sensor array such as the matrix described above. Therefore, as an example, the sensor element may have an array of pixels, such as a rectangular array having m rows and n columns, where m and n are independently positive integers. Preferably, multiple columns and multiple rows are given, i.e., n>1, m>1. Therefore, as an example, n may be 2 to 16 or more, and m may be 2 to 16 or more. Preferably, the ratio of the number of rows to the number of columns is close to 1. As an example, n and m may be selected such that 0.3 ≤ m / n ≤ 3 by selecting m / n = 1:1, 4:3, 16:9 or similar. As an example, the array may be a square array having an equal number of rows and columns by selecting m=2, n=2 or m=3, n=3, etc.

[0041] The matrix may consist of independent pixels, such as independent light sensors. Thus, a matrix of inorganic photodiodes can be constructed. However, alternatively, a commercially available matrix can be used, such as one or more CCD detectors, such as a CCD detector chip, and / or CMOS detectors, such as a CMOS detector chip. Therefore, generally, a light sensor may be and / or include at least one CCD and / or CMOS device, and / or the light sensor of a display device may form a sensor array or be part of a sensor array such as the matrix described above.

[0042] The matrix may be a rectangular matrix having at least one row, preferably multiple rows and multiple columns. For example, the rows and columns may be oriented substantially vertically. As used herein, the term “substantially vertical” refers to a vertical orientation with a tolerance of ±20° or less, preferably ±10° or less, and more preferably ±5° or less. Similarly, the term “substantially parallel” refers to a parallel orientation with a tolerance of, for example, ±20° or less, preferably ±10° or less, and more preferably ±5° or less. Thus, for example, tolerances of less than 20°, specifically less than 10°, or even less than 5° may be permitted. To provide a wide field of view, the matrix may have at least 10 rows, preferably at least 500 rows, and more preferably at least 1000 rows. Similarly, the matrix may have at least 10 columns, preferably at least 500 columns, and more preferably at least 1000 columns. The matrix may include at least 50 optical sensors, preferably at least 100,000 optical sensors, and more preferably at least 5,000,000 optical sensors. The matrix may contain a number of pixels ranging from several megapixels. However, other embodiments are also possible. Therefore, in configurations where axial rotational symmetry is expected, a circular or concentric arrangement of the optical sensors in the matrix, which may also be called pixels, may be preferred.

[0043] Therefore, as an example, the sensor element may be part of or constitute a pixelated optical device. For example, the sensor element may be at least one CCD device and / or CMOS device, and / or may include them. As an example, the sensor element may be part of or constitute a CCD device and / or CMOS device having a matrix of pixels, where each pixel forms a photosensitive area. The sensor element may employ a rolling shutter or a global shutter to read the matrix of the light sensor.

[0044] The display device may further include at least one transfer device. The display device may further include one or more additional elements, such as one or more additional optical elements. The display device may include a transfer device such as at least one lens and / or at least one lens system, and at least one optical element selected from the group of at least one diffractive optical elements. The term “transfer device,” also called a “transfer system,” can generally refer to one or more optical elements adapted to modify a light beam, such as by changing one or more of the beam parameters of the light beam, the width of the light beam, or the direction of the light beam. The transfer device may be adapted to guide the light beam to a light sensor. The transporter may specifically include: at least one lens, selected from the group consisting of, for example, at least one focusable lens, at least one aspherical lens, at least one spherical lens, and at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflection element, preferably at least one mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or beam splitting mirror; and at least one multi-lens system, one or more of the at least one lens selected from the group consisting of. As used herein, the term “focal length” of a transporter refers to the distance at which incident parallel light rays that may collide with the transporter are focused to a “focus,” also called the “focal point.” Thus, the focal length constitutes an indicator of the transporter’s ability to focus an incident light beam. Accordingly, the transporter may include one or more imaging elements that may have the effect of a focusing lens. For example, the transporter may have one or more lenses, in particular one or more refractive lenses, and / or one or more convex mirrors. In this example, the focal length can be defined as the distance from the center of the thin refractive lens to the primary focal point of the thin lens. For focusing thin refractive lenses, such as convex or biconvex thin lenses, the focal length can be considered positive and can give the distance at which parallel light striking the thin lens as a transfer device can be focused into a single spot.Furthermore, the transfer device may include at least one wavelength-selective element, such as at least one optical filter. Additionally, the transfer device may be designed to apply a predetermined beam profile to electromagnetic radiation, for example, at the location of the sensor region, specifically within the sensor area. Any of the above embodiments of the transfer device can, in principle, be implemented individually or in any desired combination.

[0045] The transfer device may have an optical axis. Specifically, the display device and the transfer device have a common optical axis. As used herein, the term “optical axis of the transfer device” generally refers to the specular or rotationally symmetric axis of a lens or lens system. The optical axis of the display device may be the line of symmetry of the optical configuration of the display device. The display device has at least one transfer device, preferably at least one transfer system having at least one lens. The transfer system may include, as an example, at least one beam path in which the elements of the transfer system within the beam path are arranged rotationally symmetrically with respect to the optical axis. Furthermore, as will be described in more detail below, one or more optical elements arranged within the beam path may be offset or inclined with respect to the optical axis. However, in this case, the optical axis may be defined sequentially by interconnecting the centers of the optical elements within the beam path, for example, by interconnecting the centers of the lenses, and in this context, optical sensors are not considered optical elements. The optical axis may generally indicate a beam path. There, the display device may have a single beampath along which the light beam travels from the object to the photosensor, or it may have multiple beampaths. For example, a single beampath may be given, or the beampath may be divided into two or more sub-beampaths. In the latter case, each sub-beampath may have its own optical axis. The photosensor may be placed in one and the same beampath or sub-beampath. Alternatively, however, the photosensor may also be placed in different sub-beampaths.

[0046] The transfer device may configure a coordinate system in which the longitudinal coordinates are aligned with the optical axis, and d is the spatial offset from the optical axis. The coordinate system may be a polar coordinate system in which the optical axis of the transfer device forms the z axis, and the distance from the z axis and the polar angle can be used as additional coordinates. Directions parallel or antiparallel to the z axis can be considered longitudinal directions, and coordinates aligned with the z axis can be considered longitudinal coordinates. Any direction perpendicular to the z axis can be considered transverse directions, and polar coordinates and / or polar angles can be considered transverse coordinates.

[0047] A display device can constitute a coordinate system in which the optical axis of the display device forms the z-axis, and further, x and y axes are provided that are perpendicular to the z-axis and mutually orthogonal. As an example, the display device and / or a part of the display device may be located at a specific point in this coordinate system, such as the origin of this coordinate system. In this coordinate system, the direction parallel or antiparallel to the z-axis can be considered the vertical direction, and the coordinates along the z-axis can be considered the vertical coordinates. Any direction perpendicular to the vertical direction can be considered the horizontal direction, and the x and / or y coordinates can be considered the horizontal coordinates.

[0048] Alternatively, other types of coordinate systems may be used. For example, a polar coordinate system can be used in which the optical axis forms the z-axis, and the distance from the z-axis and the polar angle can be used as additional coordinates. Similarly, the direction parallel or antiparallel to the z-axis can be considered the longitudinal direction, and coordinates along the z-axis can be considered the longitudinal coordinates. Any direction perpendicular to the z-axis can be considered the transverse direction, and polar coordinates and / or polar angles can be considered the transverse coordinates.

[0049] The optical sensor is configured to determine at least one first image, which includes multiple reflective features generated by the scene in response to illumination by an illumination feature. As used herein, the term “image” may specifically refer to data recorded by using the optical sensor, such as multiple electronic readings from an imaging device, such as pixels of the sensor element. The image itself may therefore include pixels, and the pixels of the image correlate to the pixels of the matrix of the sensor element. Thus, when referring to “pixel,” it may refer to a unit of image information generated by a single pixel of the sensor element, or to a single pixel of the sensor element directly. As used herein, the term “two-dimensional image” may generally refer to an image that has information about horizontal coordinates, such as only height and width dimensions. As used herein, the term “three-dimensional image” may generally refer to an image that has information about horizontal coordinates as well as information about vertical coordinates, such as height, width, and depth dimensions. As used herein, the term “reflective feature” may specifically refer to a feature in the image plane generated by the scene in response to illumination having at least one illumination feature.

[0050] The display device comprises at least one translucent display configured to display information. As used herein, the term “translucent” may refer to the property of the display to allow light, particularly light in a specific wavelength range, to pass through. The illumination source and photosensor are positioned in front of the display in the direction of propagation of the illumination pattern. The illumination source and photosensor may be positioned in fixed positions relative to each other. For example, the display device configuration may include a camera with a photosensor and lens system, and a laser projector. The laser projector and camera may be fixed behind the translucent display in the direction of propagation of light reflected by the scene. The laser projector may generate a dot pattern and illuminate it through the display. The camera can capture images through the display. However, positioning the illumination source and photosensor behind the translucent display in the direction of propagation of light reflected by the scene may result in the diffraction grating of the display generating multiple laser points on the scene and within the first image. As a result, these multiple spots on the first image may not contain useful distance information. As outlined in detail below, the evaluation device may be configured to find and evaluate the zero-order reflection features of the diffraction grating, i.e., the true features, while ignoring higher-order reflection features, i.e., false features.

[0051] The display device includes at least one evaluation device, which is configured to evaluate a first image. Where further used herein, the term “evaluation device” generally refers to any device adapted to perform a specified operation, preferably by using at least one data processing device, more preferably by using at least one processor and / or at least one application-specific integrated circuit. Thus, as an example, at least one evaluation device may include at least one data processing device having software code stored therein, which includes a number of computer commands. The evaluation device may provide one or more hardware elements for performing one or more of the specified operations, and / or provide one or more processors having software running thereon for performing one or more of the specified operations. The operations include evaluating an image. Specifically, determining a beam profile and displaying a surface may be performed by at least one evaluation device. Thus, as an example, one or more instructions may be implemented in software and / or hardware. Therefore, for example, the evaluation apparatus may consist of one or more computers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), or one or more programmable devices such as field-programmable gate arrays (FPGAs) configured to perform the evaluations described above. However, additionally or alternatively, the evaluation apparatus may also be fully or partially embodied in hardware.

[0052] The evaluation device and the display device may be integrated completely or partially into a single device. Therefore, generally, the evaluation device may also form part of the display device. Alternatively, the evaluation device and the display device may be embodied completely or partially as separate devices. The display device may include further components.

[0053] The evaluation device may be one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs), and / or one or more computers, preferably one or more microcomputers, and / or one or more data processing devices, such as microcontrollers, field-programmable gate arrays, or digital signal processors. Additional components may include data acquisition devices, such as one or more devices for receiving and / or pre-processing sensor signals, such as one or more pre-processing devices and / or one or more AD converters and / or one or more filters. Furthermore, the evaluation device may include one or more measuring devices, such as one or more measuring devices for measuring current and / or voltage. Furthermore, the evaluation device may include one or more data storage devices. Furthermore, the evaluation device may include one or more interfaces, such as one or more wireless interfaces and / or one or more wired interfaces.

[0054] The evaluation device may be connected to, or may include, at least one further data processing device that can be used for one or more of the following: displaying, visualizing, analyzing, distributing, communicating, or further processing information such as information obtained by the optical sensor and / or the evaluation device. The data processing device may be connected to, or may incorporate, at least one of the following: a display, projector, monitor, LCD, TFT, loudspeaker, multi-channel sound system, LED pattern, or further visualization device. It may further be connected to, or may incorporate, at least one of the following: a communication device or communication interface, connector, or port that can send encrypted or unencrypted information using one or more of the following: email, text message, telephone, Bluetooth®, Wi-Fi, infrared, or internet interface, port, or connection. It can further be connected to or incorporate at least one of the following: a system on a chip such as a processor, graphics processor, CPU, Open Multimedia Applications Platform (OMAP®), integrated circuit, or product from Apple A-series or Samsung S3C2-series; a system on a chip such as a microcontroller or microprocessor; one or more memory blocks such as ROM, RAM, EEPROM, or flash memory; a timing source such as an oscillator or phase-locked loop; a counter timer, real-time timer, or power-on-reset-generator, voltage regulator, power management circuit, or DMA controller. The individual units can further be connected to or integrated with an Internet of Things or Industry 4.0 type network by a bus such as an AMBA bus.

[0055] The evaluation device and / or data processing device may be connected by, or have, further external interfaces or ports such as serial or parallel interfaces or ports, one or more of USB, Centronics Port, FireWire®, HDMI®, Ethernet®, Bluetooth®, RFID, Wi-Fi, USART, or SPI, or analog interfaces or ports such as one or more of standardized interfaces or ports to further devices such as a 2D camera device using an ADC or DAC, or an RGB interface such as CameraLink. The evaluation device and / or data processing device may be further connected by one or more of inter-processor interfaces or ports, FPGA-to-FPGA interfaces, or serial or parallel interface ports. The evaluation device and data processing device may also be connected to one or more of optical disc drives, CD-RW drives, DVD+RW drives, flash drives, memory cards, disk drives, hard disk drives, solid state disks, or solid state hard disks.

[0056] The evaluation device and / or data processing device may be connected by or have one or more further external connectors, such as one or more of the following: phone connectors, RCA connectors, VGA connectors, hermaphrodite connectors, USB connectors, HDMI® connectors, 8P8C connectors, BCN connectors, IEC60320 C14 connectors, optical fiber connectors, D subminiature connectors, RF connectors, coaxial connectors, SCART connectors, and XLR connectors, and / or incorporate at least one suitable socket for one or more of these connectors.

[0057] The evaluation device is configured to evaluate a first image. The evaluation of the first image includes identifying the reflection features of the first image. The evaluation device may be configured to perform at least one image analysis and / or image processing to identify the reflection features. The image analysis and / or image processing may use at least one feature detection algorithm. The image analysis and / or image processing may include: filtering; selection of at least one region of interest; formation of a difference image between the image generated by the sensor signal and at least one offset; inversion of the sensor signal by inverting the image generated by the sensor signal; formation of a difference image between images generated by the sensor signal at different times; background correction; decomposition into color channels; decomposition into hue; saturation; luminance channels; frequency decomposition; singular value decomposition; application of a blob detector; application of a corner detector; application of a Hessian filter determinant; application of a principal curvature-based region detector; application of a maximum stable extreme region detector; application of a generalized Hough transform; application of a ridge detector; application of an affine-invariant feature detector; application of an affine adaptation point of interest operator; application of a Harris affine region detector; application of a Hessian affine region detector; application of a scale-invariant feature transform. This may include one or more of the following: application of scale-space extreme value detectors; application of local feature detectors; application of accelerated robust feature algorithms; application of gradient position and direction histogram algorithms; application of oriented gradient descriptor histograms; application of Deriche edge detectors; application of differential edge detectors; application of spatiotemporal point of interest detectors; application of Moravec corner detectors; application of Canny edge detectors; application of Laplace operators for Gaussian filters; application of differential Gaussian filters; application of Sobel operators; application of Laplace operators; application of Schall operators; application of Prewitt operators; application of Roberts operators; application of Kirsch operators; application of high-pass filters; application of low-pass filters; application of Fourier transforms; application of Radon transforms; application of Huff transforms; application of wavelet transforms; thresholding; and generation of binary images. The region of interest may be determined manually by the user or automatically by recognizing features in an image generated by an optical sensor.

[0058] For example, the illumination source may be configured to generate and / or project a point cloud such that multiple illumination regions are generated on a photosensor, such as a CMOS detector. Furthermore, disturbances such as speckle and / or external light and / or multiple reflections may be present on the photosensor. The evaluation device may be adapted to determine at least one region of interest, for example, one or more pixels illuminated by a light beam used to determine the longitudinal coordinates of an object. For example, the evaluation device may be adapted to perform filtering methods, such as blob analysis and / or edge filtering and / or object recognition methods.

[0059] The evaluation device may be configured to perform at least one image correction. The image correction may include at least one background subtraction. The evaluation device may be adapted to remove the influence of background light from the beam profile, for example, by imaging without further illumination.

[0060] Each reflective feature includes at least one beam profile. As used herein, the term “beam profile” of a reflective feature may generally refer to at least one intensity distribution of a reflective feature, such as a light spot on a light sensor, as a function of a pixel. The beam profile may be selected from the group consisting of trapezoidal beam profiles; triangular beam profiles; conical beam profiles; and linear combinations of Gaussian beam profiles. The evaluation apparatus is configured to determine beam profile information for each reflective feature by analyzing the beam profiles.

[0061] The evaluation device analyzes the beam profile to determine at least one longitudinal coordinate z of each reflection feature. DPRThe system is configured to determine the beam profile. As used herein, the term “beam profile analysis” may generally refer to the evaluation of a beam profile and may include at least one mathematical operation and / or at least one comparison and / or at least one symmetrization and / or at least one filtering and / or at least one normalization. For example, beam profile analysis may include at least one of the following steps: histogram analysis, calculation of difference measurements, application of a neural network, and application of a machine learning algorithm. The evaluation system may be configured to symmetrize and / or normalize and / or filter the beam profile in particular to remove noise or asymmetry from recordings such as recordings at larger angles and edges. The evaluation system may filter the beam profile by removing high spatial frequencies, such as by spatial frequency analysis and / or median filtering. Aggregation may be performed by the center of the light spot intensity, and all intensities at the same distance to the center may be averaged. The evaluation system may be configured to normalize the beam profile to the maximum intensity in particular to take into account the intensity difference due to the recorded distance. The evaluation device may be configured to remove the influence of background light from the beam profile, for example, by imaging without illumination.

[0062] The reflection feature may cover or extend over at least one image pixel. For example, the reflection feature may cover or extend across multiple pixels. The evaluation device may be configured to determine and / or select all pixels connected to and / or belonging to the reflection feature, e.g., the light spot. The evaluation device determines the center of intensity,

number

number

[0063] The evaluation device uses a depth technique from photon ratio, also known as beam profile analysis, to determine the vertical coordinate z for each reflection feature. DPR It may be configured to determine the depth from photon ratio (DPR) technique. See WO2018 / 091649A1, WO 2018 / 091638A1 and WO2018 / 091640A1, the full contents thereof are included by reference.

[0064] The evaluation device can be configured to determine the beam profile of each reflective feature. As used herein, the term “determine a beam profile” means identifying and / or selecting at least one reflective feature provided by a photosensor, and evaluating at least one intensity distribution of the reflective feature. As an example, a region of a matrix may be used and evaluated to determine an intensity distribution, such as a three-dimensional or two-dimensional intensity distribution, along an axis or line passing through the matrix. As an example, the center of illumination by the light beam may be determined by determining at least one pixel having the best illumination, and a cross-sectional axis may be selected through the center of illumination. The intensity distribution may be an intensity distribution as a function of coordinates along this cross-sectional axis passing through the center of illumination. Other evaluation algorithms are also possible.

[0065] Analysis of a single beam profile of reflection features may include determining at least one first area and at least one second area of ​​the beam profile. The first area of ​​the beam profile may be area A1, and the second area of ​​the beam profile may be area A2. The evaluation device may be configured to integrate the first and second areas. The evaluation device may be configured to derive a combined signal, in particular a quotient Q, by one or more of the following: dividing the integrated first and integrated second areas, dividing by a multiple of the integrated first and integrated second areas, or dividing by a linear combination of the integrated first and integrated second areas. The evaluation device may be configured to determine at least two areas of the beam profile and / or to divide the beam profile into at least two segments having different areas of the beam profile, where overlap of areas is possible as long as the areas do not coincide. For example, the evaluation device may be configured to determine multiple areas, such as two, three, four, five, or up to ten areas. The evaluation device may be configured to divide the optical spot into at least two areas of the beam profile, and / or to divide the beam profile into at least two segments containing different areas of the beam profile. The evaluation device may be configured to determine the integral of the beam profile across each of the at least two areas. The evaluation device may be configured to compare at least two of the determined integrals. Specifically, the evaluation device may be configured to determine at least one first area and at least one second area of ​​the beam profile. As used herein, the term “area of ​​beam profile” generally refers to any region of the beam profile at the optical sensor position used to determine the quotient Q. The first area and the second area of ​​the beam profile may be adjacent regions, overlapping regions, or both. The first area and the second area of ​​the beam profile may not be overlapping regions.For example, the evaluation device may be configured to divide the sensor area of ​​a CMOS sensor into at least two sub-regions, and the evaluation device may be configured to divide the sensor area of ​​a CMOS sensor into at least one left portion and at least one right portion, and / or at least one upper portion and at least one lower portion, and / or at least one inner portion and at least one outer portion. Additionally or alternatively, the display device may include at least two optical sensors, and the photosensitive areas of the first optical sensor and the second optical sensor may be arranged such that the first optical sensor determines a first area of ​​the beam profile of the reflective feature, and the second optical sensor determines a second area of ​​the beam profile of the reflective feature. The evaluation device may be adapted to integrate the first and second areas. The evaluation device may be configured to use at least one predetermined relationship between the quotient Q and the longitudinal coordinate to determine the longitudinal coordinate. The predetermined relationship may be one or more of empirical relationships, semi-empirical relationships, and analytically derived relationships. The evaluation device may include at least one data storage device for storing predetermined relationships, such as a lookup list or a lookup table.

[0066] A first area of ​​the beam profile may substantially include edge information of the beam profile, a second area of ​​the beam profile may substantially include central information of the beam profile, and / or, the first area of ​​the beam profile may substantially include information about the left portion of the beam profile, and the second area of ​​the beam profile may substantially include information about the right portion of the beam profile. The beam profile may have a center, i.e., the center point of the maximum value of the beam profile and / or the center point of the plateau of the beam profile and / or the geometric center of the light spot, and a falling edge extending from the center. The second area may include the inner region of the cross section, and the first area may include the outer region of the cross section. As used herein, the term “substantially central information” generally means that the proportion of edge information is low compared to the proportion of central information, i.e., the proportion of intensity distribution corresponding to the center. Preferably, the central information has a proportion of edge information of less than 10%, more preferably less than 5%, and most preferably, the central information does not include edge content. As used herein, the term “substantially edge information” generally means that the proportion of central information is low compared to the proportion of edge information. Edge information may include information from the entire beam profile, particularly from the central and edge regions. Edge information has a proportion of less than 10%, preferably less than 5%, of central information, and more preferably, edge information does not include central information. If the beam profile is near the center or around it and substantially includes central information, at least one area of ​​the beam profile may be determined and / or selected as the second area of ​​the beam profile. If the beam profile includes at least a portion of the falling edge of the cross section, at least one area of ​​the beam profile may be determined and / or selected as the first area of ​​the beam profile. For example, the entire area of ​​the cross section may be determined as the first area.

[0067] Other selections for the first area A1 and the second area A2 may also be feasible. For example, the first area may include the substantially outer region of the beam profile, and the second area may include the substantially inner region of the beam profile. For example, in the case of a two-dimensional beam profile, the beam profile can be divided into a left portion and a right portion, where the first area substantially includes the left portion of the beam profile, and the second area substantially includes the right portion of the beam profile.

[0068] Edge information may include information about the number of photons in the first area of ​​the beam profile, and center information may include information about the number of photons in the second area of ​​the beam profile. The evaluation device may be configured to determine the area integral of the beam profile. The evaluation device may be configured to determine the edge information by integration and / or addition of the first area. The evaluation device may be configured to determine the center information by integration and / or addition of the second area. For example, the beam profile may be a trapezoidal beam profile, and the evaluation device may be configured to determine the trapezoidal integral. Furthermore, if a trapezoidal beam profile is assumed, the determination of the edge signal and center signal may be replaced by an equivalent evaluation utilizing the characteristics of the trapezoidal beam profile, such as determining the inclination and position of the edges, determining the height of the central plateau, and deriving the edge signal and center signal by geometric considerations.

[0069] In one embodiment, A1 may correspond to the entire or complete area of ​​the feature point on the light sensor. A2 may be the central area of ​​the feature point on the light sensor. The central area may be a constant value. The central area may be smaller than the entire area of ​​the feature point. For example, in the case of a circular feature point, the central area may have a radius of 0.1 to 0.9, preferably 0.4 to 0.6, of the total radius of the feature point.

[0070] In one embodiment, the illumination pattern may include at least one line pattern. A1 may correspond to an area on the photosensor, particularly on the photosensitive area of ​​the photosensor, having the full line width of the line pattern. The line pattern on the photosensor may be enlarged and / or displaced compared to the line pattern of the illumination pattern so that the line width on the photosensor is amplified. In particular, in the case of a matrix of photosensors, the line width of the line pattern on the photosensor may vary from one column to another. A2 may be the central area of ​​the line pattern on the photosensor. The line width of the central area may be a constant value, and in particular may correspond to the line width of the illumination pattern. The line width of the central area may be smaller than the overall line width. For example, the central area may have a line width of 0.1 to 0.9 of the total line width, preferably 0.4 to 0.6 of the total line width. The line pattern may be segmented on the photosensor. Each column of the matrix of the photosensor may include central intensity information of the central area of ​​the line pattern and edge intensity information of the region extending further outward from the central area of ​​the line pattern to the edge region.

[0071] In one embodiment, the irradiation pattern may include at least one dot pattern. A1 may correspond to an area having the total radius of the dots in the dot pattern on the light sensor. A2 may be the central area of ​​the dots in the dot pattern on the light sensor. The central area may be a constant value. The central area may have a radius corresponding to the total radius. For example, the central area may have a radius of 0.1 to 0.9 of the total radius, preferably 0.4 to 0.6 of the total radius.

[0072] The irradiation pattern may include both at least one dot pattern and at least one line pattern. In addition to, or as an alternative to, line and dot patterns, other embodiments are also possible.

[0073] The evaluation device may be configured to derive the quotient Q by one or more of the following: dividing the first area by the second area, dividing the first area by a multiple of the second area, or dividing the first area by a linear combination of the second area.

number

[0074] Additionally or alternatively, the evaluation device may be adapted to determine either or both central information or edge information from at least one slice or cut of the light spot. This can be achieved, for example, by replacing the surface integral of quotient Q with a line integral along the slice or cut. To improve accuracy, several slices or cuts passing through the light spot may be used for averaging. In the case of an elliptical spot profile, averaging across several slices or cuts may yield improved distance information.

[0075] For example, in the case of an optical sensor having a matrix of pixels, the evaluation device is - Determine the pixel with the best sensor signal and form at least one central signal; - Evaluate the sensor signals of the matrix and form at least one sum signal; - Determining the quotient Q by combining the center signal and the sum signal; - By evaluating the quotient Q, we can determine at least one vertical coordinate z of the object, The system may be configured to evaluate the beam profile.

[0076] As used herein, “sensor signal” generally refers to a signal generated by a photosensor and / or at least one pixel of a photosensor in response to illumination. Specifically, the sensor signal may be or include at least one electrical signal, such as at least one analog electrical signal and / or at least one digital electrical signal. More specifically, the sensor signal may be or include at least one voltage signal and / or at least one current signal. More specifically, the sensor signal may include at least one photocurrent. Furthermore, the raw sensor signal may be used, or a display device, photosensor, or other element may be adapted to process or preprocess the sensor signal to generate a secondary sensor signal that can also be used as a sensor signal. The term “center signal” generally refers to at least one sensor signal that contains substantially central information of the beam profile. As used herein, the term “best sensor signal” refers to either or both of the local maximum or maximum values ​​in the region of interest. For example, the central signal may be the signal of the pixel with the best sensor signal among multiple sensor signals generated by pixels in the entire matrix or a region of interest within the matrix, the region of interest may be predetermined or determinable within the image generated by the pixels of the matrix. The central signal may originate from a single pixel or from a group of optical sensors, in the latter case, for example, the sensor signals of the group of pixels may be added, integrated, or averaged to determine the central signal. The group of pixels from which the central signal originates may be, for example, a group of adjacent pixels such as pixels located less than a predetermined distance from the actual pixel with the best sensor signal, or a group of pixels that generate sensor signals within a predetermined range from the best sensor signal. The group of pixels from which the central signal originates may be selected to be as large as possible to enable the maximum dynamic range. The evaluation device may be adapted to determine the central signal by integrating multiple sensor signals, for example, multiple pixels around the pixel with the best sensor signal.For example, the beam profile may be a trapezoidal beam profile, and the evaluation device may be adapted to determine the trapezoidal integral, in particular the integral of the trapezoidal plateau.

[0077] As described above, the central signal may generally be a single sensor signal, such as a sensor signal from a pixel at the center of a light spot; or a combination of multiple sensor signals, such as a combination of sensor signals originating from pixels at the center of a light spot; or a secondary sensor signal derived by processing sensor signals derived from one or more of the aforementioned possibilities. The determination of the central signal may be performed electronically, or entirely or partially by software, since the comparison of sensor signals can be performed fairly easily by conventional electronic equipment. Specifically, the central signal may be selected from a group consisting of: the best sensor signal; the average of a group of sensor signals within a predetermined tolerance range from the best sensor signal; the average of sensor signals from a group of pixels including the pixel with the best sensor signal and a predetermined group of adjacent pixels; the sum of sensor signals from a group of pixels including the pixel with the best sensor signal and a predetermined group of adjacent pixels; the sum of a group of sensor signals within a predetermined tolerance range from the best sensor signal; the average of a group of sensor signals exceeding a predetermined threshold; the sum of a group of sensor signals exceeding a predetermined threshold; the integral of sensor signals from a group of optical sensors including the optical sensor with the best sensor signal and a predetermined group of adjacent pixels; the integral of a group of sensor signals within a predetermined tolerance range from the best sensor signal; and the integral of a group of sensor signals exceeding a predetermined threshold.

[0078] Similarly, the term “sum signal” generally refers to a signal that substantially contains edge information of the beam profile. For example, the sum signal can be derived by adding, integrating, or averaging sensor signals from the entire matrix or a region of interest within the matrix, where the region of interest is predetermined or determinable within the image generated by the optical sensors of the matrix. When summing, integrating, or averaging sensor signals, the actual optical sensors from which the sensor signals are generated may be excluded from or included in the addition, integration, or averaging. The evaluation device may be adapted to determine the sum signal by integrating the signals from the entire matrix or a region of interest within the matrix. For example, the beam profile may be a trapezoidal beam profile, and the evaluation device may be adapted to determine the integral of the entire trapezoid. Furthermore, if a trapezoidal beam profile is assumed, the determination of edge and center signals can be replaced by an equivalent evaluation that utilizes the characteristics of the trapezoidal beam profile, such as the determination of the inclination and edge positions and the height of the central plateau, and derives the edge and center signals by geometric considerations.

[0079] Similarly, the center signal and edge signal can also be determined by using segments of the beam profile, such as a circular segment of the beam profile. For example, the beam profile can be divided into two segments by a dividing line or chord that does not pass through the center of the beam profile. Thus, one segment will substantially contain edge information and the other segment will substantially contain center information. For example, the edge signal may be further subtracted from the center signal to further reduce the amount of edge information in the center signal.

[0080] The quotient Q may be a signal generated by combining the center signal and the sum signal. Specifically, the decision may include one or more of the following: forming the quotient of the center signal and the sum signal, or vice versa; forming the quotient of a multiple of the center signal and a multiple of the sum signal, or vice versa; forming the quotient of a linear combination of the center signals and a linear combination of the sum signals, or vice versa. Additionally or alternatively, the quotient Q may include any signal or combination of signals that includes at least one information item relating to a comparison between the center signal and the sum signal.

[0081] As used herein, the term “longitudinal coordinates of an object” refers to the distance between the optical sensor and the object. The evaluation device may be configured to use at least one predetermined relationship between the quotient Q and the longitudinal coordinates to determine the longitudinal coordinates. The predetermined relationship may be one or more of empirical relationships, semi-empirical relationships, and analytically derived relationships. The evaluation device may include at least one data storage device for storing the predetermined relationships, such as a lookup list or a lookup table.

[0082] The evaluation device may be configured to run a depth algorithm from at least one photon ratio that calculates distances for all zero-order and higher-order reflection features.

[0083] The evaluation of the first image includes sorting the identified reflector features with respect to luminance. As used herein, the term “sort” may mean assigning a sequence of reflector features for further evaluation with respect to luminance, in particular starting with the reflector feature with the highest luminance, followed by reflector features whose luminance decreases. As used herein, the term “luminance” may mean the size of the reflector feature in the first image and / or the intensity of the reflector feature in the first image. Luminance may mean a defined passband, such as the visible spectral range or the infrared spectral range, or it may be wavelength-independent. Sorting by decreasing luminance may mean sorting according to decreasing luminance and / or sorting with respect to decreasing luminance. If the brightest reflector feature is preferred for DPR calculation, the vertical coordinate zDPR The robustness of the decision can be enhanced. This is mainly because the reflection feature having the zero order of the diffraction grating is always brighter than the false feature having higher orders.

[0084] The evaluation device is configured to clearly match the reflection feature and the corresponding irradiation feature by using the vertical coordinate z DPR The vertical coordinate determined using the depth technique from the photon ratio can be used to solve the so-called correspondence problem. In that way, the distance information for each reflection feature can be used to find the correspondence relationship of the known laser projector grid. As used herein, the term "matching" refers to determining and / or evaluating the corresponding irradiation feature and reflection feature. As used herein, the term "corresponding irradiation feature and reflection feature" may refer to the fact that each irradiation feature of the irradiation pattern generates a reflection feature in the scene, and the generated reflection feature is assigned to the irradiation feature that generated the reflection feature.

[0085] As used herein, the term "clearly matching" may refer to the fact that only one reflection feature is assigned to one irradiation feature and / or no other reflection feature is assigned to the same matched irradiation feature.

[0086] Irradiation features corresponding to reflection features can be determined using epipolar geometry. For an explanation of epipolar geometry, see, for example, Chapter 2 of X. Jiang and H. Bunke, “Dreidimensionales Computersehen” (Three-Dimensional Computers), Springer, Berlin-Heidelbel, 1997. In epipolar geometry, it can be assumed that the illumination image, i.e., an undistorted illumination pattern, and the first image may be images determined in a spatial direction with different spatial positions and / or fixed distances. The distance may be a relative distance, also called a baseline. The illumination image may also be shown as a reference image. The evaluation device may be adapted to determine epipolar lines in the reference image. The relative positions of the reference image and the first image may be known. For example, the relative positions of the reference image and the first image may be stored in at least one memory unit of the evaluation device. The evaluation device may be adapted to determine a straight line extending from a selected reflection feature of the first image to the real-world feature it emits. Thus, the straight line may include possible object features corresponding to the selected reflection feature. The straight line and baseline unfold the epipolar plane. Because the reference image is determined in a different relative position than the first image, the features of corresponding possible objects can be imaged on a straight line in the reference image called the epipolar line. The epipolar line can be the intersection of the epipolar plane and the reference image. Therefore, the features of the reference image that correspond to selected features of the first image are located on the epipolar line.

[0087] Depending on the distance to the object in the scene that reflects the illumination feature, the reflected feature corresponding to the illumination feature may be displaced in the first image. The reference image may include at least one displacement region in which the illumination feature corresponding to the selected reflected feature will be imaged. The displacement region may include only one illumination feature. The displacement region may include multiple illumination features. The displacement region may include an epipolar line or a portion of an epipolar line. The displacement region may include multiple epipolar lines or multiple portions of multiple epipolar lines. The displacement region may extend along an epipolar line, or perpendicular to an epipolar line, or both. The evaluation device may be adapted to determine the illumination feature along an epipolar line. The evaluation device may be adapted to determine the longitudinal coordinate z and error interval ±ε for the reflected feature from the coupled signal Q in order to determine a displacement region along an epipolar line or a displacement region perpendicular to an epipolar line corresponding to z±ε. Measurement uncertainty in distance measurements using the combined signal Q may result in non-circular displacement regions in the second image because the measurement uncertainty may vary depending on the direction. Specifically, the measurement uncertainty along one or more epipolar lines may be greater than the measurement uncertainty in directions orthogonal to those one or more epipolar lines. The displacement region may include extensions in directions orthogonal to one or more epipolar lines. The evaluation device may be adapted to match the selected reflection feature to at least one illumination feature within the displacement region. The evaluation device uses the determined longitudinal coordinate z DPRThe evaluation device may be adapted to match selected features of the first image with illumination features in the displacement region by using at least one evaluation algorithm that takes into account the following. The evaluation algorithm may be a linear scaling algorithm. The evaluation device may be adapted to determine the epipolar line closest to and / or within the displacement region. The evaluation device may be adapted to determine the epipolar line closest to the image position of the reflection feature. The range of the displacement region along the epipolar line may be larger than the range of the displacement region perpendicular to the epipolar line. The evaluation device may be adapted to determine the epipolar line before determining the corresponding illumination feature. The evaluation device may determine the displacement region around the image position of each reflection feature. The evaluation device may be adapted to assign an epipolar line to each displacement region at each image position of the reflection feature, for example, by assigning the epipolar line closest to the displacement region and / or within the displacement region and / or the epipolar line closest to the displacement region along the direction perpendicular to the epipolar line. The evaluation device may be adapted to determine the irradiation features corresponding to the reflection features by determining the irradiation features closest to the assigned displacement region, and / or the irradiation features within the assigned displacement region, and / or the irradiation features closest to the assigned displacement region along the assigned epipolar line, and / or the irradiation features within the assigned displacement region along the assigned epipolar line.

[0088] Additionally or alternatively, the evaluation apparatus may perform the following steps: - A step of determining the displacement region of each reflection feature relative to the image position; - The step of assigning epipolar lines to the displacement region of each reflection feature, such as by assigning the epipolar line closest to the displacement region, and / or the epipolar line within the displacement region, and / or the epipolar line closest to the displacement region along a direction perpendicular to the epipolar line; - A step of assigning and / or determining at least one irradiation feature to each reflection feature, such as by assigning the irradiation feature closest to the assigned displacement region and / or an irradiation feature within the assigned displacement region and / or an irradiation feature along the assigned epipolar line closest to the assigned displacement region and / or an irradiation feature within the assigned displacement region along the assigned epipolar line, It may also be configured to perform

[0089] Additionally or alternatively, the evaluation device may be adapted to determine among multiple epipolar line and / or reflection features assigned to irradiation features by, for example, comparing the distance of reflection features and / or epipolar lines in the irradiation image, and / or comparing the ε-weighted distance of irradiation features and / or error-weighted distance of epipolar lines in the irradiation image, and assigning epipolar lines and / or irradiation features with shorter distances and / or ε-weighted distances to irradiation features and / or reflection features.

[0090] As described above, the diffraction grating generates multiple reflection features, for example, one true feature and multiple false features for each illumination feature. Matching is performed starting with the brightest reflection feature and decreasing the brightness of the reflection features. No other reflection features can be assigned to the same matched illumination feature. Due to display artifacts, the generated false features are generally darker than the true features. By sorting the reflection features by brightness, brighter reflection features are preferred for correspondence matching. If the illumination feature correspondence has already been used, the false feature cannot be assigned to the used, i.e., matched illumination feature.

[0091] The evaluation device is configured to classify reflection features that match the illumination features as true features and reflection features that do not match the illumination features as false features. As used herein, the term “classify” may mean assigning reflection features to at least one category. As used herein, the term “true feature” may mean the zero-order reflection feature of a diffraction grating. As used herein, the term “false feature” may mean the reflection feature of a higher-order diffraction grating, i.e., a reflection feature of order ≥ 1. The zero-order features of a diffraction grating are always brighter than the higher-order false features.

[0092] The evaluation device eliminates false features and the vertical coordinate z DPR The system is configured to generate a depth map for true features by using [a specific method / tool]. As used herein, the term "depth" may refer to the distance between an object and a light sensor, and may be given by longitudinal coordinates. As used herein, the term "depth map" may refer to the spatial distribution of depth. The display device may be used to generate a 3D map from a scene, for example, a face.

[0093] Structured optics typically uses a camera and a projector with a fine dot grating, such as one with thousands of points. A well-known projector pattern is used to find correspondences between dot patches in the scene. Distance information is obtained by triangulation if the dot correspondences are resolved. When the camera is behind the display, the image is spatially distorted by diffraction. Therefore, finding dot patterns on a distorted image is a difficult task. Compared to structured optics, this invention proposes using the photon ratio technique from depth to evaluate the beam profile, which is not directly affected by the diffraction grating of the display. Distortion does not affect the beam profile.

[0094] The depth map can be further refined by using triangulation and / or depth from defocus and / or structured light techniques. The evaluation device uses triangulation and / or depth from defocus and / or structured light techniques to determine at least one second longitudinal coordinate z for each of the reflection features. triang It may be configured to determine the following.

[0095] The evaluation device may be adapted to determine the displacements of the illumination and reflection features. The evaluation device may be adapted to determine the displacements of the matched illumination feature and the selected reflection feature. The evaluation device, for example, at least one data processing device of the evaluation device, may be configured to determine the displacements of the illumination and reflection features by comparing the respective image positions of the illumination image and the first image. As used herein, the term "displacement" refers to the difference between the image position in the illumination image and the image position in the first image. The evaluation device may be adapted to determine the second longitudinal coordinate of the matched feature by using a predetermined relationship between the second longitudinal coordinate and the displacement. The evaluation device may be adapted to determine a predetermined relationship by using triangulation. If the position of the selected reflection feature in the first image, the position of the matched illumination feature, and / or the relative displacement of the selected reflection feature and the matched illumination feature are known, the longitudinal coordinate of the corresponding object feature may be determined by triangulation. Thus, the evaluation device may be adapted to select reflection features, for example, for each subsequent and / or column, and may be adapted to determine the corresponding distance value using triangulation for each potential position of the illumination feature. Displacement and corresponding distance values ​​may be stored in at least one memory device of the evaluation device. The evaluation device may include, as an example, at least one data processing device such as at least one processor, at least one DSP, at least one FPGA and / or at least one ASIC. Furthermore, at least one data storage device may be provided, such as providing one or more lookup tables for storing predetermined relationships to store at least one predetermined or decidable relationship between a second longitudinal coordinate z and displacement. The evaluation device may be adapted to store parameters for the intrinsic and / or extrinsic calibration of the camera and / or display device. The evaluation device may be adapted to generate parameters for the intrinsic and / or extrinsic calibration of the camera and / or display device, such as by performing a Tzei camera calibration.The evaluation device may be adapted to calculate and / or estimate parameters such as the focal length of the transfer device, the distortion coefficient of the radiating lens, the center coordinates of the distortion of the radiating lens, a scale factor to account for uncertainties arising from imperfections in the hardware timing of scanning and digitization, the rotation angle of the transformation between world coordinates and camera coordinates, the translation component of the transformation between world coordinates and camera coordinates, the aperture angle, the image sensor type, the principal point, the skew coefficient, the camera center, the camera orientation, the baseline, the rotation or translation parameters between the camera and / or the illumination source, the aperture, and the focal length.

[0096] The evaluation device uses the second vertical coordinate z triang and vertical coordinate z DPR The system may be configured to determine the combined vertical coordinates. The combined vertical coordinates are the second vertical coordinate z triang and vertical coordinate z DPR It can be the average value. The combined vertical coordinates can be used to determine the depth map.

[0097] The display device may include additional illumination sources. These additional illumination sources may include at least one light-emitting diode (LED). The additional illumination sources may be configured to generate light within the visual spectral range. The light sensor may be configured to determine at least one second image, including at least one two-dimensional image of the scene. The additional illumination sources may be configured to provide additional illumination for imaging the second image. For example, the configuration of the display device can be extended by additional flood illumination LEDs. The additional illumination sources may illuminate a scene, such as a face, using LEDs, in particular, without an illumination pattern, and the light sensor may be configured to capture a two-dimensional image. The two-dimensional image can be used in face detection and verification algorithms. If the impulse response of the display is known, the distorted image captured by the light sensor can be restored. The evaluation device may be configured to determine at least one corrected image I0 by deconvolving the second image I (I=I0*g) with a diffraction grating function g. The diffraction grating function is also called the impulse response. Distortion-free images can be restored by deconvolution approaches, such as Van-Cittert or Wiener deconvolution. The display device is

number

[0098] The evaluation apparatus may be configured to determine at least one material property m of an object by evaluating the beam profiles of at least one reflective feature, preferably multiple reflective features. For details of determining at least one material property by evaluating beam profiles, see WO2020 / 187719, the contents of which are included by reference.

[0099] As used herein, the term “material property” refers to at least one arbitrary property of a material configured for characterizing and / or identifying and / or classifying the material. For example, a material property may be a property selected from the group consisting of roughness, depth of light transmission into the material, properties characterizing the material as a biological or non-biological material, reflectance, specular reflectance, diffuse reflectance, surface properties, measure of light transmission, scattering, specifically backscattering behavior, etc. At least one material property may be a property selected from the group consisting of scattering coefficient, light transmission, transparency, deviation from Lambertian surface reflectance, speckle, etc. As used herein, the term “identify at least one material property” refers to one or more of determining a material property and assigning a material property to an object. The evaluation device may include at least one database containing lists and / or tables, such as lookup lists and / or lookup tables of predefined and / or predetermined material properties. The lists and / or tables of material properties can be determined and / or generated by performing at least one test measurement using the display device according to the present invention, for example by performing a material test using a sample having known material properties. Lists and / or tables of material properties can be determined and / or generated at the manufacturer's site and / or by the user of the display device. Material properties may be further assigned to one or more material classifications, such as material name, material group (biological or non-biological), translucent or opaque, metallic or non-metallic, skin or non-skin, fur or non-fur, carpet or non-carpet, reflective or non-reflective, specular or non-specular, foam or non-foam, hair or non-hair, roughness group, etc. The evaluation device may include at least one database containing lists and / or tables of material properties and associated material names and / or material groups.

[0100] For example, although we do not wish to be bound by this theory, human skin may have a reflection profile, also called an inverse scattering profile, which includes a portion produced by back reflection from the surface, called surface reflection, and a portion produced by very diffuse reflection from light passing through the skin, called the diffuse portion of back reflection. For more information on the reflection profile of human skin, see "Lasertechnikinder Medizin:Grundlagen, Systeme, Anwendungen," "Wirkungvon Laserstrahlung auf Gewebe," 1991, pp. 10171-266, Juergen Eichler, Theo Seiler, Springer Verlag, ISBN 0939-0979. Surface reflection of skin may increase as wavelength increases toward the near-infrared. Furthermore, the transmission depth may increase as wavelength increases from visible light toward the near-infrared. The diffuse portion of back reflection may increase as the transmission depth of light increases. These properties can be used to distinguish skin from other materials by analyzing the inverse scattering profile.

[0101] Specifically, the evaluation device may be configured to compare the beam profile of a reflection feature (also called a reflection beam profile) with at least one predetermined and / or pre-recorded and / or pre-defined beam profile. The predetermined and / or pre-recorded and / or pre-defined beam profiles may be stored in a table or lookup table, may be determined empirically, for example, and may be stored, as an example, in at least one data storage device of the display device. For example, the predetermined and / or pre-recorded and / or pre-defined beam profiles may be determined at the initial startup of a mobile device equipped with a display device. For example, the predetermined and / or pre-recorded and / or pre-defined beam profiles may be stored in at least one data storage device of the mobile device, for example, by software, specifically by an app downloaded from an app store, etc. A reflection feature may be identified as being generated by biological tissue if the reflection beam profile and the predetermined and / or pre-recorded and / or pre-defined beam profile are identical. The comparison may include superimposing the reflection beam profile and the predetermined or pre-defined beam profile so that their intensity centers match. The comparison may include determining the deviation between the reflected beam profile and a predetermined and / or predetermined recorded and / or predetermined beam profile, for example, the sum of squared point distances. The evaluation device may be configured to compare the determined deviation to at least one threshold, and if the determined deviation is lower than and / or equal to the threshold, the surface is indicated as biological tissue and / or the detection of biological tissue is confirmed. The threshold may be stored in a table or lookup table, may be determined empirically, for example, and may be stored in at least one data storage device of the display device.

[0102] Additionally or alternatively, to determine whether reflective features are generated by biological tissue, the evaluation device may be configured to apply at least one image filter to the image of that area. Where used further herein, the term “image” refers to a two-dimensional function f(x,y), where luminance and / or color values ​​are given for any x,y position in the image. The position may be discretized in correspondence with the recording pixels. The luminance and / or color may also be discretized in correspondence with the bit depth of the light sensor. Where used herein, the term “image filter” refers to at least one mathematical operation applied to the beam profile and / or at least one particular region of the beam profile. Specifically, the image filter Ф maps the image f or the region of interest in the image to a real number Ф(f(x,y)) = φ, where φ represents a feature, in particular a material feature. The image may be affected by noise, and so may the features. Therefore, the features may be random variables. The features may follow a normal distribution. If the features do not follow a normal distribution, they may be transformed to conform to a normal distribution using methods such as the Box-Cox Transformation.

[0103] The evaluation device applies at least one material-dependent image filter Φ2 to the image to identify at least one material feature φ 2m It can be configured to determine the material-dependent output. As used herein, the term “material-dependent” image filter refers to an image having a material-dependent output. The output of a material-dependent image filter is referred to herein as “material feature φ 2m " or "Material-dependent characteristics φ 2m This is indicated as follows. The material feature may be, or may include, at least one piece of information relating to at least one material property of the surface of the area where the reflection feature was generated.

[0104] The material-dependent image filter is at least one filter selected from the group consisting of: luminance filters; spot shape filters; square norm gradient; standard deviation; smoothing filters such as Gaussian or median filters; gray level generation-based contrast filters; gray level generation-based energy filters; gray level generation-based uniformity filters; gray level generation-based dissimilarity filters; Rho's energy filters; threshold region filters; or a linear combination thereof; or a further material-dependent image filter Ф that correlates with one or more of the luminance filters, spot shape filters, square norm gradient, standard deviation, smoothing filters, gray level generation-based energy filters, gray level generation-based uniformity filters, gray level generation-based dissimilarity filters, Rho's energy filters, or threshold region filters. 2other , or, |ρ Ф2other,Фm A linear combination of these by |≧0.40, where Ф m This may be one or more of the following: luminance filters, spot shape filters, square norm gradient filters, standard deviation filters, smoothness filters, gray level generation-based energy filters, gray level generation-based uniformity filters, gray level generation-based dissimilarity filters, low energy filters, or threshold region filters, or a linear combination thereof. Further material-dependent image filters Ф 2other This is a material-dependent image filter Ф m One or more of the following, |ρ Ф2other,Фm |≧0.60, preferably |ρ Ф2other,Фm The correlation may be greater than or equal to |≧0.80.

[0105] A material-dependent image filter can be at least one arbitrary filter Φ that passes hypothesis testing. As used herein, the term “passes hypothesis testing” means that the null hypothesis H0 is rejected and the alternative hypothesis H1 is accepted. Hypothesis testing may include verifying the material dependence of the image filter by applying the image filter to a predefined dataset. The dataset may include multiple beam profile images. As used herein, the term “beam profile image” means NB This refers to the sum of the Gaussian radial basis functions.

number

[0106]

number

[0107]

number

number

[0108] [Table 1]

[0109]

number

number

[0110]

number

number

number

[0111] Next, each image

number

[0112]

number

number

number

number

number

number

number

[0113]

number

number

number

number

number

number

number

number

[0114] Hypothesis testing may include determining the mean sum of squares between the following:

[0115]

number

[0116] Hypothesis testing may include conducting F-tests.

[0117]

number

[0118] Here,

number

number

number

[0119] Below, we will discuss image filters assuming that the reflected image contains at least one reflected feature, in particular a spot image. Spot image

number

number

[0120] For example, a material-dependent image filter may be a luminance filter. A luminance filter can return the luminance measurement of a spot as a material feature. The material feature is,

number

number

number

number

[0121] For example, a material-dependent image filter may be a filter that has an output dependent on the spot shape. This material-dependent image filter can return a value correlated with the light transmittance of the material as a material feature. The light transmittance of the material affects the spot shape. The material feature is,

number

number

number

number

number

[0122] For example, the material-dependent image filter may be a square-norm gradient. This material-dependent image filter may return values ​​correlated with the soft-to-hard transition and / or roughness measurements of the spot as material features. The material features are,

number

number

number

[0123] For example, the material-dependent image filter may be a smoothing filter such as a Gaussian filter or a median filter. In one embodiment of the smoothing filter, this image filter can refer to the observation that volume scattering exhibits less speckle contrast compared to diffuse scattering materials. This image filter can quantify the spot smoothness corresponding to the speckle contrast as a material feature. The material feature is,

number

number

number

number

[0124]

number

number

number

number

number

[0125] For example, the image filter may be a gray level generation-based contrast filter. This material filter is a gray level generation matrix.

number

number

[0126] The material characteristics of a gray level generation-based contrast filter are:

number

[0127] For example, the image filter may be a gray level generation-based energy filter. This material filter is based on the gray level generation matrix defined above.

[0128] The material characteristics of the gray level generation-based energy filter are:

number

[0129] For example, the image filter may be a gray level generation-based uniformity filter. This material filter is based on the gray level generation matrix defined above. The material characteristics of the gray level generation-based uniformity filter are:

number

[0130] For example, the image filter may be a gray level generation-based dissimilarity filter. This material filter is based on the gray level generation matrix defined above. The material characteristics of the gray level generation-based dissimilarity filter are:

number

[0131] For example, the image filter may be a low-energy filter. This material filter has low vectors L5=[1,4,6,4,1] and E5=[-1,-2,0,-2,-1] and material L5(E5) T and E5 (L5) T It is based on image f. k These are the matrices:

number

[0132] Here, the material characteristics of the Rho energy filter are:

number

[0133] For example, a material-dependent image filter may be a threshold region filter. This material feature may relate to two areas in the image plane. The first area Ω1 may be an area where the function f is greater than α times the maximum value of f. The second area Ω2 may be an area where the function f is less than α times the maximum value of f, but greater than a threshold of ε times the maximum value of f. Preferably, α may be 0.5 and ε may be 0.05. Due to speckle or noise, the areas may not simply correspond to the inner and outer circles of the spot center. For example, Ω1 may include speckle or unconnected areas of the outer circle. The material feature is,

number

[0134] The evaluation device determines the material properties of the surface from which the reflection features were generated by the material feature φ 2m The system may be configured to use at least one predetermined relationship between the material properties of the surface that generated the reflection features. The predetermined relationship may be one or more of empirical relationships, semi-empirical relationships, and analytically derived relationships. The evaluation device may include at least one data storage device for storing the predetermined relationship, such as a lookup list or a lookup table.

[0135] The evaluation device is configured to identify reflective features as generated by irradiating biological tissue if the corresponding material properties satisfy at least one predetermined or predefined criterion. Reflective features can be identified as generated by biological tissue if the material properties indicate "biological tissue". Reflective features may be identified as generated by biological tissue if the material properties are below or equal to at least one threshold or range, and if the determined deviation is below and / or equal to the threshold, the reflective features are identified as generated by biological tissue and / or the detection of biological tissue is confirmed. At least one threshold and / or range may be stored in a table or lookup table, at least one threshold and / or range may be determined empirically, for example, and at least one threshold and / or range may be stored in at least one data storage device of the display device, as an example. Otherwise, the evaluation device is configured to identify reflective features as background. Thus, the evaluation device may be configured to assign depth information and a material property, such as "yes" or "no" for skin, to each projection spot.

[0136] Material properties are related to the vertical coordinate z, which is φ 2m After determining the vertical coordinate z, φ can be taken into consideration in the evaluation. 2m This can be determined by evaluating the following:

[0137] In a further embodiment, the present invention discloses a method for depth measurement through a display device, wherein a display device according to the present invention is used. The method is as follows: a) A step of projecting at least one illumination pattern, comprising a plurality of illumination features, onto at least one scene by using at least one illumination source, wherein the illumination source is positioned in front of the display in the direction of propagation of the illumination pattern; b) A step of determining at least one first image including a plurality of reflection features generated by the scene in response to illumination by the illumination feature, by using at least one photosensitive sensor, wherein the photosensitive sensor has at least one photosensitive area, the photosensitive sensor is positioned in front of the display in the direction of propagation of the illumination pattern, and each of the reflection features includes at least one beam profile; c) A step of evaluating the first image by using at least one evaluation device, wherein the evaluation is a substep of the following: C1) A substep of identifying the reflection features of the first image and sorting the identified reflection features with respect to brightness; C2) Analysis of the beam profile reveals that for each of the reflection features, at least one longitudinal coordinate z DPR Substeps to determine; C3) The aforementioned vertical coordinate z DPR A substep of clearly matching reflective features to corresponding illuminated features by using, wherein the matching is performed by starting with the brightest reflective feature and decreasing the brightness of the reflective features; C4) A substep in which reflection features that match the illumination features are classified as true features, and reflection features that do not match the illumination features are classified as false features; C5) Eliminate false features and the vertical coordinate z DPR The process includes a substep of generating a depth map of the true features by using, Includes steps and

[0138] The method steps may be performed in a predetermined order or in a different order. Furthermore, there may be one or more additional method steps that are not listed. Furthermore, one, more than one, or even all of the method steps may be repeated. For details, options, and definitions, refer to the display devices described above. Thus, specifically, as described above, the method may include the use of a display device according to the present invention, such as by one or more embodiments given above or given in more detail below.

[0139] At least one evaluation device may be configured to run at least one computer program, such as at least one computer program configured to perform or support one, more, or even all of the method steps of the method according to the present invention. As an example, one or more algorithms for determining the position of an object may be implemented.

[0140] In further embodiments of the present invention, the use of the display device according to the present invention, such as one or more embodiments given above or provided in more detail below, is proposed for use purposes selected from the group consisting of: position measurement in traffic technology; entertainment applications; security applications; surveillance applications; safety applications; human-machine interface applications; tracking applications; photographic applications; imaging or camera applications; mapping applications for generating at least one spatial map; homing or tracking beacon detectors for vehicles; outdoor applications; mobile applications; communication applications; machine vision applications; robotic applications; quality control applications; and manufacturing applications.

[0141] For further information regarding the display apparatus of the present invention and its use, see WO2018 / 091649A1, WO2018 / 091638A1, and WO2018 / 091640A1, the contents of which are included by reference.

[0142] Overall, the following embodiments are considered preferable in the context of the present invention.

[0143] Embodiment 1: - At least one illumination source configured to project at least one illumination pattern containing multiple illumination features onto at least one scene; - At least one photosensitive sensor having at least one photosensitive area, wherein the photosensitive sensor is configured to determine at least one first image including a plurality of reflection features generated by the scene in response to illumination by the illumination feature; - A translucent display configured to display information, wherein the illumination source and the light sensor are positioned in front of the display in the direction of propagation of the illumination pattern; - At least one evaluation device, the evaluation device is configured to evaluate the first image, the evaluation of the first image includes identifying the reflection features of the first image, sorting the identified reflection features with respect to brightness, each of the reflection features includes at least one beam profile, and the evaluation device determines for each of the reflection features at least one vertical coordinate z by analysis of those beam profiles DPR The evaluation device is configured to determine the vertical coordinate z DPR By using, the system is configured to clearly match the reflection features to the corresponding illumination features, and the matching is performed by decreasing the brightness of the reflection features, starting with the brightest reflection feature, and the evaluation device is configured to classify reflection features that match the illumination features as true features and reflection features that do not match the illumination features as false features, and the evaluation device eliminates the false features, and the vertical coordinate z DPR An evaluation device configured to generate a depth map for the true features using, A display device equipped with the following features.

[0144] Embodiment 2: The evaluation device uses triangulation and / or defocus depth and / or structured light techniques to determine at least one second longitudinal coordinate z for each of the reflection features. triang A display device according to a prior embodiment, configured to determine [something].

[0145] Embodiment 3: The evaluation device, the second vertical coordinate z triang and the aforementioned vertical coordinate z DPR It is configured to determine the combined vertical coordinates of the second vertical coordinate z triang and the aforementioned vertical coordinate z DPR A display device according to a prior embodiment, wherein the average value of the combined vertical coordinates is used for determining the depth map.

[0146] Embodiment 4: A display device according to any one of the prior embodiments, wherein the irradiation source comprises at least one laser projector, and the laser projector includes at least one laser source and at least one diffractive optical element (DOE).

[0147] Embodiment 5: A display device according to any one of the preceding embodiments, wherein the illumination source is configured to generate at least one light beam having a beam path that passes from the illumination source through the display to the scene, and the display is configured to function as a grating such that the light beam is diffracted by the display, resulting in a point pattern.

[0148] Embodiment 6: A display device according to a prior embodiment, wherein the wiring of the display is configured to form gaps and / or slits and ridges of the grid.

[0149] Embodiment 7: A display device according to any one of the preceding embodiments, wherein the irradiation pattern includes a periodic dot pattern.

[0150] Embodiment 8: A display device according to any one of the preceding embodiments, wherein the irradiation pattern has a low dot density and the irradiation pattern has 2500 or fewer dots per field of view.

[0151] Embodiment 9: A display device according to any one of the preceding embodiments, wherein the evaluation device is configured to determine information on the beam profile for each of the reflection characteristics by using a depth technique from the photon ratio.

[0152] Embodiment 10: A display device according to any one of the preceding embodiments, wherein the optical sensor includes at least one CMOS sensor.

[0153] Embodiment 11: A display device according to any one of the preceding embodiments, wherein the display device further includes an additional irradiation source, and the additional irradiation source includes at least one light emitting diode (LED).

[0154] Embodiment 12: A display device according to the preceding embodiment, wherein the additional irradiation source is configured to generate light within the visual spectral range.

[0155] Embodiment 13: A display device according to any one of the preceding two embodiments, wherein the optical sensor is configured to determine at least one second image including at least one two-dimensional image of the scene, and the additional irradiation source is configured to provide additional irradiation for imaging of the second image.

[0156] Embodiment 14: A display device according to the preceding embodiment, wherein the evaluation device is configured to determine at least one corrected image I0 by deconvolving the second image I with a diffraction grating function g, where I = I0 * g.

[0157] Embodiment 15: A method for depth measurement through a translucent display, wherein at least one display device according to any one of the preceding embodiments is used, comprising the following steps: a) projecting at least one irradiation pattern including a plurality of irradiation features onto at least one scene by using at least one irradiation source, wherein the irradiation source is arranged in front of the display in the propagation direction of the irradiation pattern; b) determining at least one first image including a plurality of reflection features generated by the scene in response to irradiation by the irradiation features by using at least one optical sensor, wherein the optical sensor has at least one photosensitive area, the optical sensor is arranged in front of the display in the propagation direction of the irradiation pattern, and each of the reflection features includes at least one beam profile; c) evaluating the first image by using at least one evaluation device, wherein the evaluation comprises the following sub-steps: C1) identifying the reflection features of the first image and sorting the identified reflection features with respect to luminance; C2) determining at least one vertical coordinate z DPR for each of the reflection features by analyzing the beam profile; C3) clearly matching the reflection features to the corresponding irradiation features by using the vertical coordinate z DPR , wherein the matching is performed starting from the brightest reflection feature and reducing the luminance of the reflection features; C4) classifying the reflection features that match the irradiation features as true features and the reflection features that do not match the irradiation features as false features; C5) excluding the false features and generating a depth map of the true features by using the vertical coordinate z DPR , and including steps; A method for measuring depth via a display device, including a method for measuring depth via a display device.

[0158] Embodiment 16: Use of a display device according to any one of the prior embodiments relating to a display device, wherein the purpose of use is selected from the group consisting of: position measurement in traffic technology; entertainment use; security use; surveillance use; safety use; human-machine interface use; tracking use; photographic use; imaging or camera use; mapping use for generating a map of at least one space; homing or tracking beacon detector for vehicles; outdoor use; mobile use; communication use; machine vision use; robotic use; quality control use; manufacturing use. [Brief explanation of the drawing]

[0159] Further optional details and features of the present invention will become apparent from the description of preferred exemplary embodiments that follows in connection with the dependent claims. In this context, certain features may be implemented individually or in combination with other features. The present invention is not limited to exemplary embodiments. Exemplary embodiments are schematically shown in the figures. The same reference numerals in the individual figures refer to the same element or an element having the same function, or an element corresponding to one another with respect to that function.

[0160] Specifically, in the following diagram: [Figure 1] Figures 1A and 1B show embodiments of a display device according to the present invention. [Figure 2] Figures 2A to 2B show embodiments of the first image determined by at least one optical sensor of the display device. [Figure 3] Figures 3A to 3C show further embodiments of the first image determined by at least one optical sensor of the display device. [Figure 4] This figure shows the determination of a corrected 2D image using a display device. [Figure 5]Figures 5A to 5C show a distorted 2D image captured using a display, a 2D image captured without a display, and a corrected 2D image. Modes for carrying out the invention

[0161] Detailed description of the embodiment Figure 1A shows a very schematic representation of one embodiment of the display device 110 according to the present invention. The display device 110 has at least one translucent display 112 configured to display information. The display 112 may be a device of any shape configured to display information items such as at least one image, at least one figure, at least one histogram, at least one piece of text, or at least one symbol. The display 112 may be at least one monitor or at least one screen. The display 112 may have any shape, preferably rectangular. For example, the display device 110 may be at least one device selected from the group consisting of a television, a smartphone, a game console, a personal computer, a laptop, a tablet, at least one virtual reality device, or a combination thereof.

[0162] The display device 110 includes at least one illumination source 114 configured to project at least one illumination pattern containing multiple illumination features onto at least one scene. A scene may refer to an object such as a face or a spatial area. A scene may include at least one object and its surrounding environment.

[0163] The illumination source 114 may be adapted to illuminate the scene directly or indirectly, and the illumination pattern may be reflected or scattered by the surface of the scene, thereby directed at least partially towards the light sensor. The illumination source 114 may be adapted to illuminate the scene, for example, by directing a light beam towards the scene, and the scene reflects the light beam. The illumination source 114 may be configured to generate an illumination light beam for illuminating the scene.

[0164] The illumination source 114 may include at least one light source. The illumination source 114 may include multiple light sources. The illumination source 114 may include an artificial illumination source, in particular at least one laser source, and / or at least one incandescent lamp, and / or at least one semiconductor light source, such as at least one light-emitting diode, in particular organic and / or inorganic light-emitting diodes. As an example, the light emitted by the illumination source may have wavelengths of 300 to 1100 nm, in particular 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range, such as in the range of 780 nm to 3.0 μm, may be used. Specifically, light in the near-infrared region, particularly in the range of 700 nm to 1100 nm, to which silicon photodiodes are applicable, can be used. The illumination source 114 may be configured to generate at least one illumination pattern in the infrared region. Using light in the near-infrared region allows the light to be undetectable or only slightly detected by the human eye, but detectable by silicon sensors, in particular standard silicon sensors. The irradiation source 114 can be configured to emit light at a single wavelength. Specifically, the wavelength may be in the near-infrared region. In other embodiments, the irradiation may be adapted to emit light having multiple wavelengths, enabling additional measurements in other wavelength channels.

[0165] The illumination source 114 may be or include at least one multiple-beam light source. For example, the illumination source 114 may include at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the illumination source 114 may comprise at least one laser and / or laser source. Various types of lasers may be employed, such as semiconductor lasers, double heterostructure lasers, external cavity lasers, separated-containment heterostructure lasers, quantum cascade lasers, dispersed Bragg reflector lasers, polariton lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, indium arsenide lasers, transistor lasers, diode-pumped lasers, dispersed feedback lasers, quantum well lasers, interband cascade lasers, gallium arsenide lasers, semiconductor ring lasers, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alternatively, non-laser light sources such as LEDs and / or light bulbs may be used. The illumination source may include one or more diffractive optical elements (DOEs) adapted to generate an illumination pattern. For example, the illumination source 114 may be adapted to generate and / or project a point cloud, and for example, the illumination source may include one or more of the following: at least one digital photoprocessing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one array of light-emitting diodes; at least one array of laser light sources. Given their generally defined beam profiles and other characteristics of handling, the use of at least one laser source as the illumination source 114 is particularly preferred. The illumination source 114 may be integrated into the housing 116 of the display device 110.

[0166] Furthermore, the irradiation source 114 may be configured to emit modulated or unmodulated light. When multiple irradiation sources 114 are used, different irradiation sources may have different modulation frequencies, which can later be used to distinguish between light beams, as outlined in more detail below.

[0167] The illumination pattern may be at least one arbitrary pattern comprising at least one illumination feature adapted to illuminate at least a portion of the scene. The illumination pattern may comprise a single illumination feature. The illumination pattern may comprise multiple illumination features. The illumination pattern may be selected from the group consisting of at least one dot pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; and at least one pattern comprising a periodic or aperiodic arrangement of features. The illumination pattern may comprise a regular and / or constant and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or even a convex tile pattern. The illumination pattern may show at least one illumination feature selected from the group consisting of at least one point; at least one line; at least two lines, such as parallel or intersecting lines; at least one point and one line; at least one arrangement of periodic or aperiodic features; and at least one feature of any shape. The illumination pattern may include at least one pattern selected from the group consisting of: at least one point pattern, in particular a pseudo-random point pattern; a random point pattern or quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern containing at least one known feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern; at least one pattern containing a convex, uniform tiling; at least one line pattern containing at least one line; and at least one line pattern containing at least two lines, such as parallel or intersecting lines. For example, the illumination source may be adapted to generate and / or project a point cloud. The illumination source 114 may include at least one optical projector adapted to generate a point cloud such that the illumination pattern includes multiple point patterns. The illumination source 114 may include at least one mask adapted to generate an illumination pattern from at least one optical beam generated by the illumination source 114.

[0168] The distance between two features of the illumination pattern and / or the area of ​​at least one illumination feature may depend on the circle of confusion in the image. As outlined above, the illumination source may include at least one light source configured to produce at least one illumination pattern. Specifically, the illumination source 114 includes at least one laser source and / or at least one laser diode designated to produce laser radiation. The illumination source 114 may include at least one diffractive optical element (DOE). The display device 110 may include at least one dot projector, such as at least one laser source and DOE, adapted to project at least one periodic dot pattern. For example, the projected illumination pattern may be a periodic dot pattern. The projected illumination pattern may have a low dot density. For example, the illumination pattern may include at least one periodic dot pattern having a low dot density, and the illumination pattern may have no more than 2500 dots per field of view. Compared to structured light, which typically has a point density of 10k to 30k in a 55 × 38° field of view, the illumination pattern according to the present invention can have a lower density. This allows the proposed technique to increase the power per point so that it is less dependent on ambient light compared to structured light.

[0169] The display device 110 has at least one photosensitive sensor 118 having at least one photosensitive area 120. The photosensitive sensor 118 is configured to determine at least one first image 122, shown, for example in Figures 2A-2C and 3A-3C, which includes a plurality of reflection features generated by the scene in response to illumination by the illumination features. The display device 110 may include a single camera having a photosensitive sensor 118. The display device 110 may include a plurality of cameras, each containing a photosensitive sensor 118, or a plurality of photosensitive sensors 118.

[0170] The light sensor 118 may specifically consist of at least one photodetector, preferably an inorganic photodetector, more preferably an inorganic semiconductor photodetector, most preferably a silicon photodetector, or may include them. Specifically, the light sensor 118 may be sensitive in the infrared spectral range. All pixels of the matrix, or at least one group of light sensors in the matrix, may specifically be identical. A group of identical pixels in the matrix may specifically be provided for different spectral ranges, or all pixels may be identical with respect to spectral sensitivity. Furthermore, pixels may be identical in size and / or with respect to their electronic or optoelectronic properties. Specifically, the light sensor 118 may be at least an inorganic photodiode sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers, or may include them. Specifically, the light sensor 118 may be sensitive in the near-infrared region, particularly in the range of 700 nm to 1100 nm, where silicon photodiodes are applicable. The infrared light sensor that can be used in the light sensor may be a commercially available infrared light sensor, such as the infrared light sensor sold under the brand name Hertzstueck® by trinamX GmbH at Ludwigshafen am Rhein, Germany, D-67056. Therefore, as an example, the light sensor 118 may include at least one intrinsic photovoltaic light sensor, more preferably at least one semiconductor photodiode selected from the group consisting of Ge photodiode, InGaAs photodiode, extended InGaAs photodiode, InAs photodiode, InSb photodiode, and HgCdTe photodiode. Additionally or alternatively, the light sensor 118 may include at least one exogenous photovoltaic light sensor, more preferably at least one semiconductor photodiode selected from the group consisting of Ge:Au photodiode, Ge:Hg photodiode, Ge:Cu photodiode, Ge:Zn photodiode, Si:Ga photodiode, and Si:As photodiode.Additionally or alternatively, the optical sensor 118 may include at least one photoconductive sensor, such as a PbS or PbSe sensor, a bolometer, preferably selected from the group consisting of a VO bolometer and an amorphous Si bolometer.

[0171] The photosensor 118 may have sensitivity in one or more of the ultraviolet, visible, or infrared spectral ranges. Specifically, the photosensor may have sensitivity in the visible spectral range of 500 nm to 780 nm, most preferably 650 nm to 750 nm, or 690 nm to 700 nm. Specifically, the photosensor 118 may have sensitivity in the near-infrared region. Specifically, the photosensor 118 may have sensitivity in the near-infrared region, particularly in the range of 700 nm to 1000 nm, where silicon photodiodes are applicable. Specifically, the photosensor may have sensitivity in the infrared spectral range, specifically in the range of 780 nm to 3.0 μm. For example, the photosensor may be at least one element selected from the group consisting of photodiodes, photocells, photoconductors, phototransistors, or any combination thereof, or may include them. For example, the light sensor 118 may be at least one element selected from the group consisting of a CCD sensor element, a CMOS sensor element, a photodiode, a photocell, a photoconductor, a phototransistor, or any combination thereof, or may include such an element. Any other type of photosensitive element may be used. The photosensitive element can generally be made entirely or partially from inorganic materials and / or entirely or partially from organic materials. Most commonly, one or more commercially available photodiodes, such as inorganic semiconductor photodiodes, may be used.

[0172] The optical sensor 118 may have at least one sensor element having a matrix of pixels. Thus, as an example, the optical sensor 118 may be part of or may constitute a pixelated optical device. For example, the optical sensor 118 may be or may include at least one CCD device and / or CMOS device. As an example, the optical sensor 118 may have a matrix of pixels and each pixel may be part of or may constitute at least one CCD device and / or CMOS device that forms a photosensitive area. The sensor element may be formed as an integrated single device or may be formed as a combination of several devices. The matrix may specifically be a rectangular matrix having one or more rows and one or more columns or may include the same. The rows and columns may specifically be arranged in a rectangular pattern. However, other arrangements such as non-rectangular arrangements are also possible. As an example, a circular arrangement is also possible where the elements are arranged in concentric circles or ellipses around a central point. For example, the matrix may be a single row of pixels. Other arrangements are also possible.

[0173] The pixels of the matrix may specifically be equal in one or more of size, sensitivity, and other optical, electrical, and mechanical characteristics. The photosensitive areas 120 of all the optical sensors 118 of the matrix may specifically be arranged in a common plane, which preferably faces the scene such that a light beam propagating from an object to the display device 110 generates a light spot on the common plane. The photosensitive areas 120 may specifically be located on the surface of each optical sensor 118. However, other embodiments are also realizable. The optical sensor 118 can include, for example, at least one CCD and / or CMOS device. As an example, the optical sensor 118 may be part of or may constitute a pixelated optical device. As an example, the optical sensor 118 may have a matrix of pixels and each pixel may be part of or may constitute at least one CCD and / or CMOS device that forms a photosensitive area 120.

[0174] The display device 110 comprises at least one translucent display 112 configured to display information. An illumination source 114 and a light sensor 118 are positioned in front of the display 112 in the direction of propagation of the illumination pattern. The illumination source 114 and the light sensor 118 may be positioned in fixed positions relative to each other. For example, the display device 110 may be configured to include a camera with a light sensor 118 and a lens system, and a laser projector as the illumination source 114. The laser projector and camera may be fixed behind the translucent display in the direction of propagation of light reflected by the scene. The laser projector may generate a dot pattern and illuminate it through the display 112. The camera can capture images through the display. However, the configuration of the illumination source 114 and light sensor 118 positioned behind the translucent display in the direction of propagation of light reflected by the scene may result in the diffraction grating of the display 112 generating multiple laser points in the scene and the first image. As a result, these multiple spots on the first image may not contain useful distance information. The display device 110 includes at least one evaluation device 124. The evaluation device 124 may be configured to find and evaluate the zero-order reflection features of the diffraction grating, i.e., the true features, and to ignore higher-order reflection features, i.e., false features.

[0175] The evaluation device 124 is configured to evaluate a first image. The evaluation device 124 may include at least one data processing device, more preferably at least one processor and / or at least one application-specific integrated circuit. Thus, as an example, at least one evaluation device 124 may include at least one data processing device having software code stored thereon, which includes a number of computer commands. The evaluation device may provide one or more hardware elements for performing one or more specified operations and / or provide one or more processors having software running thereon for performing one or more specified operations. The operations include evaluating an image. Specifically, determining a beam profile and displaying a surface may be performed by at least one evaluation device. Thus, as an example, one or more instructions may be implemented in software and / or hardware. Thus, as an example, the evaluation device 124 may consist of one or more computers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), or one or more programmable devices such as field-programmable gate arrays (FPGAs) configured to perform the evaluation described above. However, additionally or alternatively, the evaluation apparatus may also be fully or partially embodied in hardware.

[0176] The evaluation of the first image includes identifying the reflection features of the first image. The evaluation device 124 may be configured to perform at least one image analysis and / or image processing to identify the reflection features. The image analysis and / or image processing may use at least one feature detection algorithm. The image analysis and / or image processing may include: filtering; selection of at least one region of interest; formation of a difference image between the image generated by the sensor signal and at least one offset; inversion of the sensor signal by inverting the image generated by the sensor signal; formation of a difference image between images generated by the sensor signal at different times; background correction; decomposition into color channels; decomposition into hue; saturation; luminance channels; frequency decomposition; singular value decomposition; application of a blob detector; application of a corner detector; application of a Hessian filter determinant; application of a principal curvature-based region detector; application of a maximum stable extreme region detector; application of a generalized Hough transform; application of a ridge detector; application of an affine-invariant feature detector; application of an affine adaptation point of interest operator; application of a Harris affine region detector; application of a Hessian affine region detector; application of a scale-invariant feature transform. This may include one or more of the following: application of scale-space extreme value detectors; application of local feature detectors; application of accelerated robust feature algorithms; application of gradient position and direction histogram algorithms; application of oriented gradient descriptor histograms; application of Deriche edge detectors; application of differential edge detectors; application of spatiotemporal point of interest detectors; application of Moravec corner detectors; application of Canny edge detectors; application of Laplace operators for Gaussian filters; application of differential Gaussian filters; application of Sobel operators; application of Laplace operators; application of Schall operators; application of Prewitt operators; application of Roberts operators; application of Kirsch operators; application of high-pass filters; application of low-pass filters; application of Fourier transforms; application of Radon transforms; application of Huff transforms; application of wavelet transforms; thresholding; and generation of binary images. The region of interest may be determined manually by the user or automatically by recognizing features in an image generated by an optical sensor.

[0177] For example, the illumination source 114 may be configured to generate and / or project a point cloud such that multiple illumination regions are generated on the light sensor 118, for example, a CMOS detector. Furthermore, disturbances such as speckle and / or external light and / or multiple reflections may be present on the light sensor 118. The evaluation device 124 may be adapted to determine at least one region of interest, for example, one or more pixels illuminated by a light beam used to determine the longitudinal coordinates of an object. For example, the evaluation device 124 may be adapted to perform filtering methods, such as blob analysis and / or edge filtering and / or object recognition methods.

[0178] The evaluation device 124 may be configured to perform at least one image correction. The image correction may include at least one background subtraction. The evaluation device 124 may be adapted to remove the influence of background light from the beam profile, for example, by imaging without further illumination.

[0179] Each reflection feature includes at least one beam profile. The beam profile may be selected from the group consisting of trapezoidal beam profiles, triangular beam profiles, conical beam profiles, and linear combinations of Gaussian beam profiles. The evaluation device is configured to determine beam profile information for each reflection feature by analyzing the beam profiles.

[0180] The evaluation device 124 analyzes the beam profile to determine at least one longitudinal coordinate z of each reflection feature. DPRIt is configured to determine the following. For example, the analysis of the beam profile may include at least one of the following steps: histogram analysis, calculation of difference measurements, application of a neural network, and application of a machine learning algorithm. The evaluation device 124 may be configured to symmetrize and / or normalize and / or filter the beam profile in particular to remove noise or asymmetry from recordings such as recordings at larger angles and edges. The evaluation device 124 may filter the beam profile by removing high spatial frequencies, such as by spatial frequency analysis and / or median filtering. Aggregation may be performed by the center of the light spot intensity, and all intensities at the same distance to the center can be averaged. The evaluation device 124 may be configured to normalize the beam profile to the maximum intensity in particular to take into account the intensity difference due to the recorded distance. The evaluation device 124 may be configured to remove the effect of background light from the beam profile, for example by imaging without illumination.

[0181] The evaluation device 124 uses depth techniques from the photon ratio to determine the vertical coordinate z for each reflection feature. DPR It may be configured to determine the depth from photon ratio (DPR) technique, see WO2018 / 091649A1, WO2018 / 091638A1 and WO2018 / 091640A1, the full contents of which are included by reference.

[0182] The evaluation device 124 can be configured to determine the beam profile of each reflection feature. Determining the beam profile may include identifying and / or selecting at least one reflection feature provided by the optical sensor 118, and evaluating at least one intensity distribution of the reflection feature. For example, an area of ​​an image may be used and evaluated to determine an intensity distribution, such as a three-dimensional or two-dimensional intensity distribution, along an axis or line passing through the image. For example, the center of illumination by the light beam may be determined by determining at least one pixel having the best illumination, and a cross-sectional axis may be selected passing through the center of illumination. The intensity distribution may be an intensity distribution as a function of coordinates along this cross-sectional axis passing through the center of illumination. Other evaluation algorithms are also possible.

[0183] Analysis of one beam profile of a reflection feature may include determining at least one first area and at least one second area of ​​the beam profile. The first area of ​​the beam profile may be area A1, and the second area of ​​the beam profile may be area A2. The evaluation device 124 may be configured to integrate the first area and the second area. The evaluation device may be configured to derive a combined signal, in particular a quotient Q, by dividing the integrated first area by the integrated second area, by a multiple of the integrated first area and the integrated second area, or by a linear combination of the integrated first area and the integrated second area. The evaluation device 124 may be configured to determine at least two areas of the beam profile and / or to divide the beam profile into at least two segments containing different areas of the beam profile, where overlap of areas is possible as long as the areas do not coincide. For example, the evaluation device 124 may be configured to determine multiple areas, such as two, three, four, five, or up to ten areas. The evaluation device 124 may be configured to divide a light spot into at least two areas of the beam profile, and / or to divide the beam profile into at least two segments containing different areas of the beam profile. The evaluation device 124 may be configured to determine the integral of the beam profile across each of the at least two areas. The evaluation device 124 may be configured to compare at least two of the determined integrals. Specifically, the evaluation device 124 may be configured to determine at least one first area and at least one second area of ​​the beam profile. The first area and the second area of ​​the beam profile may be adjacent regions, overlapping regions, or both. The regions of the first area and the second area of ​​the beam profile do not have to coincide.For example, the evaluation device 124 may be configured to divide the sensor area of ​​the CMOS sensor into at least two sub-regions, and the evaluation device may be configured to divide the sensor area of ​​the CMOS sensor into at least one left portion and at least one right portion, and / or at least one upper portion and at least one lower portion, and / or at least one inner portion and at least one outer portion.

[0184] Additionally or alternatively, the display device 110 may include at least two light sensors 118, the photosensitive areas of the first and second light sensors may be arranged such that the first light sensor determines a first area of ​​the beam profile of the reflective feature, and the second light sensor determines a second area of ​​the beam profile of the reflective feature. The evaluation device 124 may be adapted to integrate the first and second areas.

[0185] In one embodiment, A1 may correspond to the entire or complete area of ​​the feature point on the light sensor. A2 may be the central area of ​​the feature point on the light sensor. The central area may be a constant value. The central area may be smaller than the entire area of ​​the feature point. For example, in the case of a circular feature point, the central area may have a radius of 0.1 to 0.9, preferably 0.4 to 0.6, of the total radius of the feature point.

[0186] The evaluation device 124 may be configured to derive the quotient Q by one or more of the following: dividing the first area by the second area, dividing the first area by a multiple of the second area, or dividing the first area by a linear combination of the second area.

number

[0187] The evaluation device 124 may be configured to use at least one predetermined relationship between the quotient Q and the vertical coordinate to determine the vertical coordinate. The predetermined relationship may be one or more of empirical relationships, semi-empirical relationships, and analytically derived relationships. The evaluation device may include at least one data storage device for storing the predetermined relationship, such as a lookup list or a lookup table.

[0188] The evaluation device 124 may be configured to run a depth algorithm from at least one photon ratio that calculates distances for all zero-order and higher-order reflection features.

[0189] The evaluation of the first image includes sorting the identified reflective features with respect to luminance. Sorting may include assigning a sequence of reflective features for further evaluation, starting with the reflective feature with the highest luminance, and then decreasing in luminance. If the brightest reflective feature is preferred for DPR calculation, the vertical coordinate z DPR This can improve the robustness of the decision. This is mainly because the zero-order reflection features of the diffraction grating are always brighter than the higher-order spurious features.

[0190] The evaluation device 124 uses the vertical coordinate z DPR By using this method, the reflection features are configured to clearly match with the corresponding illumination features. The longitudinal coordinates determined using depth techniques from the photon ratio can be used to solve the so-called correspondence problem. In this way, distance information for each reflection feature can be used to find the correspondences of known laser projector grids.

[0191] Irradiation features corresponding to reflection features can be determined using epipolar geometry. For an explanation of epipolar geometry, see, for example, Chapter 2 of X. Jiang and H. Bunke, “Dreidimensionales Computersehen” (Three-Dimensional Computers), Springer, Berlin-Heidelbel, 1997. In epipolar geometry, it can be assumed that the illumination image, i.e., the undistorted illumination pattern and the first image, may be images determined in a spatial direction with different spatial positions and / or fixed distances. The distance may be a relative distance, also called a baseline. The illumination image may also be shown as a reference image. The evaluation device 124 may be adapted to determine epipolar lines in the reference image. The relative positions of the reference image and the first image may be known. For example, the relative positions of the reference image and the first image may be stored in at least one memory unit of the evaluation device. The evaluation device 124 may be adapted to determine a straight line extending from a selected reflection feature of the first image to the real-world feature it emits. Thus, the straight line may include possible object features corresponding to the selected reflection feature. The straight line and baseline unfold the epipolar plane. Because the reference image is determined in a different relative position than the first image, the features of corresponding possible objects can be imaged on a straight line in the reference image called the epipolar line. The epipolar line can be the intersection of the epipolar plane and the reference image. Therefore, the features of the reference image that correspond to selected features of the first image are located on the epipolar line.

[0192] Depending on the distance to the object in the scene that reflects the illumination feature, the reflected feature corresponding to the illumination feature may be displaced within the first image 122. The reference image may include at least one displacement region where the illumination feature corresponding to the selected reflected feature will be imaged. The displacement region may include only one illumination feature. The displacement region may include multiple illumination features. The displacement region may include an epipolar line or a portion of an epipolar line. The displacement region may include multiple epipolar lines or multiple portions of multiple epipolar lines. The displacement region may extend along an epipolar line, or perpendicular to an epipolar line, or both. The evaluation device 124 may be adapted to determine the illumination feature along an epipolar line. The evaluation device 124 may be adapted to determine the longitudinal coordinate z and error interval ±ε of the reflected feature from the coupled signal Q in order to determine a displacement region along an epipolar line corresponding to z±ε, or a displacement region perpendicular to an epipolar line. Measurement uncertainty in distance measurement using the combined signal Q may result in non-circular displacement regions in the second image because the measurement uncertainty may vary depending on the direction. Specifically, the measurement uncertainty along one or more epipolar lines may be greater than the measurement uncertainty in directions orthogonal to one or more epipolar lines. The displacement region may include extension in directions orthogonal to one or more epipolar lines. The evaluation device 124 may be adapted to match the selected reflection feature with at least one illumination feature within the displacement region. The evaluation device 124 uses the determined longitudinal coordinate z DPRThe evaluation device 124 may be adapted to match selected features of the first image with illumination features in the displacement region by using at least one evaluation algorithm that takes into account the following. The evaluation algorithm may be a linear scaling algorithm. The evaluation device 124 may be adapted to determine the epipolar line closest to and / or within the displacement region. The evaluation device may be adapted to determine the epipolar line closest to the image position of the reflection feature. The range of the displacement region along the epipolar line may be larger than the range of the displacement region perpendicular to the epipolar line. The evaluation device 124 may be adapted to determine the epipolar line before determining the corresponding illumination feature. The evaluation device 124 may determine the displacement region around the image position of each reflection feature. The evaluation device 124 may be adapted to assign an epipolar line to each displacement region at each image position of the reflection feature, for example, by assigning the epipolar line closest to the displacement region and / or within the displacement region and / or the epipolar line closest to the displacement region along the direction perpendicular to the epipolar line. The evaluation device 124 may be adapted to determine the irradiation features corresponding to the reflection features by determining the irradiation features closest to the assigned displacement region, and / or the irradiation features within the assigned displacement region, and / or the irradiation features closest to the assigned displacement region along the assigned epipolar line, and / or the irradiation features within the assigned displacement region along the assigned epipolar line.

[0193] Additionally or alternatively, the evaluation device 124 performs the following steps: - Determining the displacement region of each reflection feature relative to its image position; - Assigning epipolar lines to the displacement region of each reflection feature, such as by assigning the epipolar line closest to the displacement region, and / or the epipolar line within the displacement region, and / or the epipolar line closest to the displacement region along a direction perpendicular to the epipolar line; - Assigning and / or determining at least one irradiation feature to each reflection feature, such as by assigning the irradiation feature closest to the assigned displacement region, and / or the irradiation feature within the assigned displacement region, and / or the irradiation feature closest to the assigned displacement region along the assigned epipolar line, and / or the irradiation feature within the assigned displacement region along the assigned epipolar line. It may also be configured to perform

[0194] Additionally or alternatively, the evaluation device 124 may be adapted to determine among the irradiation features assigned to a plurality of epipolar lines and / or reflection features by, for example, comparing the distance of reflection features and / or epipolar lines in the irradiation image, and / or comparing the ε-weighted distance of the irradiation features and / or the error-weighted distance of the epipolar lines in the irradiation image, and assigning epipolar lines and / or irradiation features with shorter distances and / or ε-weighted distances to the irradiation features and / or reflection features.

[0195] As described above, the diffraction grating generates multiple reflection features, for example, one true feature and multiple false features for each illumination feature. Matching is performed starting with the brightest reflection feature and decreasing the brightness of the reflection features. No other reflection feature can be assigned to the same matched illumination feature. Due to display artifacts, the generated false features are generally darker than the true features. By sorting the reflection features by brightness, brighter reflection features are preferred for correspondence matching. If the correspondence of illumination features has already been used, the false feature cannot be assigned to the used, i.e., matched illumination feature.

[0196] Figure 2A shows a simulated first image 122 without the display 112 for an illumination pattern containing a single light spot. Figure 2B shows the first image 122 captured by the optical sensor 118 behind the display 112. It is observed that the diffraction grating is generating multiple spots. In Figure 2B, true features are indicated by reference numeral 126, and exemplary false features are indicated by reference numeral 128. Figure 2C shows a further example of the first image 122 captured by the optical sensor 118 behind the display 112, in this case the illumination pattern is a projected laser grid. Multiple spots appear due to the diffraction grating.

[0197] Figure 3A shows a further exemplary first image 122 of a scene having a projected laser spot. Zero-order and higher-order 132 reflection features of the diffraction grating 130 are shown. Figures 3B and 3C show the matching of reflection features with illumination features. The left portion of Figures 3B and 3C shows the first image 122, and the right portion shows the corresponding illumination pattern including two illumination features. The first image 122 may contain six reflection features. The evaluation device 124 may be configured to identify the reflection features in the first image 122 and sort them with respect to brightness. As shown in Figure 3B, two of the reflection features may be brighter than the others. The evaluation device 124 may initiate beam profile analysis and begin matching the illumination feature with one of the two brighter reflection features indicated by the circle 134. Each of the two brighter reflection features may be matched with one illumination feature indicated by an arrow. The evaluation device 124 may classify the matched features as true features. As shown in Figure 3C, two illumination features of the illumination pattern have already been matched with brighter reflection features. No other reflection features can be assigned to the same matched illumination features. By sorting the reflection features by brightness, brighter reflection features take precedence in the corresponding matches. If the correspondence of illumination features has already been used, a false feature cannot be assigned to the used illumination feature, i.e., the matched illumination feature. Therefore, the two remaining reflection features indicated by circle 136 do not have corresponding illumination features and cannot be assigned to any point in the pattern. These remaining reflection features are classified as false features by the evaluation device 124.

[0198] The evaluation device 124 eliminates false features and the vertical coordinate z DPR By using this, it is configured to generate a depth map for true features. The display device 110 may be used to generate a 3D map from a scene, for example, a face scene.

[0199] The depth map can be further refined by using further depth measurement techniques such as triangulation and / or depth from defocus and / or structured light. The evaluation device uses triangulation and / or depth from defocus and / or structured light techniques to determine at least one second longitudinal coordinate z for each of the reflection features. triang The evaluation device 124 may be configured to determine the second vertical coordinate z triang and vertical coordinate z DPR The system may be configured to determine the combined vertical coordinates. The combined vertical coordinates are the second vertical coordinate z triang and vertical coordinate z DPR It can be the average value. The combined vertical coordinates can be used to determine the depth map.

[0200] As shown in Figure 1B, the display device 110 may include a further illumination source 138. The further illumination source 138 may include at least one light-emitting diode (LED). The further illumination source 138 may be configured to generate light within the visual spectral range. The light sensor 118 may be configured to determine at least one second image, which includes at least one two-dimensional image of the scene. The further illumination source 138 may be configured to provide additional illumination for imaging the second image. For example, the configuration of the display device 110 can be extended by additional flood illumination LEDs. The further illumination source 138 may illuminate a scene, such as a face, using LEDs, in particular, without an illumination pattern, and the light sensor 118 may be configured to capture a two-dimensional image. The two-dimensional image can be used in face detection and verification algorithms.

[0201] If the impulse response of the display 112 is known, the distorted image captured by the optical sensor 118 can be restored. The evaluation device 124 may be configured to determine at least one corrected image I0 by deconvolving the second image I (I=I0*g) with a diffraction grating function g. The diffraction grating function is also called the impulse response. The undistorted image can be restored by a deconvolution approach, such as Van-Cittert or Wiener deconvolution.

[0202] As shown in Figure 4, the display device 110 can be configured to determine the diffraction grating function g. The display device 110 may be configured to illuminate a black scene with an illumination pattern having a small single bright spot indicated by reference numeral 140. The captured image 142 may be the diffraction grating function. This procedure may be performed only once, such as during calibration. To determine a corrected image even when imaging through the display 112, the display device 110 may be configured to capture an image and use a deconvolution approach with the captured impulse response g. The resulting image may be a reconstructed image with fewer display artifacts and can be used in several applications, such as face recognition. Figures 5A to 5C show examples of two-dimensional images captured by the optical sensor 118. In Figure 5A, an exemplary scene was captured by the optical sensor 118 located behind the display 112. In Figure 5B, an exemplary scene was captured by the optical sensor 118 without the display 112. Figure 5C shows an image reconstructed using the deconvolution approach.

[0203] Explanation of the symbols 110 Display device 112 displays 114 Irradiation source 116 Housing 118 Light Sensor 120 Photosensitive Area 122 Image 1 124 Evaluation device 126 True Characteristics 128 characteristics of fakes 130 Zero-order diffraction grating 132 Higher Order 134 yen 136 yen 138 Further Irradiation Sources 140 Illuminating a black scene 142 captured images [Prior art documents] [Patent Documents]

[0204] [Patent Document 1] DE202018003644U1 [Patent Document 2] US9,870,024B2 [Patent Document 3] US10,057,541B2 [Patent Document 4] US10,215,988B2 [Patent Document 5] WO2018 / 091649A1 [Patent Document 6] WO2018 / 091638A1 [Patent Document 7] WO2018 / 091640A1 [Patent Document 8] WO2019 / 042956A1

Claims

1. A display device (110): - An illumination source (114) configured to project an illumination pattern containing multiple illumination features onto the scene; - A camera configured to acquire a first image (122) of the scene under the illumination pattern; - A translucent display (112), wherein the illumination source (114) and the camera are positioned in front of the translucent display (112) in the propagation direction of the illumination pattern; - The reflection features and their brightness are identified from the first image (122), and the vertical coordinate z of the reflection features is determined by analyzing their beam profiles. ＤＰＲ Determine the vertical coordinate z ＤＰＲ By using the brightness, reflection features that match the illumination features are classified as true features, and reflection features that do not match the illumination features are classified as false features, and the vertical coordinate z ＤＰＲ An evaluation device (124) is configured to generate a depth map for the true features by using, A display device (110) equipped with the following:

2. The display device (110) according to claim 1, wherein the irradiation source (114) includes at least one laser projector, and the laser projector includes at least one laser source and at least one diffractive optical element (DOE).

3. The display device (110) according to claim 1 or 2, wherein the irradiation source (114) includes an array of vertical cavity surface-emitting lasers.

4. The display device (110) according to any one of claims 1 to 3, wherein the irradiation pattern includes a periodic dot pattern.

5. The display device (110) according to any one of claims 1 to 4, wherein the irradiation pattern has 2,500 or fewer points per field of view.

6. The display device (110) according to any one of claims 1 to 5, wherein the irradiation source (114) is configured to project light having a wavelength in the range of 700 to 1000 nm.

7. The display device (110) according to claim 6, wherein the wiring of the light-transmitting display (112) is configured to form grid gaps and / or slits and ridges.

8. The evaluation device (124) uses triangulation and / or defocus depth and / or structured light techniques to determine at least one second longitudinal coordinate z for each of the reflection features. ｔｒｉａｎｇ A display device (110) according to any one of claims 1 to 7, configured to determine the following.

9. The evaluation device (124) is the second vertical coordinate z ｔｒｉａｎｇ and the aforementioned vertical coordinate z ＤＰＲ It is configured to determine the combined vertical coordinates, and the combined vertical coordinates are the second vertical coordinate z ｔｒｉａｎｇ and vertical coordinate z ＤＰＲ The display device (110) according to claim 8, wherein the average value of the combined vertical coordinates is used to generate the depth map.

10. Vertical coordinate z of the reflection feature ＤＰＲ The display device (110) according to any one of claims 1 to 9, which is determined by the ratio between the intensity at the edge of the reflection feature and the intensity at the center.

11. The evaluation device (124) deconvolves the second image I with the diffraction grating function g to obtain at least one corrected image I ０ It is configured to determine, where I = I ０ *g, the display device (110) according to any one of claims 1 to 10.

12. The display device (110) according to any one of claims 1 to 11, wherein the evaluation device (124) is configured to determine the material properties m of an object by evaluating at least one beam profile of the reflection features.

13. The display apparatus (110) according to claim 12, wherein the material property m is light transmission, transparency, or deviation from Lambertian surface reflection.

14. A method for measuring depth through a translucent display (112) using a display device (110) according to any one of claims 1 to 13, comprising the following steps: a) A step of projecting at least one illumination pattern including a plurality of illumination features onto a scene using at least one illumination source (114), wherein the illumination source (114) is positioned in front of the translucent display (112) in the direction of propagation of the illumination pattern; b) A step of acquiring a first image (122) including a plurality of reflection features generated by the scene in response to illumination by the illumination feature, by using at least one camera, wherein the camera is positioned in front of the translucent display (112) in the direction of propagation of the illumination pattern; c) A step of evaluating the first image (122) using an evaluation device (124), the following substeps: C1) A substep of identifying the reflection features of the first image (122) and their brightness; C2) Analysis of the beam profile of the reflection feature determines the vertical coordinate z of the reflection feature. ＤＰＲ Substeps to determine; C3) The aforementioned vertical coordinate z ＤＰＲ By using the brightness, reflection features that match the illumination features are classified as true features, and reflection features that do not match the illumination features are classified as false features, and the vertical coordinate z ＤＰＲ A substep of generating a depth map for the aforementioned true features by using; C4) The aforementioned vertical coordinate z ＤＰＲ A substep of generating a depth map for the aforementioned true features by using; Steps including; A method that includes this.

15. Use of a display device (110) according to any one of claims 1 to 13 relating to a display device for an object recognition method.

Images