Depth measurement by display

Abstract

To provide a device and a method that allow depth measurement with high reliability through a display, with low technical efforts and low requirements from the viewpoint of technical resources and cost.SOLUTION: A display device comprises: an irradiation source 114 configured to project an irradiation pattern including a plurality of irradiation characteristics; an optical sensor 120 configured of an optical sensor 118 having a photosensitive area 120; a light-transmissive display 112 configured to display information, the light-transmissive display 112 having the irradiation source 114 and the optical sensor 118 arranged in a propagation direction of the irradiation pattern in front of the display 112; and an evaluation device 124 configured to evaluate a first image 122. The evaluation device 124 is configured to generate a depth map for true characteristics.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a display device and a method for depth measurement with a translucent display, as well as various uses of the display device. The device, method and uses according to the invention may in particular be employed in various fields, for example in everyday life, security technology, gaming, traffic technology, production technology, art, photography, such as digital photography or videography for documentation or technical purposes, security technology, information technology, agriculture, crop protection, maintenance, cosmetics, medical technology or science, although other applications are also possible. [Background technology]

[0002]

[0003] Several display devices are known. Recent developments in devices with displays have shown that the display area needs to cover the entire available space and that the frame surrounding the display needs to be as small as possible. As a result, electronic components and sensors (such as front-facing cameras, flashlights, proximity sensors, and even 3D image sensors) can no longer be located within the frame but must be placed below the display. However, the most common 3D imaging technologies and systems (such as 3D imaging systems based on structured light or 3D time-of-flight (ToF)) cannot be placed below the display without difficulty.

[0003] To date, no 3D imaging system based on structured light or 3D-ToF has been known to operate underneath a display, i.e., without creating an empty window, free of microcircuits and / or microwiring, into which components or devices of the 3D imaging system can be placed in order to "see" through the window.

[0004] In the case of structured light, the main problem is the microstructure of the microcircuits and / or microwiring in transparent displays and the resulting low light transmittance through the display. This microstructure is due to the electrode matrix used to address the single pixels. Furthermore, the single pixel's metal cathode is not transparent, so the pixel itself exhibits a reciprocal lattice. In principle, the display structure can be made entirely transparent or translucent, including the electrodes, by using certain materials, but to date, no transparent or translucent displays exist that do not have a lattice-like microstructure.

[0005] Structured light-based 3D imaging devices are based on projecting a point cloud with thousands of points and a known pattern into a scene. The microstructure of a transparent or translucent display acts like a diffraction grating structure for laser light. Because most projectors for structured light imaging devices are based on laser sources that project a well-defined dot pattern, this pattern is subject to the display's diffraction grating effect, causing all spots in the dot pattern to exhibit high diffraction orders. This has a devastating effect on structured light imaging devices, because the additional unwanted points caused by the diffraction grating structure make the algorithm for searching for the original expected pattern extremely complex.

[0006] Furthermore, the number of projection points used in conventional structured light imaging devices is quite large. Because transparent displays have very low light transmittance, even at infrared (IR) wavelengths of 850 nm and 940 nm, which are typical wavelengths of 3D imaging devices, very high output power is required for the structured light projector to have enough power to pass through the display and be detected by the imaging device, and the structured light projector must be installed below the display, which results in additional light absorption. The combination of a large number of points and low light transmittance can result in poor ambient light robustness.

[0007] In the case of 3D-ToF sensors, due to the reflections of the display surface, which results in multiple reflections, as well as differences in the delay of light as it passes through the display, different display structures have different refractive indices, preventing robust function when used behind a display. Furthermore, 3D-ToF sensors also require a large amount of light to illuminate the scene. Furthermore, the illumination must be uniform. The low light transmittance of the display makes it difficult to provide sufficient light, and the lattice structure affects the uniformity of the illumination.

[0008] Typical 3D sensing systems have trouble measuring through a transparent display, and current devices use a cutout in the display so that the sensor is not obstructed by diffractive optical effects.

[0009] DE202018003644U1 describes a portable electronic device comprising: a bottom wall and side walls that cooperate with the bottom wall to define a cavity, the side walls having edges that define an opening leading to the cavity; a protective layer that covers the opening and surrounds the cavity; and a vision subsystem that is disposed within the cavity and between the protective layer and the bottom wall and serves to provide a depth map of objects outside the protective layer, the vision subsystem comprising a clip assembly for carrying optical components that cooperate to generate depth map information, the clip assembly comprising a first bracket arranged to support and hold the optical components at a fixed distance from each other, and a second bracket having a body fixed to the first bracket (the second bracket having a protrusion extending away from the body).

[0010] US9,870,024B2 describes an electronic display including multiple layers, such as a cover layer, a color filter layer, a display layer including light-emitting diodes or organic light-emitting diodes, and a thin-film transistor layer. In one embodiment, the layers include a substantially transparent region disposed above a camera. The substantially transparent region allows external light to reach the camera, enabling the camera to record an image.

[0011] US 10,057,541 B2 describes an imaging device and an imaging method, which includes a transparent display panel and a camera facing the bottom surface of the transparent display panel, for capturing an image located in front of the transparent display panel by synchronizing a shutter time with a period during which the transparent display panel displays a black image.

[0012] US 10,215,988 B2 describes an optical system for displaying light from a scene, including an active optical component including a first plurality of light-directing apertures, a photodetector, a processor, a display, and a second plurality of light-directing apertures. The first plurality of light-directing apertures are arranged to provide optical input to the photodetector. The photodetector is arranged to receive the optical input and convert the optical input into electrical signals corresponding to intensity data and position data. The processor is connected to receive data from the optical detector and process the data for display. The second plurality of light-directing apertures are arranged to provide optical output from the display.

[0013] WO 2019 / 042956 A1 describes a detector for determining the position of at least one object, comprising: at least one sensor element having a matrix of optical sensors, each having a photosensitive area, each optical sensor designed to generate at least one sensor signal in response to illumination of its respective photosensitive area by a reflected light beam propagating from the object to the detector, the sensor element being adapted to determine at least one reflected image; and at least one evaluation device adapted to select at least one reflection feature of the reflected image and to determine at least one longitudinal region of the selected reflection feature of the reflected image by evaluating a combination signal Q from the sensor signals, the evaluation device being adapted to determine at least one displacement region in at least one reference image corresponding to the longitudinal region, and to match the selected reflection feature with at least one reference feature in the displacement region. Summary of the Invention [Problem to be solved by the invention]

[0014] It is therefore an object of the present invention to provide an apparatus and a method that overcomes the above-mentioned technical problems of known apparatus and methods, in particular to provide an apparatus and a method that allows reliable depth measurements through a display with low technical effort and low demands in terms of technical resources and costs. [Means for solving the problem]

[0015] This problem is solved by the invention with the features of the independent patent claims. Advantageous developments of the invention, which can be realized individually or in combination, are set out in the dependent claims and / or in the following description and detailed embodiments.

[0016] When used below, the terms "have," "comprise," or "include," or any grammatical variations thereof, are used in a non-exclusive manner. Thus, these terms can refer both to a situation in which, besides the features introduced by these terms, no further features are present in the entity described in this context, and to a situation in which one or more further features are present. As an example, the expressions "A has B," "A comprises B," and "A includes B" can refer both to a situation in which no other elements are present in A besides B (i.e., a situation in which A solely and exclusively consists of B), and to a situation in which, in addition to B, one or more elements are present in entity A, such as element C, elements C and D, or further elements.

[0017] Furthermore, it should be noted that the terms "at least one," "one or more," or similar expressions indicating that a feature or element may be present more than one time are typically used only once when introducing each feature or element. Note that in most cases hereinafter, when referring to each feature or element, the expressions "at least one" or "one or more" will not be repeated, despite the fact that the feature or element may appear more than one time.

[0018] Furthermore, when used hereinafter, the terms "preferably," "more preferably," "particularly," "more particularly," "particularly," "more particularly," or similar terms are used in connection with any feature without limiting the possibility of substitution. Features introduced by these terms are therefore optional features and are not intended to limit the scope of the claims in any way. The present invention can be practiced using alternative features, as will be recognized by those skilled in the art. Similarly, features introduced by "in one embodiment of the present invention" or similar expressions are intended to be optional features, without any limitation regarding alternative embodiments of the invention, without any limitation regarding the scope of the invention, and without any limitation regarding the possibility of combining features introduced in this way with other optional or non-optional features of the invention.

[0019] In a first aspect of the present invention, a display device is disclosed. As used herein, the term "display" may refer to a device of any shape configured to display an item of information, such as at least one image, at least one diagram, at least one histogram, at least one text, or at least one symbol. The display may be at least one monitor or at least one screen. The display may have any shape, preferably a rectangular shape. As used herein, the term "display device" may generally refer to at least one electronic device including at least one display. For example, the display device may be at least one device selected from the group consisting of a television, a smartphone, a game console, a personal computer, a laptop, a tablet, at least one virtual reality device, or a combination thereof.

[0020] The display device - at least one illumination source configured to project at least one illumination pattern including a plurality of illumination features onto at least one scene; - at least one light sensor having at least one light-sensitive area, the light sensor configured to determine at least one first image comprising a plurality of reflective features produced by the scene in response to illumination by the illumination feature; and - at least one light-transmitting display configured to display information, wherein the illumination source and the light sensor are arranged in front of the display in a direction of propagation of the illumination pattern; at least one evaluation device, the evaluation device configured to evaluate the first image, wherein the evaluation of the first image comprises identifying the reflection features of the first image and sorting the identified reflection features with respect to brightness, each of the reflection features comprising at least one beam profile, the evaluation device determining at least one vertical coordinate z of each of the reflection features by analysis of those beam profiles; DPR at least one evaluation device configured to determine It is equipped with The evaluation device determines the vertical coordinate z DPR and the evaluation device is configured to unambiguously match reflection features to corresponding illumination features by using a vertical coordinate z DPR The present invention is configured to generate a depth map for the true features using

[0021] As used herein, the term "scene" may refer to any at least one object or region of space. A scene may include at least one object and its surrounding environment.

[0022] The illumination source is configured to project at least one illumination pattern including a plurality of illumination features onto the scene. As used herein, the term "illumination source" may generally refer to at least one device adapted to provide at least one illumination light beam for illumination of the scene. The illumination source may be adapted to directly or indirectly illuminate the scene, and the illumination pattern is reflected or scattered by a surface of the scene and thereby directed at least in part to the light sensor. The illumination source may be adapted to illuminate the scene, for example, by directing a light beam onto the scene, which reflects the light beam. The illumination source may be configured to generate an illumination light beam for illuminating the scene.

[0023] The illumination source may include at least one light source. The illumination source may also include multiple light sources. The illumination source may include an artificial illumination source, particularly at least one laser source, and / or at least one incandescent lamp, and / or at least one semiconductor light source, such as at least one light-emitting diode, particularly an organic and / or inorganic light-emitting diode. By way of example, the light emitted by the illumination source may have a wavelength of 300 to 1100 nm, particularly 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range, such as the range of 780 nm to 3.0 μm, may be used. Specifically, light in the near-infrared range, particularly the range of 700 nm to 1100 nm, to which silicon photodiodes are applicable, may be used. The illumination source may be configured to generate at least one illumination pattern in the infrared range. Using light in the near-infrared range allows light that is not or only weakly detectable by the human eye but can be detected by silicon sensors, particularly standard silicon sensors.

[0024] As used herein, the term "light ray" generally refers to a line perpendicular to the wavefront of light, indicating the direction of energy flow. As used herein, the term "beam" generally refers to a collection of light rays. Hereinafter, the terms "light ray" and "beam" are used synonymously. As further used herein, the term "light beam" generally refers to a quantity of light, specifically a quantity of light traveling in essentially the same direction, including the possibility that the light beam has an expansion or divergence angle. The light beam can have a spatial extent. Specifically, the light beam can have a non-Gaussian beam profile. The beam profile may be selected from the group consisting of a trapezoidal beam profile; a triangular beam profile; or a conical beam lateral intensity profile. The trapezoidal beam profile can have a plateau region and at least one edge region. The light beam can specifically be a Gaussian light beam or a linear combination of Gaussian light beams, as outlined in more detail below. However, other embodiments are possible.

[0025] The illumination source can be configured to emit light at a single wavelength. In particular, the wavelength may be in the near-infrared region. In other embodiments, the illumination may be adapted to emit light having multiple wavelengths, allowing for additional measurements in other wavelength channels.

[0026] The illumination source may be or may include at least one multiple beam light source. For example, the illumination source may include at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the illumination source may comprise at least one laser and / or laser source. Various types of lasers may be employed, such as semiconductor lasers, double heterostructure lasers, external cavity lasers, separate confinement heterostructure lasers, quantum cascade lasers, distributed Bragg reflector lasers, polariton lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, indium arsenide lasers, transistor lasers, diode-pumped lasers, distributed feedback lasers, quantum well lasers, interband cascade lasers, gallium arsenide lasers, semiconductor ring lasers, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alternatively, non-laser light sources, such as LEDs and / or light bulbs, may be used. The illumination source may include one or more diffractive optical elements (DOEs) adapted to generate an illumination pattern. For example, the illumination source may be adapted to generate and / or project a point cloud, and may include one or more of the following: at least one digital light processing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one array of light-emitting diodes; or at least one array of laser light sources. Considering their generally defined beam profile and other characteristics of ease of handling, the use of at least one laser source as the illumination source is particularly preferred. The illumination source may be integrated into the housing of the display device.

[0027] In one embodiment, the illumination source may be a single or multiple beam source and may be configured to project at least one illumination pattern, such as at least one dot pattern. The illumination pattern may be generated as follows: The illumination source may be configured to generate at least one light beam. The illumination source may be positioned in front of the display in the propagation direction of the illumination pattern. Thus, the beam path of the light beam may pass from the illumination source through the display to the scene. While passing through the display, the light beam may be diffracted by the display, which may result in a characteristic illumination pattern, such as a dot pattern. The display in this embodiment may function as a grating. The wiring of the display, particularly the wiring of the screen, may be configured to form gaps and / or slits and ridges of the grating.

[0028] Furthermore, the illumination sources may be configured to emit modulated or unmodulated light. When multiple illumination sources are used, different illumination sources may have different modulation frequencies, which may then be used to distinguish between the light beams, as outlined in more detail below.

[0029] The one or more light beams generated by the illumination source may generally propagate parallel to the optical axis or at an angle relative to the optical axis, for example, at an angle with the optical axis. The display device may be configured such that the one or more light beams propagate from the display device toward the scene along the optical axis of the display device. To this end, the display device may include at least one reflective element, preferably at least one prism, for deflecting the illumination light beam onto the optical axis. As an example, the one or more light beams, such as laser light beams, and the optical axis may have an angle of less than 10°, preferably less than 5°, or even less than 2°. However, other embodiments are also possible. Furthermore, the one or more light beams may be on the optical axis or off the optical axis. As an example, the one or more light beams may be parallel to the optical axis or even coincident with the optical axis at a distance of less than 10 mm, preferably less than 5 mm, or even less than 1 mm from the optical axis.

[0030] As used herein, the term "at least one illumination pattern" refers to at least one arbitrary pattern including at least one illumination feature adapted to illuminate at least a portion of a scene. As used herein, the term "illumination feature" refers to at least one feature extending over at least a portion of a pattern. An illumination pattern may include a single illumination feature. An illumination pattern may include multiple illumination features. An illumination pattern may be selected from the group consisting of at least one dot pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; and at least one pattern including a periodic or aperiodic feature arrangement. An illumination pattern may include a regular and / or regular and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or even a convex tiling pattern. An illumination pattern may exhibit at least one illumination feature selected from the group consisting of at least one dot; at least one line; at least two lines, such as parallel or intersecting lines; at least one dot and one line; at least one periodic or aperiodic feature arrangement; and at least one feature of any shape. The illumination pattern may include at least one pattern selected from the group consisting of: at least one point pattern, in particular a pseudorandom point pattern; a random point pattern or a quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern including at least one known feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern; at least one pattern including a convex uniform tiling; at least one line pattern including at least one line; or at least one line pattern including at least two lines, such as parallel or intersecting lines. For example, the illumination source may be adapted to generate and / or project a point cloud. The illumination source may include at least one light projector adapted to generate a point cloud so that the illumination pattern can include multiple point patterns.The illumination source may include at least one mask adapted to generate an illumination pattern from at least one light beam generated by the illumination source.

[0031] The distance between two features of the illumination pattern and / or the area of ​​at least one illumination feature may depend on the circle of confusion in the image. As outlined above, the illumination source may include at least one light source configured to generate at least one illumination pattern. Specifically, the illumination source includes at least one laser source and / or at least one laser diode assigned to generate laser radiation. The illumination source may include at least one diffractive optical element (DOE). The display device may include at least one laser source and at least one point projector, such as a DOE, adapted to project at least one periodic point pattern.

[0032] As further used herein, the term "projecting at least one illumination pattern" refers to providing at least one illumination pattern for illuminating at least one scene.

[0033] For example, the projected illumination pattern may be a periodic dot pattern. The projected illumination pattern may have a low dot density. For example, the illumination pattern may include at least one periodic dot pattern with a low dot density, such that the illumination pattern has 2500 or fewer dots per field of view. Compared to structured light, which typically has a dot density of 10k to 30k in a 55 x 38° field of view, the illumination pattern according to the present invention may have a lower density. This allows the proposed technology to increase the power per dot so that it is less dependent on ambient light compared to structured light.

[0034] The display device may include one camera with a light sensor.The display device may include multiple cameras, each with one or more light sensors.

[0035] An optical sensor has at least one photosensitive area. As used herein, "optical sensor" generally refers to a photosensitive device for detecting a light beam, such as for detecting illumination and / or a light spot generated by at least one optical beam. As further used herein, "photosensitive area" generally refers to an area of ​​an optical sensor that is externally illuminated by at least one optical beam and generates at least one sensor signal in response to the illumination. The photosensitive area may specifically be located on the surface of each optical sensor. However, other embodiments are possible. A display device may include multiple optical sensors, each having a photosensitive area. As used herein, the term "optical sensor each having at least one photosensitive area" refers to a configuration including multiple single optical sensors, each having a photosensitive area, and a configuration including a combined optical sensor having multiple photosensitive areas. The term "optical sensor" further refers to a photosensitive device configured to generate an output signal. When the display device includes multiple light sensors, each light sensor may be implemented by providing, for example, exactly one light-sensitive area that can be illuminated, such that exactly one light-sensitive area is present within each light sensor, and generating exactly one uniform sensor signal for the entire light sensor in response to illumination of the light-sensitive area. Thus, each light sensor may be a single-area light sensor. The use of single-area light sensors, however, makes the construction of the display device particularly simple and efficient. Thus, by way of example, commercially available light sensors, such as commercially available silicon photodiodes, each having exactly one light-sensitive area, may be used in the construction. However, other embodiments are possible.

[0036] Preferably, the photosensitive area may be oriented substantially perpendicular to the optical axis of the display device, which may be a straight optical axis or may even be refracted or split, such as by using one or more deflection elements and / or one or more beam splitters, in the latter case the substantially perpendicular orientation refers to the local optical axis of each branch or beam path of the optical arrangement.

[0037] The light sensor may be or include at least one photodetector, preferably an inorganic photodetector, more preferably an inorganic semiconductor photodetector, and most preferably a silicon photodetector. Specifically, the light sensor may be sensitive in the infrared spectral range. All pixels of the matrix, or at least a group of light sensors of the matrix, may be identical. Groups of identical pixels of the matrix may be provided for different spectral ranges, or all pixels may be identical in terms of spectral sensitivity. Furthermore, the pixels may be identical in terms of size and / or their electronic or optoelectronic properties. Specifically, the light sensor may be or include at least an inorganic photodiode sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers. Specifically, the light sensor may be sensitive in the part of the near-infrared range, particularly in the range of 700 nm to 1100 nm, where silicon photodiodes are applicable. Infrared light sensors that can be used in the optical sensor may be commercially available infrared light sensors, such as those sold under the brand name Hertzstueck® by trinamiX GmbH, D-67056 Ludwigshafen am Rhein, Germany. Thus, by way of example, the optical sensor may include at least one optical sensor of an intrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of a Ge photodiode, an InGaAs photodiode, an extended InGaAs photodiode, an InAs photodiode, an InSb photodiode, and an HgCdTe photodiode. Additionally or alternatively, the optical sensor may include at least one optical sensor of an extrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of a Ge:Au photodiode, a Ge:Hg photodiode, a Ge:Cu photodiode, a Ge:Zn photodiode, a Si:Ga photodiode, and a Si:As photodiode.Additionally or alternatively, the optical sensor may include at least one photoconductive sensor, such as a PbS or PbSe sensor, a bolometer, preferably a bolometer selected from the group consisting of a VO bolometer and an amorphous Si bolometer.

[0038] The optical sensors may be sensitive in one or more of the ultraviolet, visible, or infrared spectral ranges. Specifically, the optical sensors may be sensitive in the visible spectral range of 500 nm to 780 nm, most preferably 650 nm to 750 nm, or 690 nm to 700 nm. Specifically, the optical sensors may be sensitive in the near-infrared range. Specifically, the optical sensors may be sensitive in the near-infrared range, particularly in the range of 700 nm to 1000 nm, where silicon photodiodes are applicable. Specifically, the optical sensors may be sensitive in the infrared spectral range, specifically in the range of 780 nm to 3.0 μm. For example, the optical sensors may each independently be or include at least one element selected from the group consisting of a photodiode, a photocell, a photoconductor, a phototransistor, or any combination thereof. For example, the optical sensor may be or include at least one element selected from the group consisting of a CCD sensor element, a CMOS sensor element, a photodiode, a photocell, a photoconductor, a phototransistor, or any combination thereof. Any other type of photosensitive element may also be used. The photosensitive element generally can be made entirely or partially of inorganic materials and / or can be made entirely or partially of organic materials. Most commonly, one or more photodiodes, such as commercially available photodiodes, e.g., inorganic semiconductor photodiodes, may be used.

[0039] The optical sensor may include at least one sensor element that includes a matrix of pixels. Thus, by way of example, the optical sensor may be part of or constitute a pixelated optical device. For example, the optical sensor may be and / or include at least one CCD device and / or CMOS device. By way of example, the optical sensor may be part of or constitute at least one CCD device and / or CMOS device having a matrix of pixels, each pixel forming a light-sensitive area.

[0040] As used herein, the term "sensor element" generally refers to a device or a combination of devices configured to sense at least one parameter. In this case, the parameter may specifically be an optical parameter, and the sensor element may specifically be an optical sensor element. The sensor element may be formed as a single, integrated device or as a combination of several devices. The sensor element includes a matrix of optical sensors. The sensor element may include at least one CMOS sensor. The matrix may be composed of individual pixels, such as individual optical sensors. Thus, a matrix of inorganic photodiodes may be formed. However, alternatively, commercially available matrices may be used, for example, one or more CCD detectors, such as a CCD detector chip, and / or CMOS detectors, such as a CMOS detector chip. Thus, in general, the sensor element may be and / or include at least one CCD device and / or CMOS device, and / or the optical sensor may form a sensor array or be part of a sensor array, such as the matrix described above. Thus, by way of example, the sensor element may comprise an array of pixels, such as a rectangular array having m rows and n columns, where m and n are independently positive integers. Preferably, multiple columns and multiple rows are provided, i.e., n>1, m>1. Thus, by way of example, n may be 2 to 16 or more, and m may be 2 to 16 or more. Preferably, the ratio of the number of rows to the number of columns is close to 1. By way of example, n and m may be selected such that 0.3≦m / n≦3, such as by selecting m / n=1:1, 4:3, 16:9, or the like. By way of example, the array may be a square array having equal numbers of rows and columns, such as by selecting m=2, n=2, or m=3, n=3, etc.

[0041] The matrix may be composed of individual pixels, such as individual photosensors. It may thus constitute a matrix of inorganic photodiodes. However, alternatively, commercially available matrices may be used, for example one or more CCD detectors, such as CCD detector chips, and / or CMOS detectors, such as CMOS detector chips. In general, the photosensor may therefore be and / or include at least one CCD and / or CMOS device, and / or the photosensor of the display device may form a sensor array or be part of a sensor array, such as the matrix described above.

[0042] The matrix may be specifically a rectangular matrix having at least one row, preferably multiple rows and multiple columns. As an example, the rows and columns may be oriented in a substantially perpendicular direction. As used herein, the term "substantially perpendicular" refers to a perpendicular orientation with a tolerance of ±20° or less, preferably ±10° or less, and more preferably ±5° or less. Similarly, the term "substantially parallel" refers to a parallel orientation with a tolerance of, for example, ±20° or less, preferably ±10° or less, and more preferably ±5° or less. Thus, as an example, a tolerance of less than 20°, specifically less than 10°, or even less than 5° may be tolerated. To provide a wide field of view, the matrix may specifically have at least 10 rows, preferably at least 500 rows, and more preferably at least 1,000 rows. Similarly, the matrix may have at least 10 columns, preferably at least 500 columns, and more preferably at least 1,000 columns. The matrix may include at least 50 optical sensors, preferably at least 100,000 optical sensors, and more preferably at least 5,000,000 optical sensors. The matrix may contain a number of pixels in the range of several megapixels. However, other embodiments are possible. Thus, in configurations where axial rotational symmetry is expected, a circular or concentric arrangement of the photosensors of the matrix, which may also be called pixels, may be preferred.

[0043] Thus, by way of example, the sensor element may be part of or constitute a pixelated optical device. For example, the sensor element may be and / or include at least one CCD and / or CMOS device. By way of example, the sensor element may be part of or constitute at least one CCD and / or CMOS device having a matrix of pixels, each pixel forming a light-sensitive area. The sensor element may employ a rolling shutter or a global shutter scheme for reading the matrix of optical sensors.

[0044] The display device may further include at least one transfer device. The display device may further include one or more additional elements, such as one or more additional optical elements. The display device may include at least one optical element selected from the group of transfer devices, such as at least one lens and / or at least one lens system, and at least one diffractive optical element. The term "transfer device," also referred to as "transfer system," may generally refer to one or more optical elements adapted to modify a light beam, such as by changing one or more of the beam parameters of the light beam, the width of the light beam, or the direction of the light beam. The transfer device may be adapted to direct the light beam to a light sensor. The transfer device may specifically include one or more of the following: at least one lens, for example, at least one lens selected from the group consisting of at least one adjustable-focus lens, at least one aspherical lens, at least one spherical lens, and at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflecting element, preferably at least one mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitting mirror; and at least one multi-lens system. As used herein, the term "focal length" of a transfer device refers to the distance at which incident parallel light rays that may impinge on the transfer device are focused to a "focus," also called a "focal point." Thus, the focal length constitutes an indicator of the transfer device's ability to converge an incident light beam. Therefore, the transfer device may include one or more imaging elements that may have the effect of a focusing lens. For example, the transfer device may include one or more lenses, in particular one or more refractive lenses, and / or one or more convex mirrors. In this example, focal length can be defined as the distance from the center of the thin refractive lens to the principal focus of the thin lens. For a converging thin refractive lens, such as a convex or biconvex thin lens, focal length can be considered positive and give the distance over which collimated light striking the thin lens as a translator can be focused to a single spot.Furthermore, the transfer device can include at least one wavelength-selective element, for example at least one optical filter. Furthermore, the transfer device can be designed to impart a predetermined beam profile to the electromagnetic radiation, for example at the location of the sensor region, specifically at the sensor area. In principle, any of the above-described embodiments of the transfer device can be realized individually or in any desired combination.

[0045] The transfer device may have an optical axis. Specifically, the display device and the transfer device have a common optical axis. As used herein, the term "optical axis of the transfer device" generally refers to an axis of mirror symmetry or rotational symmetry of a lens or lens system. The optical axis of the display device may be a line of symmetry of the optical configuration of the display device. The display device has at least one transfer device, preferably at least one transfer system having at least one lens. The transfer system may, for example, include at least one beam path, in which the elements of the transfer system in the beam path are arranged rotationally symmetrically with respect to the optical axis. Furthermore, as will be explained in more detail below, one or more optical elements arranged in the beam path may be off-centered or tilted with respect to the optical axis. However, in this case, the optical axis may be subsequently defined by interconnecting the centers of the optical elements in the beam path, such as by interconnecting the centers of the lenses, and in this context, the optical sensor is not considered an optical element. The optical axis may generally refer to the beam path. There, the display device may have a single beam path along which the light beam travels from the object to the light sensor, or may have multiple beam paths. As an example, a single beam path may be provided, or the beam path may be divided into two or more partial beam paths. In the latter case, each partial beam path may have its own optical axis. The light sensor may be located in one and the same beam path or partial beam path. Alternatively, however, the light sensor may also be located in different partial beam paths.

[0046] The transfer device may form a coordinate system where the longitudinal coordinate is the coordinate along the optical axis and d is the spatial offset from the optical axis. The coordinate system may be a polar coordinate system where the optical axis of the transfer device forms the z-axis and distance and polar angle from the z-axis can be used as additional coordinates. Directions parallel or anti-parallel to the z-axis can be considered longitudinal directions, and coordinates along the z-axis can be considered longitudinal coordinates. Any direction perpendicular to the z-axis can be considered horizontal directions, and polar coordinates and / or polar angles can be considered horizontal coordinates.

[0047] A display device may constitute a coordinate system in which the optical axis of the display device forms the z-axis, and in addition, x- and y-axes are provided that are orthogonal to the z-axis and orthogonal to each other. As an example, the display device and / or a portion of the display device may be located at a particular point in this coordinate system, such as the origin of the coordinate system. In this coordinate system, a direction parallel or anti-parallel to the z-axis may be considered a vertical direction, and a coordinate along the z-axis may be considered a vertical coordinate. Any direction perpendicular to the vertical direction may be considered a horizontal direction, and the x- and / or y-coordinates may be considered horizontal coordinates.

[0048] Alternatively, other types of coordinate systems may be used. Thus, as an example, a polar coordinate system may be used in which the optical axis forms the z-axis, and distance from the z-axis and polar angle may be used as additional coordinates. Similarly, directions parallel or anti-parallel to the z-axis may be considered longitudinal directions, and coordinates along the z-axis may be considered longitudinal coordinates. Any direction perpendicular to the z-axis may be considered transverse, and polar coordinates and / or polar angles may be considered transverse coordinates.

[0049] The optical sensor is configured to determine at least one first image including a plurality of reflective features generated by the scene in response to illumination by the illumination feature. As used herein, without limitation, the term "image" may specifically refer to data recorded by using the optical sensor, such as a plurality of electronic readings from an imaging device, such as pixels of a sensor element. The image itself may therefore include pixels, and the pixels of the image correlate to pixels of the matrix of sensor elements. Thus, a reference to a "pixel" may refer either to a unit of image information generated by a single pixel of a sensor element or directly to a single pixel of a sensor element. As used herein, the term "two-dimensional image" may generally refer to an image having information about lateral coordinates, such as height and width dimensions only. As used herein, the term "three-dimensional image" may generally refer to an image having information about lateral coordinates as well as information about vertical coordinates, such as height, width, and depth dimensions. As used herein, the term "reflective feature" may specifically refer to a feature in an image plane generated by the scene in response to illumination by at least one illumination feature.

[0050] The display device includes at least one translucent display configured to display information. As used herein, the term "translucent" may refer to the property of a display that allows light, particularly light within a specific wavelength range, to pass through. An illumination source and a light sensor are positioned in front of the display in the direction of propagation of the illumination pattern. The illumination source and the light sensor may be positioned at fixed positions relative to each other. For example, a display device configuration may include a laser projector, a camera with a light sensor and a lens system, and the laser projector and camera may be fixed behind the translucent display in the direction of propagation of light reflected by the scene. The laser projector may generate a dot pattern and project it through the display. The camera may then capture an image through the display. However, positioning the illumination source and the light sensor behind the translucent display in the direction of propagation of light reflected by the scene may result in the display's diffraction grating generating multiple laser points on the scene and in the first image. As a result, these multiple spots on the first image may not contain useful distance information. As outlined in detail below, the evaluation device may be configured to find and evaluate the zero-order reflection features of the diffraction grating, i.e., true features, and can ignore higher order reflection features, i.e., false features.

[0051] The display device includes at least one evaluation device. The evaluation device is configured to evaluate the first image. As further used herein, the term "evaluation device" generally refers to any device adapted to perform specified operations, preferably by using at least one data processing device, more preferably by using at least one processor and / or at least one application-specific integrated circuit. Thus, by way of example, the at least one evaluation device may include at least one data processing device having software code stored thereon, including a number of computer commands. The evaluation device may provide one or more hardware elements for performing one or more of the specified operations and / or one or more processors having software running thereon for performing one or more of the specified operations. The operations include evaluating the image. Specifically, determining a beam profile and displaying a surface may be performed by the at least one evaluation device. Thus, by way of example, one or more instructions may be implemented in software and / or hardware. Thus, by way of example, the evaluation device may consist of one or more programmable devices, such as one or more computers, application specific integrated circuits (ASICs), digital signal processors (DSPs), or field programmable gate arrays (FPGAs), configured to perform the evaluations described above. However, additionally or alternatively, the evaluation device may also be embodied fully or partly in hardware.

[0052] The evaluation device and the display device may be fully or partially integrated into a single device. Thus, in general, the evaluation device may also form part of the display device. Alternatively, the evaluation device and the display device may be embodied, fully or partially, as separate devices. The display device may include further components.

[0053] The evaluation device may be or include one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs), and / or one or more computers, preferably one or more microcomputers and / or one or more data processing devices, such as microcontrollers, field programmable gate arrays, or digital signal processors. Additional components may be included, for example, data acquisition devices, such as one or more preprocessing devices and / or one or more devices for receiving and / or preprocessing the sensor signals, such as one or more analog-to-digital converters and / or one or more filters. Furthermore, the evaluation device may comprise one or more measurement devices, such as one or more measurement devices for measuring current and / or voltage. Furthermore, the evaluation device may include one or more data storage devices. Furthermore, the evaluation device may include one or more interfaces, for example one or more wireless interfaces and / or one or more wired interfaces.

[0054] The evaluation device can be connected to or include at least one further data processing device that can be used for one or more of displaying, visualizing, analyzing, disseminating, communicating, or further processing information, such as information obtained by the optical sensor and / or the evaluation device. The data processing device can be connected to or incorporate at least one of a display, projector, monitor, LCD, TFT, loudspeaker, multi-channel sound system, LED pattern, or further visualization device, by way of example. It can further be connected to or incorporate at least one communication device or communication interface, connector, or port that can send encrypted or unencrypted information using one or more of email, text message, telephone, Bluetooth, Wi-Fi, infrared, or Internet interfaces, ports, or connections. It may further be connected to or incorporate at least one of the following: a processor, a graphics processor, a CPU, an Open Multimedia Applications Platform (OMAP®), an integrated circuit, a system on a chip such as a product from the Apple A series or Samsung S3C2 series, a system on a chip such as a microcontroller or microprocessor, one or more memory blocks such as ROM, RAM, EEPROM or flash memory, a timing source such as an oscillator or a phase locked loop, a counter timer, a real time timer or a power-on-reset-generator, a voltage regulator, a power management circuit or a DMA controller. The individual units may further be connected to or integrated into an Internet of Things or Industry 4.0 type network by means of a bus such as an AMBA bus.

[0055] The evaluation device and / or data processing device may be connected by or have a serial or parallel interface or port, or an analog interface or port, such as one or more of a serial or parallel interface or port, a USB, a Centronics Port, FireWire®, HDMI®, Ethernet®, Bluetooth®, RFID, Wi-Fi, USART, or SPI, or one or more standardized interfaces or ports to further devices, such as a 2D camera device using an ADC, a DAC, or an RGB interface such as CameraLink. The evaluation device and / or data processing device may be further connected by one or more of a processor-to-processor interface or port, an FPGA-FPGA interface, or a serial or parallel interface port. The evaluation device and data processing device may further be connected to one or more of an optical disk drive, a CD-RW drive, a DVD+RW drive, a flash drive, a memory card, a disk drive, a hard disk drive, a solid-state disk, or a solid-state hard disk.

[0056] The evaluation device and / or the data processing device may be connected by or may have one or more further external connectors, such as one or more of the following: phone connectors, RCA connectors, VGA connectors, hermaphroditic connectors, USB connectors, HDMI® connectors, 8P8C connectors, BCN connectors, IEC60320 C14 connectors, optical fiber connectors, D-subminiature connectors, RF connectors, coaxial connectors, SCART connectors, XLR connectors, and / or may incorporate at least one suitable socket for one or more of these connectors.

[0057] The evaluation device is configured to evaluate a first image. The evaluation of the first image includes identifying reflective features in the first image. The evaluation device may be configured to perform at least one image analysis and / or image processing to identify the reflective features. The image analysis and / or image processing may use at least one feature detection algorithm. The image analysis and / or image processing may include: filtering; selecting at least one region of interest; forming a difference image between an image generated by the sensor signal and at least one offset; inverting the sensor signal by inverting the image generated by the sensor signal; forming a difference image between images generated by the sensor signal at different times; background correction; decomposition into color channels; decomposition into hue; saturation; luminance channel; frequency decomposition; singular value decomposition; applying a blob detector; applying a corner detector; applying a determinant of a Hessian filter; applying a principal curvature-based region detector; applying a maximum stable extremum region detector; applying a generalized Hough transform; applying a ridge detector; applying an affine-invariant feature detector; applying an affine-adaptive interest point operator; applying a Harris affine region detector; applying a Hessian affine region detector; applying a scale-invariant feature transform. The method may include one or more of: applying a scale-space extremum detector; applying a local feature detector; applying a fast robust feature algorithm; applying a histogram of gradient position and orientation algorithm; applying a histogram of oriented gradient descriptors; applying a Deriche edge detector; applying a differential edge detector; applying a space-time interest point detector; applying a Moravec corner detector; applying a Canny edge detector; applying a Laplace operator of a Gaussian filter; applying a differential Gaussian filter; applying a Sobel operator; applying a Laplace operator; applying a Schall operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying a Radon transform; applying a Hough transform; applying a wavelet transform; thresholding; or generating a binary image. The region of interest may be determined manually by a user or automatically, such as by recognizing features in an image generated by an optical sensor.

[0058] For example, the illumination source may be configured to generate and / or project a point cloud such that a plurality of illumination areas are generated on an optical sensor, e.g., a CMOS detector. Furthermore, disturbances, e.g., due to speckle and / or extraneous light and / or multiple reflections, may be present on the optical sensor. The evaluation device may be adapted to determine at least one region of interest, e.g., one or more pixels illuminated by the light beam, which are used to determine the longitudinal coordinate of the object. For example, the evaluation device may be adapted to perform filtering methods, e.g., blob analysis and / or edge filters and / or object recognition methods.

[0059] The evaluation device may be configured to perform at least one image correction, which may include at least one background subtraction, and may be adapted to remove background light from the beam profile, for example by imaging without further illumination.

[0060] Each of the reflective features includes at least one beam profile. As used herein, the term "beam profile" of a reflective feature may generally refer to at least one intensity distribution of the reflective feature, such as a light spot on an optical sensor, as a function of pixel. The beam profile may be selected from the group consisting of a trapezoidal beam profile; a triangular beam profile; a linear combination of a conical beam profile and a Gaussian beam profile. The evaluation device is configured to determine beam profile information for each of the reflective features by analyzing the beam profiles.

[0061] The evaluation device determines at least one vertical coordinate z of each reflection feature by analyzing the beam profile. DPRThe evaluation device may be configured to determine the beam profile. As used herein, the term "beam profile analysis" may generally refer to the evaluation of the beam profile and may include at least one mathematical operation, and / or at least one comparison, and / or at least one symmetrization, and / or at least one filtering, and / or at least one normalization. For example, the analysis of the beam profile may include at least one of a histogram analysis step, calculation of a difference measurement, application of a neural network, and application of a machine learning algorithm. The evaluation device may be configured to symmetrize and / or normalize and / or filter the beam profile, particularly to remove noise or asymmetries from the recording, such as recordings at larger angles, edges, etc. The evaluation device may filter the beam profile by removing high spatial frequencies, such as by spatial frequency analysis and / or median filtering. The aggregation may be performed by the center of the intensity of the light spot, averaging all intensities at the same distance to the center. The evaluation device may be configured to normalize the beam profile to the maximum intensity, particularly to take into account intensity differences due to the recorded distance. The evaluation device may be configured to remove the influence of background light from the beam profile, for example by imaging without illumination.

[0062] The reflection feature may cover or extend over at least one pixel of the image. For example, the reflection feature may cover or extend over multiple pixels. The evaluation device may be configured to determine and / or select all pixels that are connected to and / or belong to the reflection feature, e.g., the light spot. The evaluation device may determine the center of intensity by:

number

number

[0063] The evaluation device determines the vertical coordinate z for each of the reflective features by using a depth from photon ratio technique, also known as beam profile analysis. DPR With regard to Depth from Photon Ratio (DPR) techniques, reference is made to WO2018 / 091649A1, WO 2018 / 091638A1 and WO2018 / 091640A1, the entire contents of which are incorporated by reference.

[0064] The evaluation device can be configured to determine a beam profile for each of the reflective features. As used herein, the term "determining a beam profile" refers to identifying at least one reflective feature provided by an optical sensor and / or selecting at least one reflective feature provided by an optical sensor and evaluating the intensity distribution of at least one of the reflective features. As an example, a region of the matrix can be used and evaluated to determine an intensity distribution, such as a three-dimensional intensity distribution or a two-dimensional intensity distribution, such as along an axis or line through the matrix. As an example, the center of illumination by the light beam can be determined, such as by determining at least one pixel with the highest illumination, and a cross-sectional axis can be selected through the center of illumination. The intensity distribution can be the intensity distribution as a function of coordinate along this cross-sectional axis through the center of illumination. Other evaluation algorithms are also possible.

[0065] The analysis of the beam profile of one of the reflection features may include determining at least one first area and at least one second area of ​​the beam profile. The first area of ​​the beam profile may be area A1, and the second area of ​​the beam profile may be area A2. The evaluation device may be configured to integrate the first area and the second area. The evaluation device may be configured to derive a combined signal, in particular a quotient Q, by one or more of: dividing the integrated first area and the integrated second area; dividing a multiple of the integrated first area and the integrated second area; or dividing a linear combination of the integrated first area and the integrated second area. The evaluation device may be configured to determine at least two areas of the beam profile and / or divide the beam profile into at least two segments having different areas of the beam profile, although overlapping of the areas is possible as long as the areas do not coincide. For example, the evaluation device may be configured to determine multiple areas, such as two, three, four, five, or up to ten areas. The evaluation device may be configured to divide the light spot into at least two areas of the beam profile and / or divide the beam profile into at least two segments comprising different areas of the beam profile. The evaluation device may be configured to determine, for each of the at least two areas, an integral of the beam profile over each of the areas. The evaluation device may be configured to compare at least two of the determined integrals. Specifically, the evaluation device may be configured to determine at least one first area and at least one second area of ​​the beam profile. As used herein, the term "area of ​​the beam profile" generally refers to any region of the beam profile at the optical sensor location used to determine the quotient Q. The first area of ​​the beam profile and the second area of ​​the beam profile may be adjacent or overlapping, or both. The first area of ​​the beam profile and the second area of ​​the beam profile may not be coincident in area.For example, the evaluation device may be configured to divide the sensor area of ​​the CMOS sensor into at least two sub-areas, such as at least one left portion and at least one right portion, and / or at least one upper portion and at least one lower portion, and / or at least one inner portion and at least one outer portion. Additionally or alternatively, the display device may include at least two photosensors, and the light-sensitive areas of the first and second photosensors may be arranged such that the first photosensor is adapted to determine a first area of ​​a beam profile of the reflective feature, and the second photosensor is adapted to determine a second area of ​​the beam profile of the reflective feature. The evaluation device may be adapted to integrate the first and second areas. The evaluation device may be configured to use at least one predetermined relationship between the quotient Q and the vertical coordinate to determine the vertical coordinate. The predetermined relationship may be one or more of an empirical relationship, a semi-empirical relationship, and an analytically derived relationship. The evaluation device may comprise at least one data store for storing the predetermined relationships, such as for example a look-up list or a look-up table.

[0066] The first area of ​​the beam profile may substantially include edge information of the beam profile, the second area of ​​the beam profile may substantially include central information of the beam profile, and / or the first area of ​​the beam profile may substantially include information about the left portion of the beam profile, and the second area of ​​the beam profile may substantially include information about the right portion of the beam profile. The beam profile may have a center, i.e., the maximum value of the beam profile and / or the center point of the plateau of the beam profile and / or the geometric center of the light spot, and a trailing edge extending from the center. The second area may include an inner region of the cross-section, and the first area may include an outer region of the cross-section. As used herein, the term "substantially central information" generally refers to a low percentage of edge information, i.e., a low percentage of the intensity distribution corresponding to the edge, compared to the percentage of central information, i.e., a percentage of the intensity distribution corresponding to the center. Preferably, the central information has an edge information percentage of less than 10%, more preferably less than 5%, and most preferably, the central information does not include edge content. As used herein, the term "substantially edge information" generally refers to a low percentage of central information compared to the percentage of edge information. The edge information may include information from the entire beam profile, particularly the central and edge regions. The edge information has a percentage of central information of less than 10%, preferably less than 5%, and more preferably the edge information does not include central information. If the beam profile is close to or around the center and substantially includes central information, at least one area of ​​the beam profile may be determined and / or selected as the second area of ​​the beam profile. If the beam profile includes at least a portion of the falling edge of the cross section, at least one area of ​​the beam profile may be determined and / or selected as the first area of ​​the beam profile. For example, the entire area of ​​the cross section may be determined as the first area.

[0067] Other selections of the first area A1 and the second area A2 may also be feasible. For example, the first area may include a substantially outer region of the beam profile, and the second area may include a substantially inner region of the beam profile. For example, in the case of a two-dimensional beam profile, the beam profile may be divided into a left portion and a right portion, where the first area may substantially include the area of ​​the left portion of the beam profile, and the second area may substantially include the area of ​​the right portion of the beam profile.

[0068] The edge information may include information about the number of photons in a first area of ​​the beam profile, and the center information may include information about the number of photons in a second area of ​​the beam profile. The evaluation device may be configured to determine a surface integral of the beam profile. The evaluation device may be configured to determine the edge information by integration and / or summation of the first area. The evaluation device may be configured to determine the center information by integration and / or summation of the second area. For example, the beam profile may be a trapezoidal beam profile, and the evaluation device may be configured to determine the integral of the trapezoid. Furthermore, if a trapezoidal beam profile is assumed, the determination of the edge and center signals may be replaced by equivalent evaluations utilizing characteristics of the trapezoidal beam profile, such as determining the slope and position of the edges and the height of the central plateau, and deriving the edge and center signals through geometric considerations.

[0069] In one embodiment, A1 may correspond to the entire or complete area of ​​the feature on the optical sensor. A2 may be the central area of ​​the feature on the optical sensor. The central area may be a constant value. The central area may be smaller than the total area of ​​the feature. For example, for a circular feature, the central area may have a radius between 0.1 and 0.9 of the total radius of the feature, preferably between 0.4 and 0.6 of the total radius.

[0070] In one embodiment, the illumination pattern may include at least one line pattern. A1 may correspond to an area having the full line width of the line pattern on the optical sensor, particularly on the light-sensitive area of ​​the optical sensor. The line pattern on the optical sensor may be enlarged and / or displaced compared to the line pattern of the illumination pattern, so that the line width on the optical sensor is amplified. In particular, in the case of a matrix of optical sensors, the line width of the line pattern on the optical sensor may vary from one column to another. A2 may be the central area of ​​the line pattern on the optical sensor. The line width of the central area may be a constant value, particularly corresponding to the line width of the illumination pattern. The line width of the central area may be smaller than the full line width. For example, the central area may have a line width of 0.1 to 0.9 of the full line width, preferably 0.4 to 0.6 of the full line width. The line pattern may be segmented on the optical sensor. Each column of the matrix of optical sensors may include center information of intensity in the central area of ​​the line pattern and edge information of intensity from areas extending further outward from the central area to the edge region of the line pattern.

[0071] In one embodiment, the illumination pattern may include at least one dot pattern. A1 may correspond to an area having a full radius of the dots of the dot pattern on the optical sensor. A2 may be a central area of ​​the dots of the dot pattern on the optical sensor. The central area may be a constant value. The central area may have a radius corresponding to the full radius. For example, the central area may have a radius between 0.1 and 0.9 of the full radius, preferably between 0.4 and 0.6 of the full radius.

[0072] The illumination pattern may include both at least one dot pattern and at least one line pattern. In addition to or as an alternative to line and dot patterns, other embodiments are possible.

[0073] The evaluation device may be configured to derive the quotient Q by one or more of dividing the first area and the second area, dividing the first area and a multiple of the second area, or dividing a linear combination of the first area and the second area.

number

[0074] Additionally or alternatively, the evaluation device may be adapted to determine one or both of center and edge information from at least one slice or cut of the light spot. This may be achieved, for example, by replacing the surface integral of the quotient Q with a line integral along the slice or cut. To improve accuracy, several slices or cuts through the light spot may be used and averaged. In the case of an elliptical spot profile, averaging over several slices or cuts may provide improved distance information.

[0075] For example, in the case of an optical sensor having a matrix of pixels, the evaluation device - determining the pixel with the highest sensor signal and forming at least one center signal; - evaluating the matrix of sensor signals and forming at least one sum signal; - determining a quotient Q by combining the center signal and the sum signal; - determining at least one longitudinal coordinate z of the object by evaluating the quotient Q; The beam profile may be evaluated by

[0076] As used herein, the term "sensor signal" generally refers to a signal generated by a light sensor and / or at least one pixel of the light sensor in response to illumination. Specifically, the sensor signal may be or include at least one electrical signal, such as at least one analog electrical signal and / or at least one digital electrical signal. More specifically, the sensor signal may be or include at least one voltage signal and / or at least one current signal. More specifically, the sensor signal may include at least one photocurrent. Furthermore, raw sensor signals may be used, or a display device, light sensor, or other element may be adapted to process or preprocess the sensor signal, thereby generating a secondary sensor signal that can also be used as the sensor signal. The term "center signal" generally refers to at least one sensor signal that contains substantially central information of the beam profile. As used herein, the term "highest sensor signal" refers to one or both of a local maximum or maximum value in a region of interest. For example, the central signal may be the signal of a pixel having the highest sensor signal among multiple sensor signals generated by pixels of the entire matrix or a region of interest within the matrix, which region of interest may be predetermined or determinable within the image generated by the pixels of the matrix. The central signal may originate from a single pixel or from a group of optical sensors; in the latter case, for example, the sensor signals of the group of pixels may be added, integrated, or averaged to determine the central signal. The group of pixels from which the central signal originates may be, for example, a group of neighboring pixels, such as pixels that are less than a predetermined distance away from the actual pixel with the highest sensor signal, or a group of pixels that generate sensor signals within a predetermined range from the highest sensor signal. The group of pixels from which the central signal originates may be selected to be as large as possible to allow for the largest dynamic range. The evaluation device may be adapted to determine the central signal by integrating multiple sensor signals, for example, multiple pixels around the pixel with the highest sensor signal.For example, the beam profile may be a trapezoidal beam profile and the evaluation device may be adapted to determine the integral of the trapezoid, in particular the integral of the plateau of the trapezoid.

[0077] As mentioned above, the center signal may generally be a single sensor signal, such as a sensor signal from a pixel at the center of the light spot, or may be a combination of multiple sensor signals, such as a combination of sensor signals arising from pixels at the center of the light spot, or may be a secondary sensor signal derived by processing sensor signals derived from one or more of the foregoing possibilities. The determination of the center signal may be performed electronically, as comparison of sensor signals is fairly easily performed by conventional electronics, or may be performed fully or partially by software. Specifically, the center signal may be selected from the group consisting of: the highest sensor signal; the average of a group of sensor signals that are within a predetermined tolerance of the highest sensor signal; the average of sensor signals from a group of pixels including the pixel with the highest sensor signal and a predetermined group of adjacent pixels; the sum of sensor signals from a group of pixels including the pixel with the highest sensor signal and a predetermined group of adjacent pixels; the sum of a group of sensor signals that are within a predetermined tolerance of the highest sensor signal; the average of a group of sensor signals that exceed a predetermined threshold; the sum of a group of sensor signals that exceed a predetermined threshold; the integral of sensor signals from a group of optical sensors including the optical sensor with the highest sensor signal and a predetermined group of adjacent pixels; the integral of a group of sensor signals that are within a predetermined tolerance of the highest sensor signal; the integral of a group of sensor signals that exceed a predetermined threshold.

[0078] Similarly, the term "sum signal" generally refers to a signal that contains substantially edge information of the beam profile. For example, the sum signal can be derived by summing, integrating, or averaging sensor signals for the entire matrix or for a region of interest within the matrix, where the region of interest is predetermined or determinable within the image generated by the optical sensors of the matrix. When summing, integrating, or averaging sensor signals, the actual optical sensors at which the sensor signals are generated may be excluded from the summation, integration, or averaging, or may be included in the summation, integration, or averaging. The evaluation device may be adapted to determine the sum signal by integrating signals for the entire matrix or for a region of interest within the matrix. For example, the beam profile may be a trapezoidal beam profile, and the evaluation device may be adapted to determine the integral of the entire trapezoid. Furthermore, when a trapezoidal beam profile is assumed, the determination of the edge and center signals can be replaced with an equivalent evaluation that utilizes characteristics of the trapezoidal beam profile, such as determining the slope and edge location and the height of the central plateau, and derives the edge and center signals through geometric considerations.

[0079] Similarly, the center signal and the edge signal can also be determined by using a segment of the beam profile, such as a circular segment of the beam profile. For example, the beam profile can be divided into two segments by a dividing line or chord that does not pass through the center of the beam profile. Thus, one segment will substantially contain edge information, and the other segment will substantially contain center information. For example, the edge signal can be further subtracted from the center signal to further reduce the amount of edge information in the center signal.

[0080] The quotient Q may be a signal generated by combining the center signal and the sum signal. Specifically, the determination may include one or more of: forming a quotient of the center signal and the sum signal, or its inverse; forming a quotient of a multiple of the center signal and a multiple of the sum signal, or its inverse; forming a quotient of a linear combination of the center signals and a linear combination of the sum signal, or its inverse. Additionally or alternatively, the quotient Q may include any signal or combination of signals that includes at least one item of information regarding the comparison between the center signal and the sum signal.

[0081] As used herein, the term "longitudinal coordinate of an object" refers to the distance between the optical sensor and the object. The evaluation device may be configured to use at least one predetermined relationship between the quotient Q and the longitudinal coordinate to determine the longitudinal coordinate. The predetermined relationship may be one or more of an empirical relationship, a semi-empirical relationship, and an analytically derived relationship. The evaluation device may include at least one data storage device for storing the predetermined relationship, such as a look-up list or a look-up table.

[0082] The evaluation unit may be configured to execute at least one depth from photon ratio algorithm that calculates distances for all zero order and higher order reflectance features.

[0083] Evaluating the first image includes sorting the identified reflective features with respect to brightness. As used herein, the term "sorting" may refer to assigning a sequence of reflective features for further evaluation with respect to brightness, particularly starting with the reflective feature with the greatest brightness, followed by reflective features with decreasing brightness. As used herein, the term "brightness" may refer to the magnitude of the reflective features in the first image and / or the intensity of the reflective features in the first image. Brightness may refer to a defined passband, such as the visible or infrared spectral range, or may be wavelength independent. Sorting by decreasing brightness may refer to sorting according to and / or with respect to decreasing brightness. If the brightest reflective features are prioritized for calculating the DPR, the vertical coordinate zDPR This can increase the robustness of the determination of , mainly because the reflective features with the zeroth order of the diffraction grating are always brighter than spurious features with higher orders.

[0084] The evaluation device is DPR The technique is configured to unambiguously match reflection features with corresponding illumination features by using depth from photon ratio techniques. The longitudinal coordinates determined using the depth from photon ratio technique can be used to solve the so-called correspondence problem. In that way, the distance information for each reflection feature can be used to find correspondences for a known laser projector grid. As used herein, the term "matching" refers to determining and / or evaluating corresponding illumination and reflection features. As used herein, the term "corresponding illumination and reflection features" can refer to the fact that each illumination feature of an illumination pattern generates a reflection feature in a scene, and the generated reflection feature is assigned to the illumination feature that generated it.

[0085] As used herein, the term "unequivocally matching" may refer to only one reflectance feature being assigned to one illumination feature and / or no other reflectance features being assigned to the same matching illumination feature.

[0086] Illumination features corresponding to reflection features can be determined using epipolar geometry. For a description of epipolar geometry, see, for example, Chapter 2 of X. Jiang and H. Bunke, "Dreidimensionales Computersehen," Springer, Berlin, Heidelberg, 1997. In epipolar geometry, it can be assumed that the illumination image, i.e., the undistorted illumination pattern, and the first image can be images determined at different spatial positions and / or spatial directions with a fixed distance. The distance can be a relative distance, also called a baseline. The illumination image can also be referred to as a reference image. The evaluation device can be adapted to determine an epipolar line within the reference image. The relative positions of the reference image and the first image can be known. For example, the relative positions of the reference image and the first image can be stored in at least one storage unit of the evaluation device. The evaluation device can be adapted to determine a straight line extending from a selected reflection feature in the first image to a real-world feature from which it emanates. Thus, the straight line can include a possible object feature corresponding to the selected reflection feature. The line and baseline define an epipolar plane. Because the reference image is determined at a different relative position from the first image, corresponding possible object features can be imaged on a line called an epipolar line in the reference image. The epipolar line can be the intersection of the epipolar plane and the reference image. Therefore, the feature in the reference image that corresponds to the selected feature in the first image lies on the epipolar line.

[0087] Depending on the distance to the object in the scene that reflected the illumination feature, the reflection feature corresponding to the illumination feature may be displaced in the first image. The reference image may include at least one displacement region where the illumination feature corresponding to the selected reflection feature will be imaged. The displacement region may include only one illumination feature. The displacement region may include multiple illumination features. The displacement region may include an epipolar line or a portion of an epipolar line. The displacement region may include multiple epipolar lines or multiple portions of multiple epipolar lines. The displacement region may extend along the epipolar line, or orthogonal to the epipolar line, or both. The evaluation device may be adapted to determine the illumination feature along the epipolar line. The evaluation device may be adapted to determine a vertical coordinate z and an error interval ±ε for the reflection feature from the combined signal Q to determine a displacement region along the epipolar line or a displacement region orthogonal to the epipolar line corresponding to z±ε. The measurement uncertainty of the distance measurement using the combined signal Q may result in a non-circular displacement region in the second image, since the measurement uncertainty may vary with direction. Specifically, the measurement uncertainty along one or more epipolar lines may be larger than the measurement uncertainty in a direction orthogonal to the one or more epipolar lines. The displacement region may include an extension in a direction orthogonal to the one or more epipolar lines. The evaluation device may be adapted to match the selected reflection feature to at least one illumination feature within the displacement region. The evaluation device may be adapted to match the determined vertical coordinate z DPRThe evaluation device may be adapted to match selected features of the first image with illumination features within the displacement region by using at least one evaluation algorithm that takes into account the above. The evaluation algorithm may be a linear scaling algorithm. The evaluation device may be adapted to determine an epipolar line closest to the displacement region and / or within the displacement region. The evaluation device may be adapted to determine an epipolar line closest to the image position of the reflection feature. The extent of the displacement region along the epipolar line may be greater than the extent of the displacement region perpendicular to the epipolar line. The evaluation device may be adapted to determine the epipolar line before determining the corresponding illumination feature. The evaluation device may determine the displacement region around the image position of each reflection feature. The evaluation device may be adapted to assign an epipolar line to each displacement region at each image position of the reflection feature, for example, by assigning the epipolar line closest to the displacement region and / or the epipolar line within the displacement region and / or the epipolar line closest to the displacement region along a direction perpendicular to the epipolar line. The evaluation device may be adapted to determine the illumination feature corresponding to the reflection feature by determining the illumination feature closest to the assigned displacement region and / or the illumination feature within the assigned displacement region and / or the illumination feature closest to the assigned displacement region along the assigned epipolar line and / or the illumination feature within the assigned displacement region along the assigned epipolar line.

[0088] Additionally or alternatively, the evaluation device may perform the following steps: - determining a displacement area for each reflection feature's image location; - assigning epipolar lines to the displacement region of each reflection feature, such as by assigning the epipolar line closest to the displacement region and / or the epipolar line within the displacement region and / or the epipolar line closest to the displacement region along a direction orthogonal to the epipolar line; - assigning and / or determining at least one illumination feature to each reflection feature, such as by assigning an illumination feature closest to the assigned displacement area and / or an illumination feature within the assigned displacement area and / or an illumination feature closest to the assigned displacement area along the assigned epipolar line and / or an illumination feature within the assigned displacement area along the assigned epipolar line; may be configured to perform the following.

[0089] Additionally or alternatively, the evaluation device may be adapted to determine among the illumination features assigned to a plurality of epipolar lines and / or reflection features, e.g. by comparing the distances of the reflection features and / or epipolar lines in the illumination image and / or by comparing the ε-weighted distances of the illumination features and / or the error-weighted distances of the epipolar lines in the illumination image, and assigning the epipolar line and / or illumination feature with the shorter distance and / or ε-weighted distance to the illumination feature and / or reflection feature.

[0090] As described above, the diffraction grating generates multiple reflection features, e.g., one true feature and multiple false features for each illumination feature. Matching is performed starting with the brightest reflection feature and proceeding by decreasing the brightness of the reflection features. Other reflection features cannot be assigned to the same matched illumination feature. Due to display artifacts, the generated false features are generally darker than the true features. By sorting the reflection features by brightness, brighter reflection features are prioritized for correspondence matching. If the illumination feature correspondence has already been used, the false feature cannot be assigned to the used, i.e., matched, illumination feature.

[0091] The evaluation device is configured to classify reflection features that match the illumination features as true features and to classify reflection features that do not match the illumination features as false features. As used herein, the term "classifying" may refer to assigning reflection features to at least one category. As used herein, the term "true feature" may refer to a zero-order reflection feature of a diffraction grating. As used herein, the term "false feature" may refer to a higher-order reflection feature of a diffraction grating, i.e., an order ≧1. The zero-order of a diffraction grating is always brighter than higher-order false features.

[0092] The evaluation device rejects false features and calculates the vertical coordinate z DPR The display device may be configured to generate a depth map for a true feature by using a 3D map of a scene, such as a face. As used herein, the term "depth" may refer to the distance between an object and a light sensor, and may be given by a vertical coordinate. As used herein, the term "depth map" may refer to the spatial distribution of depth. The display device may be used to generate a 3D map from a scene, such as a face.

[0093] Structured light methods typically use a camera and a projector with a fine point grid, e.g., several thousand points. A well-known projector pattern is used to find the correspondences of point patches on the scene. Distance information is obtained by triangulation once the point correspondences have been resolved. If the camera is behind the display, the image is spatially distorted due to diffraction. Therefore, finding the point pattern on the distorted image is a difficult task. Compared to structured light methods, this invention proposes to use a depth-to-photon ratio technique to evaluate the beam profile, which is not directly affected by the display's diffraction grating. Distortion does not affect the beam profile.

[0094] The depth map can be further refined by using additional depth measurement techniques such as triangulation and / or depth from defocus and / or structured light. The evaluation device uses triangulation and / or depth from defocus and / or structured light techniques to determine at least one second vertical coordinate z for each of the reflective features. triang may be configured to determine:

[0095] The evaluation device may be adapted to determine the displacement of the illumination feature and the reflection feature. The evaluation device may be adapted to determine the displacement of the matched illumination feature and the selected reflection feature. The evaluation device, e.g., at least one data processing unit of the evaluation device, may be configured to determine the displacement of the illumination feature and the reflection feature, particularly by comparing the respective image positions of the illumination image and the first image. As used herein, the term "displacement" refers to the difference between the image position in the illumination image and the image position in the first image. The evaluation device may be adapted to determine the second vertical coordinate of the matched feature by using a predetermined relationship between the second vertical coordinate and the displacement. The evaluation device may be adapted to determine the predetermined relationship by using a triangulation method. If the position of the selected reflection feature and the position of the matched illumination feature in the first image and / or the relative displacement of the selected reflection feature and the matched illumination feature are known, the vertical coordinate of the corresponding object feature can be determined by triangulation. Thus, the evaluation device may be adapted to select a reflection feature, e.g., sequentially and / or row by row, and to determine a corresponding distance value using triangulation for each potential position of the illumination feature. The displacements and corresponding distance values ​​may be stored in at least one storage device of the evaluation device. The evaluation device may, for example, include at least one data processing device, such as at least one processor, at least one DSP, at least one FPGA, and / or at least one ASIC. Furthermore, at least one data storage device may be provided to store at least one predetermined or determinable relationship between the second vertical coordinate z and the displacement, such as by providing one or more look-up tables for storing predetermined relationships. The evaluation device may be adapted to store parameters for intrinsic and / or extrinsic calibration of the camera and / or display device. The evaluation device may be adapted to generate parameters for intrinsic and / or extrinsic calibration of the camera and / or display device, such as by performing a Tsai camera calibration.The evaluation device may be adapted to calculate and / or estimate parameters such as the focal length of the transfer device, the distortion coefficient of the radiation lens, the coordinate of the center of distortion of the radiation lens, a scale factor to account for uncertainties due to imperfections in the scanning and digitization hardware timing, the rotation angle of the transformation between world coordinates and camera coordinates, the translation component of the transformation between world coordinates and camera coordinates, the aperture angle, the image sensor type, the principal point, the skew factor, the camera center, the camera orientation, the baseline, the rotation or translation parameters between the camera and / or the illumination source, the aperture, the focal length, etc.

[0096] The evaluation device is triang and the vertical coordinate z DPR The bond vertical coordinate may be determined by the second vertical coordinate z triang and the vertical coordinate z DPR The combined vertical coordinate can be used for determining the depth map.

[0097] The display device may include an additional illumination source. The additional illumination source may include at least one light-emitting diode (LED). The additional illumination source may be configured to generate light within the visual spectrum. The optical sensor may be configured to determine at least one second image including at least one two-dimensional image of the scene. The additional illumination source may be configured to provide additional illumination for imaging the second image. For example, the display device configuration can be extended with an additional flood illumination LED. The additional illumination source may illuminate a scene, such as a face, using an LED, particularly without an illumination pattern, and the optical sensor may be configured to capture a two-dimensional image. The two-dimensional image may be used for face detection and verification algorithms. If the impulse response of the display is known, the distorted image captured by the optical sensor can be restored. The evaluation device may be configured to determine at least one corrected image I0 by deconvolving the second image I (I = I0 * g) with a diffraction grating function g. The diffraction grating function is also called the impulse response. The undistorted image can be restored by a deconvolution approach, such as Van-Cittert or Wiener deconvolution.

number

[0098] The evaluation device may be configured to determine at least one material property m of the object by evaluating at least one beam profile of the reflective feature, preferably the beam profiles of a plurality of reflective features. For details of determining at least one material property m by evaluating the beam profile, see WO2020 / 187719, the contents of which are incorporated by reference.

[0099] As used herein, the term "material property" refers to at least one property of a material configured for characterization and / or identification and / or classification of the material. For example, the material property may be a property selected from the group consisting of roughness, penetration depth of light into the material, a property characterizing the material as a biological or non-biological material, reflectance, specular reflectance, diffuse reflectance, surface properties, a measure of translucency, scattering, particularly backscattering behavior, etc. The at least one material property may also be a property selected from the group consisting of scattering coefficient, translucency, transparency, deviation from Lambertian surface reflectance, speckle, etc. As used herein, the term "identifying at least one material property" refers to one or more of determining a material property and assigning a material property to an object. The evaluation device may include at least one database including a list and / or table, such as a look-up list and / or look-up table, of predefined and / or predetermined material properties. The list and / or table of material properties can be determined and / or generated by performing at least one test measurement using the display device according to the present invention, for example, by performing material testing using samples with known material properties. The lists and / or tables of material properties can be determined and / or generated at the manufacturer site and / or by a user of the display device. The material properties may be further assigned to a material classification such as, for example, one or more of: material name, material group such as biological or non-biological material, translucent or non-translucent material, metal or non-metal, skin or non-skin, fur or non-fur, carpet or non-carpet, reflective or non-reflective, specular or non-specular, foam or non-foam, hair or non-hair, roughness group, etc. The evaluation device may include at least one database including lists and / or tables containing material properties and associated material names and / or material groups.

[0100] For example, without wishing to be bound by this theory, human skin may have a reflectance profile, also referred to as an inverse scattering profile, that includes a portion generated by retroreflection from the surface, referred to as surface reflectance, and a portion generated by highly diffuse reflection from light penetrating the skin, referred to as the diffuse portion of retroreflection. For the reflectance profile of human skin, see "Lasertechnikinder Medizin: Grundlagen, Systeme, Anwendungen," "Wirkung von Laserstrahlung auf Gewebe," 1991, pp. 10171-266, Jürgen Eichler, Theo Seiler, Springer Verlag, ISBN 0939-0979. The surface reflectance of skin may increase as the wavelength increases toward the near infrared. Furthermore, the penetration depth may increase as the wavelength increases from visible light toward the near infrared. The diffuse portion of retroreflection may increase with increasing light penetration depth. These characteristics can be used to distinguish skin from other materials by analyzing the inverse scattering profile.

[0101] Specifically, the evaluation device may be configured to compare the beam profile of the reflection feature (also referred to as the reflected beam profile) with at least one predetermined and / or pre-recorded and / or pre-defined beam profile. The predetermined and / or pre-recorded and / or pre-defined beam profile may be stored in a table or a look-up table, for example, empirically determined, and may be stored in at least one data storage device of the display device, for example. For example, the predetermined and / or pre-recorded and / or pre-defined beam profile may be determined upon initial startup of a mobile device including the display device. For example, the predetermined and / or pre-recorded and / or pre-defined beam profile may be stored in at least one data storage device of the mobile device, for example, by software, specifically an app downloaded from an app store, etc. The reflection feature may be identified as being generated by biological tissue if the reflected beam profile and the predetermined and / or pre-recorded and / or pre-defined beam profile are identical. The comparison may include overlaying the reflected beam profile and the predetermined or pre-defined beam profile so that their centers of intensity match. The comparison may include determining a deviation, e.g., a sum of squares of inter-point distances, between the reflected beam profile and a predetermined and / or pre-recorded and / or pre-defined beam profile. The evaluation device may be configured to compare the determined deviation with at least one threshold value, and if the determined deviation is lower than and / or equal to the threshold value, the surface is indicated as biological tissue and / or the detection of biological tissue is confirmed. The threshold value may be stored in a table or look-up table, may be determined, e.g., empirically, and may, for example, be stored in at least one data storage device of the display device.

[0102] Additionally or alternatively, to identify whether a reflection feature is generated by biological tissue, the evaluation device can be configured to apply at least one image filter to the image of the area. As further used herein, the term "image" refers to a two-dimensional function f(x,y), where brightness and / or color values ​​are given for any x,y position in the image. The positions may be discretized corresponding to the recording pixels. Furthermore, the brightness and / or color may be discretized corresponding to the bit depth of the optical sensor. As used herein, the term "image filter" refers to at least one mathematical operation applied to the beam profile and / or at least one specific region of the beam profile. Specifically, the image filter Φ maps the image f or a region of interest in the image to a real number Φ(f(x,y))=φ, where φ indicates a feature, particularly a material feature. Images may be affected by noise, as may features. Therefore, features may be random variables. Features may follow a normal distribution. If the features do not follow a normal distribution, they may be transformed to have a normal distribution by, for example, Box-Cox transformation.

[0103] The evaluation device determines at least one material characteristic φ by applying at least one material-dependent image filter Φ2 to the image. 2m As used herein, the term "material dependent" image filter refers to an image having a material dependent output. The output of a material dependent image filter is referred to herein as a "material feature φ 2m " or "material-dependent feature φ 2m The material signature may be or may include at least one piece of information regarding at least one material property of the surface of the area that produced the reflection signature.

[0104] a gray-level generation-based contrast filter; a gray-level generation-based energy filter; a gray-level generation-based uniformity filter; a gray-level generation-based dissimilarity filter; a low-energy filter; a threshold domain filter; or a linear combination thereof; or a further material-dependent image filter Φ correlated to one or more of the brightness filter, spot-shape filter, square-norm gradient, standard deviation, smoothness filter, gray-level generation-based energy filter, gray-level generation-based uniformity filter, gray-level generation-based dissimilarity filter, low-energy filter, or threshold domain filter. 2other , or |ρ Ф2other,Фm their linear combination with | ≥ 0.40, where Φ m can be one or more of a brightness filter, a spot shape filter, a squared norm gradient, a standard deviation, a smoothness filter, a gray level generation based energy filter, a gray level generation based uniformity filter, a gray level generation based dissimilarity filter, a low energy filter, or a threshold area filter, or a linear combination thereof. 2other is the material-dependent image filter Ф m and one or more of |ρ Ф2other,Фm |≧0.60, preferably |ρ Ф2other,Фm may be correlated by |≧0.80.

[0105] The material-dependent image filter may be at least one arbitrary filter Φ that passes hypothesis validation. As used herein, the term "passes hypothesis validation" refers to the fact that the null hypothesis H0 is rejected and an alternative hypothesis H1 is accepted. The hypothesis validation may include validating the material dependency of the image filter by applying the image filter to a predefined data set. The data set may include multiple beam profile images. As used herein, the term "beam profile image" refers to NB refers to the sum of Gaussian radial basis functions,

number

[0106]

number

[0107]

number

number

[0108] [Table 1]

[0109]

number

number

[0110]

number

number

number

[0111] Next, each image

number

[0112]

number

number

number

number

number

number

number

[0113]

number

number

number

number

number

number

number

number

[0114] Hypothesis testing may involve determining the mean sum of squares between:

[0115]

number

[0116] Hypothesis testing may include performing an F-test.

[0117]

number

[0118] where:

number

number

number

[0119] In the following, the image filter is described assuming that the reflection image includes at least one reflection feature, in particular a spot image.

number

number

[0120] For example, the material dependent image filter may be a luminance filter, which may return a luminance measurement of a spot as a material characteristic.

number

number

number

number

[0121] For example, a material dependent image filter may be a filter whose output depends on the spot shape. The material dependent image filter may return a value as a material characteristic that correlates with the translucency of the material. The translucency of the material affects the shape of the spot. The material characteristic may be:

number

number

number

number

number

[0122] For example, the material dependent image filter may be a squared norm gradient, which may return a value that correlates to a measure of the soft-hard transition and / or roughness of the spot as a material characteristic.

number

number

number

[0123] For example, the material-dependent image filter may be a smoothness filter, such as a Gaussian filter or a median filter. In one embodiment of the smoothness filter, the image filter may refer to the observation that volumetric scattering exhibits less speckle contrast compared to diffuse scattering materials. The image filter may quantify the smoothness of spots corresponding to the speckle contrast as a material feature. The material feature may be:

number

number

number

number

[0124]

number

number

number

number

number

[0125] For example, the image filter may be a gray level generation based contrast filter. The material filter may be a gray level generation matrix

number

number

[0126] The material characteristics of the gray level generation based contrast filter are:

number

[0127] For example, the image filter may be a gray level generation based energy filter, which is based on the gray level generation matrix defined above.

[0128] The material characteristics of the gray level generation based energy filter are:

number

[0129] For example, the image filter may be a gray level generation based uniformity filter, where the material filter is based on the gray level generation matrix defined above. The material characteristics of the gray level generation based uniformity filter are:

number

[0130] For example, the image filter may be a gray level generation-based dissimilarity filter, which is based on the gray level generation matrix defined above. The material features of the gray level generation-based dissimilarity filter are:

number

[0131] For example, the image filter may be a low energy filter, where the material filter is a low vector L5=[1,4,6,4,1] and E5=[-1,-2,0,-2,-1] and material L5(E5) T and E5(L5) T Based on image f k These are the matrices:

number

[0132] where the material characteristics of the low energy filter are:

number

[0133] For example, the material-dependent image filter may be a threshold domain filter. This material feature may relate to two areas in the image plane. The first area, Ω1, may be the area where the function f is greater than α times the maximum value of f. The second area, Ω2, may be the area where the function f is less than α times the maximum value of f but greater than a threshold of ε times the maximum value of f. Preferably, α may be 0.5 and ε may be 0.05. Due to speckle or noise, the areas may not simply correspond to inner and outer circles around the spot center. For example, Ω1 may include speckle or disconnected areas in the outer circle. The material feature may be

number

[0134] The evaluation device calculates the material characteristic φ to determine the material properties of the surface that generated the reflection characteristic. 2m and a material property of the surface that produced the reflection feature. The predetermined relationship may be one or more of an empirical relationship, a semi-empirical relationship, and an analytically derived relationship. The evaluation device may comprise at least one data storage device for storing the predetermined relationship, such as a look-up list or a look-up table.

[0135] The evaluation device is configured to identify a reflection feature as having been generated by irradiating biological tissue if the corresponding material characteristic satisfies at least one predetermined or predefined criterion. A reflection feature can be identified as having been generated by biological tissue if the material characteristic indicates "biological tissue." A reflection feature may be identified as having been generated by biological tissue if the material characteristic is below or equal to at least one threshold or range, and a reflection feature is identified as having been generated by biological tissue and / or detection of biological tissue is confirmed if the determined deviation is below and / or equal to the threshold. The at least one threshold and / or range may be stored in a table or look-up table, for example, or may be empirically determined, and the at least one threshold and / or range may be stored in at least one data storage device of the display device, for example. The evaluation device is configured to otherwise identify the reflection feature as being background. Thus, the evaluation device may be configured to assign depth information and a material characteristic, for example, "yes" or "no" for skin, to each projection spot.

[0136] The material properties are given by the information about the vertical coordinate z in terms of φ 2m After determining the vertical coordinate z, φ 2m It may be determined by evaluating

[0137] In a further aspect, the present invention discloses a method for depth measurement through a display device, in which a display device according to the present invention is used. The method comprises: a) projecting at least one illumination pattern comprising a plurality of illumination features onto at least one scene by using at least one illumination source, the illumination source being positioned in front of a display in a propagation direction of the illumination pattern; b) determining at least one first image comprising a plurality of reflective features produced by the scene in response to illumination by the illumination features by using at least one optical sensor, the optical sensor having at least one light-sensitive area, the optical sensor being positioned in front of the display in a propagation direction of the illumination pattern, and each of the reflective features comprising at least one beam profile; c) evaluating said first image by using at least one evaluation device, said evaluation comprising the following sub-steps: C1) identifying the reflective features of the first image and sorting the identified reflective features with respect to brightness; C2) by analyzing the beam profile, at least one vertical coordinate z for each of said reflection features is obtained. DPR and the substep of determining; C3) The vertical coordinate z DPR a sub-step of unambiguously matching reflective features to corresponding illumination features by using a reflective feature vector, said matching being performed starting from the brightest reflective feature and decreasing the brightness of said reflective features; C4) classifying the reflection features that match the illumination features as true features and the reflection features that do not match the illumination features as false features; C5) Rejecting false features and determining the vertical coordinate z DPR and generating a depth map of the true features by using The method includes the steps of:

[0138] The method steps may be performed in a predetermined order or in a different order. Furthermore, one or more additional method steps not listed may be present. Furthermore, one, more than one, or even all of the method steps may be performed repeatedly. For details, options, and definitions, reference may be made to the display device described above. Thus, in particular, as described above, the method may include use of a display device according to the present invention, such as according to one or more embodiments given above or given in more detail below.

[0139] The at least one evaluation device may be configured to execute at least one computer program, such as at least one computer program configured to perform or support one or more or even all of the method steps of the method according to the invention. As an example, one or more algorithms for determining the position of the object may be implemented.

[0140] In a further aspect of the present invention, the use of a display device according to the present invention, such as according to one or more embodiments given above or in more detail below, is proposed for use purposes selected from the group consisting of: position determination in traffic technology; entertainment applications; security applications; surveillance applications; safety applications; human-machine interface applications; tracking applications; photography applications; imaging or camera applications; mapping applications for generating at least one spatial map; homing or tracking beacon detectors for vehicles; outdoor applications; mobile applications; communication applications; machine vision applications; robotics applications; quality control applications; manufacturing applications.

[0141] For further uses of the display device and device of the present invention, reference is made to WO2018 / 091649A1, WO2018 / 091638A1 and WO2018 / 091640A1, the contents of which are incorporated by reference.

[0142] Overall, in the context of the present invention, the following embodiments are considered preferred.

[0143] Embodiment 1: - at least one illumination source configured to project at least one illumination pattern including a plurality of illumination features onto at least one scene; - at least one light sensor having at least one light-sensitive area, the light sensor configured to determine at least one first image comprising a plurality of reflective features produced by the scene in response to illumination by the illumination feature; and - at least one light-transmitting display configured to display information, wherein the illumination source and the light sensor are arranged in front of the display in a direction of propagation of the illumination pattern; at least one evaluation device, the evaluation device configured to evaluate the first image, wherein the evaluation of the first image comprises identifying the reflection features of the first image and sorting the identified reflection features with respect to brightness, each of the reflection features comprising at least one beam profile, and the evaluation device determining at least one vertical coordinate z for each of the reflection features by analysis of the beam profiles; DPR and the evaluation device is configured to determine the vertical coordinate z DPR and the evaluation device is configured to unambiguously match reflection features to corresponding illumination features by using a vertical coordinate z DPR at least one evaluation unit configured to generate a depth map for said true features using A display device comprising:

[0144] Embodiment 2: The evaluation device uses triangulation and / or depth from defocus and / or structured light techniques to determine at least one second vertical coordinate z for each of the reflective features. triang 2. A display device according to the preceding embodiment, configured to determine:

[0145] Embodiment 3: The evaluation device is configured to triang and the vertical coordinate z DPR and the bond vertical coordinate is determined by the second vertical coordinate z triang and the vertical coordinate z DPR , and said combined vertical coordinate is used for determining said depth map.

[0146] Embodiment 4: A display device according to any one of the preceding embodiments, wherein the illumination source comprises at least one laser projector, the laser projector including at least one laser source and at least one diffractive optical element (DOE).

[0147] Embodiment 5: A display device according to any one of the preceding embodiments, wherein the illumination source is configured to generate at least one light beam having a beam path passing from the illumination source through the display to the scene, and the display is configured to function as a grating such that the light beam is diffracted by the display, resulting in a dot pattern.

[0148] Embodiment 6: A display device according to the preceding embodiment, wherein wiring of the display is configured to form gaps and / or slits and ridges of the grating.

[0149] Embodiment 7: A display device according to any one of the preceding embodiments, wherein the illumination pattern comprises a periodic dot pattern.

[0150] Embodiment 8: A display device according to any one of the preceding embodiments, wherein the illumination pattern has a low point density, the illumination pattern having no more than 2500 points per field of view.

[0151] Embodiment 9: A display device according to any one of the preceding embodiments, wherein the evaluation device is configured to determine the beam profile information for each of the reflective features by using a depth from photon ratio technique.

[0152] Embodiment 10: A display device according to any one of the preceding embodiments, wherein the light sensor includes at least one CMOS sensor.

[0153] Embodiment 11: A display device according to any one of the preceding embodiments, wherein the display device comprises a further illumination source, the further illumination source comprising at least one light emitting diode (LED).

[0154] Embodiment 12: A display device according to the preceding embodiment, wherein the further illumination source is configured to generate light within the visual spectrum.

[0155] Embodiment 13: A display device according to any one of the preceding two embodiments, wherein the optical sensor is configured to determine at least one second image comprising at least one two-dimensional image of the scene, and the further illumination source is configured to provide additional illumination for imaging of the second image.

[0156] Embodiment 14: A display device according to the preceding embodiment, wherein the evaluation device is configured to determine at least one corrected image I0 by deconvolving the second image I with a diffraction grating function g, where I=I0*g.

[0157] Embodiment 15: A method for depth measurement through a light-transmitting display, wherein at least one display device according to any one of the preceding embodiments is used, comprising the following steps: a) projecting at least one illumination pattern comprising a plurality of illumination features onto at least one scene by using at least one illumination source, the illumination source being positioned in front of a display in a propagation direction of the illumination pattern; b) determining at least one first image comprising a plurality of reflective features produced by the scene in response to illumination by the illumination features by using at least one optical sensor, the optical sensor having at least one light-sensitive area, the optical sensor being positioned in front of the display in a propagation direction of the illumination pattern, and each of the reflective features comprising at least one beam profile; c) evaluating said first image by using at least one evaluation device, said evaluation comprising the following sub-steps: C1) identifying the reflective features of the first image and sorting the identified reflective features with respect to brightness; C2) by analyzing the beam profile, at least one vertical coordinate z for each of said reflection features is obtained. DPR and the substep of determining; C3) The vertical coordinate z DPR a sub-step of unambiguously matching reflective features to corresponding illumination features by using a reflective feature vector, said matching being performed starting from the brightest reflective feature and decreasing the brightness of said reflective features; C4) classifying the reflection features that match the illumination features as true features and the reflection features that do not match the illumination features as false features; C5) Rejecting false features and determining the vertical coordinate z DPR generating a depth map of said true features by using and 1. A method for depth measurement via a display device, comprising:

[0158] Embodiment 16: Use of a display device according to any one of the preceding embodiments relating to a display device, wherein the purpose of use is selected from the group consisting of: position measurement in traffic technology; entertainment applications; security applications; surveillance applications; safety applications; human-machine interface applications; tracking applications; photography applications; imaging or camera applications; mapping applications for generating maps of at least one space; homing or tracking beacon detectors for vehicles; outdoor applications; mobile applications; communication applications; machine vision applications; robotics applications; quality control applications; manufacturing applications. [Brief explanation of the drawings]

[0159] Further optional details and features of the invention are apparent from the following description of preferred exemplary embodiments in conjunction with the dependent claims. In this context, certain features may be implemented individually or in combination with other features. The invention is not limited to the exemplary embodiments. The exemplary embodiments are shown diagrammatically in the figures. The same reference numerals in the individual figures refer to identical elements or elements with the same function or elements which correspond to each other in terms of their function.

[0160] Specifically, in the diagram below: [Figure 1] 1A and 1B are diagrams illustrating an embodiment of a display device according to the present invention. [Figure 2] 2A-2B show an embodiment of a first image determined by at least one optical sensor of a display device. [Figure 3] 3A-3C show further embodiments of a first image determined by at least one optical sensor of a display device. [Figure 4] FIG. 1 illustrates the determination of a corrected 2D image using a display device. [Figure 5]5A-5C show a distorted 2D image captured with a display, a 2D image captured without a display, and a corrected 2D image. Modes for carrying out the invention

[0161] Detailed Description of the Embodiments 1A highly schematically illustrates one embodiment of a display device 110 according to the present invention. The display device 110 has at least one light-transmissive display 112 configured to display information. The display 112 can be any shape device configured to display information items such as at least one image, at least one diagram, at least one histogram, at least one text, at least one symbol, etc. The display 112 can be at least one monitor or at least one screen. The display 112 can have any shape, preferably a rectangular shape. For example, the display device 110 can be at least one device selected from the group consisting of a television set, a smartphone, a game console, a personal computer, a laptop, a tablet, at least one virtual reality device, or a combination thereof.

[0162] The display device 110 includes at least one illumination source 114 configured to project at least one illumination pattern including a plurality of illumination features onto at least one scene. The scene may refer to an object, such as a face, or a region of space. The scene may include at least one object and its surrounding environment.

[0163] The illumination source 114 may be adapted to directly or indirectly illuminate the scene, where the illumination pattern is reflected or scattered by surfaces in the scene and thereby directed at least in part to the light sensor. The illumination source 114 may be adapted to illuminate the scene, for example, by directing a light beam towards the scene, where the scene reflects the light beam. The illumination source 114 may be configured to generate an illumination light beam for illuminating the scene.

[0164] The illumination source 114 may include at least one light source. Alternatively, the illumination source 114 may include multiple light sources. The illumination source 114 may include an artificial illumination source, particularly at least one laser source, and / or at least one incandescent lamp, and / or at least one semiconductor light source, such as at least one light-emitting diode, particularly an organic and / or inorganic light-emitting diode. By way of example, the light emitted by the illumination source may have a wavelength of 300 to 1100 nm, particularly 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range, such as the range of 780 nm to 3.0 μm, may be used. Specifically, light in the near-infrared range, particularly the range of 700 nm to 1100 nm, to which silicon photodiodes are applicable, may be used. The illumination source 114 may be configured to generate at least one illumination pattern in the infrared range. Using light in the near-infrared range allows light that is not or only weakly detectable by the human eye but can be detected by silicon sensors, particularly standard silicon sensors. The illumination source 114 can be configured to emit light at a single wavelength. Specifically, the wavelength may be in the near-infrared region. In other embodiments, the illumination may be adapted to emit light having multiple wavelengths, allowing for additional measurements in other wavelength channels.

[0165] The illumination source 114 may be or may include at least one multi-beam light source. For example, the illumination source 114 may include at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the illumination source 114 may comprise at least one laser and / or laser source. Various types of lasers may be employed, such as semiconductor lasers, double heterostructure lasers, external cavity lasers, separate confinement heterostructure lasers, quantum cascade lasers, distributed Bragg reflector lasers, polariton lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, indium arsenide lasers, transistor lasers, diode-pumped lasers, distributed feedback lasers, quantum well lasers, interband cascade lasers, gallium arsenide lasers, semiconductor ring lasers, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alternatively, non-laser light sources, such as LEDs and / or light bulbs, may be used. The illumination source may include one or more diffractive optical elements (DOEs) adapted to generate an illumination pattern. For example, the illumination source 114 may be adapted to generate and / or project a point cloud, and may include one or more of the following: at least one digital light processing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one array of light-emitting diodes; or at least one array of laser light sources. Considering their generally defined beam profile and other characteristics of ease of handling, the use of at least one laser source as the illumination source 114 is particularly preferred. The illumination source 114 may be integrated into the housing 116 of the display device 110.

[0166] Furthermore, the illumination source 114 may be configured to emit modulated or unmodulated light. When multiple illumination sources 114 are used, the different illumination sources may have different modulation frequencies, which may then be used to distinguish between the light beams, as outlined in more detail below.

[0167] The illumination pattern may be at least one arbitrary pattern including at least one illumination feature adapted to illuminate at least a portion of a scene. The illumination pattern may include a single illumination feature. The illumination pattern may include multiple illumination features. The illumination pattern may be selected from the group consisting of at least one dot pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; and at least one pattern including a periodic or aperiodic feature arrangement. The illumination pattern may include a regular and / or regular and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or even a convex tiling pattern. The illumination pattern may exhibit at least one illumination feature selected from the group consisting of at least one dot; at least one line; at least two lines, such as parallel or intersecting lines; at least one dot and one line; at least one periodic or aperiodic feature arrangement; and at least one arbitrary shaped feature. The illumination pattern may include at least one pattern selected from the group consisting of: at least one dot pattern, particularly a pseudorandom dot pattern; a random dot pattern or a quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern including at least one known feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern; at least one pattern including a convex uniform tiling; at least one line pattern including at least one line; or at least one line pattern including at least two lines, such as parallel or intersecting lines. For example, the illumination source may be adapted to generate and / or project a point cloud. The illumination source 114 may include at least one light projector adapted to generate a point cloud so that the illumination pattern can include multiple point patterns. The illumination source 114 may include at least one mask adapted to generate the illumination pattern from at least one light beam generated by the illumination source 114.

[0168] The distance between two features of the illumination pattern and / or the area of ​​at least one illumination feature may depend on the circle of confusion in the image. As outlined above, the illumination source may include at least one light source configured to generate at least one illumination pattern. Specifically, the illumination source 114 includes at least one laser source and / or at least one laser diode designated to generate laser radiation. The illumination source 114 may include at least one diffractive optical element (DOE). The display device 110 may include at least one laser source and at least one point projector, such as a DOE, adapted to project at least one periodic point pattern. For example, the projected illumination pattern may be a periodic point pattern. The projected illumination pattern may have a low point density. For example, the illumination pattern may include at least one periodic point pattern with a low point density, the illumination pattern having 2500 points or less per field of view. Compared to structured light, which typically has a point density of 10k-30k in a 55 x 38° field of view, the illumination pattern according to the present invention can be less dense, which allows the proposed technology to increase the power per point so that it is less dependent on ambient light compared to structured light.

[0169] The display device 110 has at least one light sensor 118 having at least one light-sensitive area 120. The light sensor 118 is configured to determine at least one first image 122, e.g., as shown in Figures 2A-2C and 3A-3C, including a plurality of reflective features produced by a scene in response to illumination by the illumination feature. The display device 110 may include a single camera having the light sensor 118. The display device 110 may include multiple cameras, each including the light sensor 118 or multiple light sensors 118.

[0170] The photosensor 118 may specifically be or include at least one photodetector, preferably an inorganic photodetector, more preferably an inorganic semiconductor photodetector, and most preferably a silicon photodetector. Specifically, the photosensor 118 may be sensitive in the infrared spectral range. All pixels of the matrix, or at least a group of photosensors of the matrix, may specifically be identical. Groups of identical pixels of the matrix may specifically be provided for different spectral ranges, or all pixels may be identical in terms of spectral sensitivity. Furthermore, the pixels may be identical in terms of size and / or their electronic or optoelectronic properties. Specifically, the photosensor 118 may be or include at least an inorganic photodiode sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers. Specifically, the photosensor 118 may be sensitive in the part of the near-infrared range, particularly in the range of 700 nm to 1100 nm, where silicon photodiodes are applicable. The infrared light sensor that can be used in the light sensor may be a commercially available infrared light sensor, such as the infrared light sensor sold under the brand name Hertzstueck® by trinamiX GmbH, D-67056 Ludwigshafen am Rhein, Germany. Thus, by way of example, the light sensor 118 may include at least one light sensor of an intrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of a Ge photodiode, an InGaAs photodiode, an extended InGaAs photodiode, an InAs photodiode, an InSb photodiode, and an HgCdTe photodiode. Additionally or alternatively, the light sensor 118 may include at least one light sensor of an extrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of a Ge:Au photodiode, a Ge:Hg photodiode, a Ge:Cu photodiode, a Ge:Zn photodiode, a Si:Ga photodiode, and a Si:As photodiode.Additionally or alternatively, the optical sensor 118 may include at least one photoconductive sensor, such as a PbS or PbSe sensor, a bolometer, preferably a bolometer selected from the group consisting of a VO bolometer and an amorphous Si bolometer.

[0171] The optical sensors 118 may be sensitive in one or more of the ultraviolet, visible, or infrared spectral ranges. Specifically, the optical sensors may be sensitive in the visible spectral range of 500 nm to 780 nm, most preferably 650 nm to 750 nm, or 690 nm to 700 nm. Specifically, the optical sensors 118 may be sensitive in the near-infrared range. Specifically, the optical sensors 118 may be sensitive in the near-infrared range, particularly in the range of 700 nm to 1000 nm, where silicon photodiodes are applicable. Specifically, the optical sensors may be sensitive in the infrared spectral range, specifically in the range of 780 nm to 3.0 μm. For example, the optical sensors may each independently be or include at least one element selected from the group consisting of a photodiode, a photocell, a photoconductor, a phototransistor, or any combination thereof. For example, the light sensor 118 may be or include at least one element selected from the group consisting of a CCD sensor element, a CMOS sensor element, a photodiode, a photocell, a photoconductor, a phototransistor, or any combination thereof. Any other type of photosensitive element may also be used. The photosensitive element may generally be made entirely or partially of inorganic materials and / or may be made entirely or partially of organic materials. Most commonly, one or more photodiodes, such as commercially available photodiodes, e.g., inorganic semiconductor photodiodes, may be used.

[0172] The light sensor 118 may have at least one sensor element having a matrix of pixels. Thus, by way of example, the light sensor 118 may be part of or constitute a pixelated optical device. For example, the light sensor 118 may be and / or include at least one CCD and / or CMOS device. By way of example, the light sensor 118 may be part of or constitute at least one CCD and / or CMOS device having a matrix of pixels, each pixel forming a light-sensitive area. The sensor element may be formed as a single, integral device or as a combination of several devices. The matrix may specifically be or include a rectangular matrix having one or more rows and one or more columns. The rows and columns may specifically be arranged in a rectangular manner. However, other arrangements, such as non-rectangular arrangements, are also possible. By way of example, a circular arrangement is also possible, in which the elements are arranged in concentric circles or ellipses around a central point. For example, the matrix may be a single row of pixels. Other arrangements are also possible.

[0173] The pixels of the matrix may be identical in one or more of size, sensitivity, and other optical, electrical, and mechanical properties. The photosensitive areas 120 of all photosensors 118 of the matrix may be arranged in a common plane, which preferably faces the scene so that a light beam propagating from an object to the display device 110 generates a light spot on the common plane. The photosensitive areas 120 may be located on the surface of each photosensor 118. However, other embodiments are also possible. The photosensors 118 may include, for example, at least one CCD and / or CMOS device. By way of example, the photosensors 118 may be part of or constitute a pixelated optical device. By way of example, the photosensors 118 may be part of or constitute at least one CCD and / or CMOS device having a matrix of pixels, each pixel forming a photosensitive area 120.

[0174] The display device 110 includes at least one translucent display 112 configured to display information. An illumination source 114 and a light sensor 118 are positioned in front of the display 112 in the direction of propagation of the illumination pattern. The illumination source 114 and the light sensor 118 may be positioned at fixed positions relative to each other. For example, the display device 110 configuration can include a laser projector as the illumination source 114, a camera with a light sensor 118 and a lens system, and the laser projector and camera may be fixed behind the translucent display in the direction of propagation of light reflected by the scene. The laser projector may generate a dot pattern and project it through the display 112. The camera can then view through the display. However, a configuration of the illumination source 114 and the light sensor 118 positioned behind the translucent display in the direction of propagation of light reflected by the scene may result in the diffraction grating of the display 112 generating multiple laser points on the scene and the first image. As a result, these multiple spots on the first image may not contain useful distance information. The display device 110 includes at least one evaluation device 124. The evaluation device 124 may be configured to find and evaluate the zero-order reflection features of the diffraction grating, i.e., true features, and can ignore higher order reflection features, i.e., false features.

[0175] The evaluation device 124 is configured to evaluate the first image. The evaluation device 124 can include at least one data processing device, more preferably by using at least one processor and / or at least one application-specific integrated circuit. Thus, by way of example, the at least one evaluation device 124 may include at least one data processing device having software code stored thereon, including a number of computer commands. The evaluation device may provide one or more hardware elements for performing one or more specified operations and / or one or more processors having software running thereon for performing one or more specified operations. The operations include evaluating the image. Specifically, determining the beam profile and displaying the surface may be performed by the at least one evaluation device. Thus, by way of example, one or more instructions can be implemented in software and / or hardware. Thus, by way of example, the evaluation device 124 may be comprised of one or more computers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), or one or more programmable devices, such as field-programmable gate arrays (FPGAs), configured to perform the above-mentioned evaluations. However, additionally or alternatively, the evaluation device may also be embodied completely or partly in hardware.

[0176] Evaluating the first image includes identifying reflective features in the first image. The evaluator 124 may be configured to perform at least one image analysis and / or image processing to identify the reflective features. The image analysis and / or image processing may use at least one feature detection algorithm. The image analysis and / or image processing may include: filtering; selecting at least one region of interest; forming a difference image between an image generated by the sensor signal and at least one offset; inverting the sensor signal by inverting the image generated by the sensor signal; forming a difference image between images generated by the sensor signal at different times; background correction; decomposition into color channels; decomposition into hue; saturation; luminance channel; frequency decomposition; singular value decomposition; applying a blob detector; applying a corner detector; applying a determinant of a Hessian filter; applying a principal curvature-based region detector; applying a maximum stable extremum region detector; applying a generalized Hough transform; applying a ridge detector; applying an affine-invariant feature detector; applying an affine-adaptive interest point operator; applying a Harris affine region detector; applying a Hessian affine region detector; and applying a scale-invariant feature transform. The method may include one or more of: applying a scale-space extremum detector; applying a local feature detector; applying a fast robust feature algorithm; applying a histogram of gradient position and orientation algorithm; applying a histogram of oriented gradient descriptors; applying a Deriche edge detector; applying a differential edge detector; applying a space-time interest point detector; applying a Moravec corner detector; applying a Canny edge detector; applying a Laplace operator of a Gaussian filter; applying a differential Gaussian filter; applying a Sobel operator; applying a Laplace operator; applying a Schall operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying a Radon transform; applying a Hough transform; applying a wavelet transform; thresholding; or generating a binary image. The region of interest may be determined manually by a user or automatically, such as by recognizing features in an image generated by an optical sensor.

[0177] For example, the illumination source 114 may be configured to generate and / or project a point cloud such that a plurality of illumination areas are generated on the optical sensor 118, e.g., a CMOS detector. Furthermore, disturbances, e.g., due to speckle and / or extraneous light and / or multiple reflections, may be present at the optical sensor 118. The evaluation unit 124 may be adapted to determine at least one region of interest, e.g., one or more pixels illuminated by the light beam, which are used to determine the longitudinal coordinate of the object. For example, the evaluation unit 124 may be adapted to perform filtering methods, e.g., blob analysis and / or edge filters and / or object recognition methods.

[0178] The evaluation unit 124 may be configured to perform at least one image correction, which may include at least one background subtraction, and may be adapted to remove background light contributions from the beam profile, for example by imaging without further illumination.

[0179] Each of the reflective features includes at least one beam profile, which may be selected from the group consisting of a trapezoidal beam profile, a triangular beam profile, a conical beam profile, and a linear combination of Gaussian beam profiles. The evaluation device is configured to determine beam profile information for each of the reflective features by analyzing the beam profiles.

[0180] The evaluation device 124 determines at least one vertical coordinate z of each reflection feature by analyzing the beam profile. DPRThe evaluation unit 124 is configured to determine the beam profile. For example, the analysis of the beam profile may include at least one of a histogram analysis step, calculation of difference measurements, application of a neural network, and application of a machine learning algorithm. The evaluation unit 124 may be configured to symmetrize and / or normalize and / or filter the beam profile to remove noise or asymmetries from the recording, especially recordings at larger angles, edges, etc. The evaluation unit 124 may filter the beam profile by removing high spatial frequencies, such as by spatial frequency analysis and / or median filtering. Aggregation may be performed by the center of the intensity of the light spot, averaging all intensities at the same distance to the center. The evaluation unit 124 may be configured to normalize the beam profile to a maximum intensity, especially to take into account intensity differences due to the recorded distance. The evaluation unit 124 may also be configured to remove the influence of background light from the beam profile, for example by imaging without illumination.

[0181] The evaluation unit 124 calculates the vertical coordinate z for each of the reflective features by using a depth from photon ratio technique. DPR For Depth from Photon Ratio (DPR) techniques, see WO2018 / 091649A1, WO2018 / 091638A1 and WO2018 / 091640A1, the entire contents of which are incorporated by reference.

[0182] The evaluation device 124 can be configured to determine a beam profile for each of the reflective features. Determining the beam profile can include identifying at least one reflective feature provided by the optical sensor 118 and / or selecting at least one reflective feature provided by the optical sensor 118 and evaluating the intensity distribution of at least one of the reflective features. As an example, a region of the image can be used and evaluated to determine an intensity distribution, such as a three-dimensional intensity distribution or a two-dimensional intensity distribution, such as along an axis or line through the image. As an example, the center of illumination by the light beam can be determined, such as by determining at least one pixel with the highest illumination, and a cross-sectional axis can be selected through the center of illumination. The intensity distribution can be the intensity distribution as a function of coordinate along this cross-sectional axis through the center of illumination. Other evaluation algorithms are possible.

[0183] The analysis of the beam profile of one of the reflective features may include determining at least one first area and at least one second area of ​​the beam profile. The first area of ​​the beam profile may be area A1, and the second area of ​​the beam profile may be area A2. The evaluation unit 124 may be configured to integrate the first area and the second area. The evaluation unit may be configured to derive a combined signal, in particular a quotient Q, by one or more of: dividing the integrated first area by the integrated second area; dividing the integrated first area by a multiple of the integrated second area; or dividing a linear combination of the integrated first area by the integrated second area. The evaluation unit 124 may be configured to determine at least two areas of the beam profile and / or to divide the beam profile into at least two segments comprising different areas of the beam profile, although overlapping of the areas is possible as long as the areas do not coincide. For example, the evaluation unit 124 may be configured to determine a plurality of areas, such as two, three, four, five, or up to ten areas. The evaluation unit 124 may be configured to divide the light spot into at least two areas of the beam profile and / or divide the beam profile into at least two segments including different areas of the beam profile. The evaluation unit 124 may be configured to determine an integral of the beam profile over each of the at least two areas. The evaluation unit 124 may be configured to compare at least two of the determined integrals. Specifically, the evaluation unit 124 may be configured to determine at least one first area of ​​the beam profile and at least one second area of ​​the beam profile. The first area of ​​the beam profile and the second area of ​​the beam profile may be adjacent areas or overlapping areas, or both. The first area of ​​the beam profile and the second area of ​​the beam profile may not coincide in area.For example, the evaluation device 124 may be configured to divide the sensor area of ​​the CMOS sensor into at least two sub-areas, and the evaluation device may be configured to divide the sensor area of ​​the CMOS sensor into at least one left portion and at least one right portion, and / or at least one upper portion and at least one lower portion, and / or at least one inner portion and at least one outer portion.

[0184] Additionally or alternatively, the display device 110 may comprise at least two light sensors 118, the light sensitive areas of the first and second light sensors being arranged such that the first light sensor is adapted to determine a first area of ​​the beam profile of the reflective feature and the second light sensor is adapted to determine a second area of ​​the beam profile of the reflective feature, and the evaluation device 124 may be adapted to integrate the first area and the second area.

[0185] In one embodiment, A1 may correspond to the entire or complete area of ​​the feature on the optical sensor. A2 may be the central area of ​​the feature on the optical sensor. The central area may be a constant value. The central area may be smaller than the total area of ​​the feature. For example, for a circular feature, the central area may have a radius between 0.1 and 0.9 of the total radius of the feature, preferably between 0.4 and 0.6 of the total radius.

[0186] The evaluation unit 124 may be configured to derive the quotient Q by one or more of dividing the first area by the second area, dividing the first area by a multiple of the second area, or dividing a linear combination of the first area by the second area.

number

[0187] The evaluation device 124 may be configured to use at least one predetermined relationship between the quotient Q and the vertical coordinate to determine the vertical coordinate. The predetermined relationship may be one or more of an empirical relationship, a semi-empirical relationship, and an analytically derived relationship. The evaluation device may include at least one data storage device for storing the predetermined relationship, such as a look-up list or a look-up table.

[0188] The evaluation unit 124 may be configured to execute at least one depth from photon ratio algorithm that calculates distances for all zero-order and higher order reflectance features.

[0189] The evaluation of the first image includes sorting the identified reflective features with respect to brightness. The sorting may include assigning a sequence of reflective features for further evaluation with respect to brightness, in particular starting with the reflective feature having the greatest brightness, followed by reflective features of decreasing brightness. If the brightest reflective features are prioritized for the calculation of the DPR, then the vertical coordinate z DPR This can increase the robustness of the determination of , mainly because the zero-order reflection feature of the diffraction grating is always brighter than the spurious features of higher orders.

[0190] The evaluation device 124 calculates the vertical coordinate z DPR The technique is configured to unambiguously match reflection features to corresponding illumination features by using a depth from photon ratio technique. The longitudinal coordinates determined using the depth from photon ratio technique can be used to solve the so-called correspondence problem. In that way, the distance information for each reflection feature can be used to find the correspondence of a known laser projector grid.

[0191] Illumination features corresponding to reflection features can be determined using epipolar geometry. For a description of epipolar geometry, see, for example, Chapter 2 of X. Jiang and H. Bunke: "Dreidimensionales Computersehen," Springer, Berlin, Heidelberg, 1997. Epipolar geometry can assume that the illumination image, i.e., the undistorted illumination pattern and the first image, can be images determined at different spatial positions and / or spatial directions with a fixed distance. The distance can be a relative distance, also referred to as a baseline. The illumination image can also be referred to as a reference image. The evaluation device 124 can be adapted to determine an epipolar line within the reference image. The relative positions of the reference image and the first image can be known. For example, the relative positions of the reference image and the first image can be stored in at least one storage unit of the evaluation device. The evaluation device 124 can be adapted to determine a straight line extending from a selected reflection feature in the first image to a real-world feature from which it emanates. Thus, the straight line can include a possible object feature corresponding to the selected reflection feature. The line and baseline define an epipolar plane. Because the reference image is determined at a different relative position from the first image, corresponding possible object features can be imaged on a line called an epipolar line in the reference image. The epipolar line can be the intersection of the epipolar plane and the reference image. Therefore, the feature in the reference image that corresponds to the selected feature in the first image lies on the epipolar line.

[0192] Depending on the distance to the object in the scene that reflected the illumination feature, the reflection feature corresponding to the illumination feature may be displaced in the first image 122. The reference image may include at least one displacement region where the illumination feature corresponding to the selected reflection feature will be imaged. The displacement region may include only one illumination feature. The displacement region may include multiple illumination features. The displacement region may include an epipolar line or a portion of an epipolar line. The displacement region may include multiple epipolar lines or multiple portions of multiple epipolar lines. The displacement region may extend along the epipolar line, or may extend perpendicular to the epipolar line, or both. The evaluation unit 124 may be adapted to determine the illumination feature along the epipolar line. The evaluation unit 124 may be adapted to determine a vertical coordinate z and an error interval ±ε of the reflection feature from the combined signal Q to determine a displacement region along the epipolar line corresponding to z±ε or a displacement region perpendicular to the epipolar line. The measurement uncertainty of the distance measurement using the combined signal Q may result in a non-circular displacement region in the second image, since the measurement uncertainty may vary with direction. Specifically, the measurement uncertainty along one or more epipolar lines may be larger than the measurement uncertainty in a direction perpendicular to the one or more epipolar lines. The displacement region may include an extension in a direction perpendicular to the one or more epipolar lines. The evaluation unit 124 may be adapted to match the selected reflection feature with at least one illumination feature within the displacement region. The evaluation unit 124 may be adapted to match the determined vertical coordinate z DPRThe evaluation unit 124 may be adapted to match selected features of the first image with illumination features within the displacement region by using at least one evaluation algorithm that takes into account the above. The evaluation algorithm may be a linear scaling algorithm. The evaluation unit 124 may be adapted to determine an epipolar line closest to the displacement region and / or within the displacement region. The evaluation unit may be adapted to determine an epipolar line closest to the image position of the reflection feature. The extent of the displacement region along the epipolar line may be greater than the extent of the displacement region perpendicular to the epipolar line. The evaluation unit 124 may be adapted to determine the epipolar line before determining the corresponding illumination feature. The evaluation unit 124 may determine the displacement region around the image position of each reflection feature. The evaluation unit 124 may be adapted to assign an epipolar line to each displacement region at each image position of the reflection feature, for example, by assigning the epipolar line closest to the displacement region and / or the epipolar line within the displacement region and / or the epipolar line closest to the displacement region along a direction perpendicular to the epipolar line. The evaluation device 124 may be adapted to determine the illumination feature corresponding to the reflection feature by determining the illumination feature closest to the assigned displacement region and / or the illumination feature within the assigned displacement region and / or the illumination feature closest to the assigned displacement region along the assigned epipolar line and / or the illumination feature within the assigned displacement region along the assigned epipolar line.

[0193] Additionally or alternatively, the evaluation device 124 performs the following steps: - determining the displacement area for each reflection feature's image position; - assigning an epipolar line to the displacement region of each reflection feature, such as by assigning the epipolar line closest to the displacement region and / or the epipolar line within the displacement region and / or the epipolar line closest to the displacement region along a direction orthogonal to the epipolar line; - assigning and / or determining at least one illumination feature to each reflection feature, such as by assigning an illumination feature closest to the assigned displacement area and / or an illumination feature within the assigned displacement area and / or an illumination feature closest to the assigned displacement area along the assigned epipolar line and / or an illumination feature within the assigned displacement area along the assigned epipolar line; may be configured to perform the following.

[0194] Additionally or alternatively, the evaluation device 124 may be adapted to determine among the illumination features assigned to a plurality of epipolar lines and / or reflection features, for example by comparing the distances of the reflection features and / or epipolar lines in the illumination image and / or by comparing the ε-weighted distances of the illumination features and / or the error-weighted distances of the epipolar lines in the illumination image, and assigning the epipolar line and / or illumination feature with the shorter distance and / or ε-weighted distance to the illumination feature and / or reflection feature.

[0195] As described above, the diffraction grating generates multiple reflection features, e.g., one true feature and multiple false features for each illumination feature. Matching is performed starting with the brightest reflection feature and proceeding by decreasing the brightness of the reflection features. Other reflection features cannot be assigned to the same matched illumination feature. Due to display artifacts, the generated false features are generally darker than the true features. By sorting the reflection features by brightness, brighter reflection features are prioritized for correspondence matching. If the illumination feature correspondence has already been used, the false feature cannot be assigned to the used, i.e., matched, illumination feature.

[0196] FIG. 2A shows a simulated first image 122 without the display 112 for an illumination pattern including a single light spot. FIG. 2B shows the first image 122 captured by the optical sensor 118 behind the display 112. It can be seen that the diffraction grating produces multiple spots. In FIG. 2B, a true feature is indicated by reference numeral 126, and an exemplary false feature is indicated by reference numeral 128. FIG. 2C shows a further example of the first image 122 captured by the optical sensor 118 behind the display 112, where the illumination pattern is a projected laser grid. The multiple spots appear due to the diffraction grating.

[0197] FIG. 3A shows a further exemplary first image 122 of a scene with a projected laser spot. Reflection features of zero-order and higher-order 132 of the diffraction grating 130 are shown. FIGS. 3B and 3C illustrate matching of reflection features with illumination features. The left portions of FIGS. 3B and 3C show the first image 122, and the right portions show the corresponding illumination pattern including two illumination features. The first image 122 may include six reflection features. The evaluation device 124 may be configured to identify the reflection features in the first image 122 and sort them by brightness. As shown in FIG. 3B, two of the reflection features may be brighter than the other reflection features. The evaluation device 124 may begin beam profile analysis and begin matching illumination features with one of two brighter reflection features indicated by circles 134. Each of the two brighter reflection features may be matched with one illumination feature indicated by an arrow. The evaluation device 124 may classify the matching features as true features. As shown in FIG. 3C , two illumination features of the illumination pattern have already been matched with brighter reflection features. Other reflection features cannot be assigned to the same matched illumination feature. By sorting the reflection features by brightness, brighter reflection features are prioritized for correspondence matching. If the illumination feature correspondence has already been used, a false feature cannot be assigned to the used illumination feature, i.e., the matched illumination feature. Therefore, the two remaining reflection features, indicated by circles 136, do not have corresponding illumination features and cannot be assigned to any point in the pattern. The remaining reflection features are classified as false features by the evaluation device 124.

[0198] The evaluation unit 124 rejects false features and calculates the vertical coordinate z DPR The display device 110 may be used to generate a 3D map from a scene, for example a facial scene.

[0199] The depth map can be further refined by using additional depth measurement techniques such as triangulation and / or depth from defocus and / or structured light. The evaluation device uses triangulation and / or depth from defocus and / or structured light techniques to determine at least one second vertical coordinate z for each of the reflective features. triang The evaluation device 124 may be configured to determine the second vertical coordinate z triang and the vertical coordinate z DPR The bond vertical coordinate may be determined by the second vertical coordinate z triang and the vertical coordinate z DPR The combined vertical coordinate can be used for determining the depth map.

[0200] As shown in FIG. 1B , the display device 110 may include an additional illumination source 138. The additional illumination source 138 may include at least one light-emitting diode (LED). The additional illumination source 138 may be configured to generate light within the visual spectrum. The light sensor 118 may be configured to determine at least one second image including at least one two-dimensional image of the scene. The additional illumination source 138 may be configured to provide additional illumination for imaging the second image. For example, the configuration of the display device 110 can be extended with an additional flood illumination LED. The additional illumination source 138 may illuminate a scene, such as a face, using an LED, particularly without an illumination pattern, and the light sensor 118 may be configured to capture a two-dimensional image. The two-dimensional image may be used for face detection and verification algorithms.

[0201] If the impulse response of the display 112 is known, the distorted image captured by the optical sensor 118 can be restored. The evaluation unit 124 may be configured to determine at least one corrected image I by deconvolving the second image I (I = I * g) with a grating function g. The grating function is also called the impulse response. The undistorted image can be restored by a deconvolution approach, for example, Van-Cittert or Wiener deconvolution.

[0202] As shown in FIG. 4, the display device 110 can be configured to determine the diffraction grating function g. The display device 110 may be configured to illuminate a black scene with an illumination pattern having a small, single bright spot, indicated by reference numeral 140. The captured image 142 may be the diffraction grating function. This procedure may be performed only once, such as during calibration. To determine a corrected image even when imaging through the display 112, the display device 110 may be configured to capture an image and use a deconvolution approach using the captured impulse response g. The resulting image may be a reconstructed image with fewer display artifacts and may be used for several applications, such as face recognition. FIGS. 5A-5C show example two-dimensional images captured by the optical sensor 118. In FIG. 5A, an example scene was captured with the optical sensor 118 behind the display 112. In FIG. 5B, an example scene was captured with the optical sensor 118 without the display 112. FIG. 5C shows an image reconstructed using the deconvolution approach.

[0203] Explanation of symbols 110 Display device 112 Display 114 Irradiation source 116 Housing 118 Optical Sensor 120 photosensitive area 122 Image 1 124 Evaluation Device 126 True Characteristics 128 Fake Characteristics 130 Zero-order diffraction grating 132 Higher 134 yen 136 yen 138 Further sources of radiation 140 Illuminating a Black Scene 142 captured images [Prior art documents] [Patent documents]

[0204] [Patent Document 1] DE202018003644U1 [Patent Document 2] US9,870,024B2 [Patent Document 3] US10,057,541B2 [Patent Document 4] US10,215,988B2 [Patent Document 5] WO2018 / 091649A1 [Patent Document 6] WO2018 / 091638A1 [Patent Document 7] WO2018 / 091640A1 [Patent Document 8] WO2019 / 042956A1

Claims

1. at least one illumination source (114) configured to project at least one illumination pattern comprising a plurality of illumination features onto at least one scene; at least one light sensor (118) having at least one light-sensitive area (120), said light sensor (118) configured to determine at least one first image (122) comprising a plurality of reflection features produced by said scene in response to illumination by said illumination feature; and at least one light-transmitting display (112) configured to display information, said illumination source (114) and said light sensor (118) being arranged in front of said display (112) in the direction of propagation of said illumination pattern; at least one evaluation device (124), said evaluation device (124) being configured to evaluate said first image (122), said evaluation of said first image (122) comprising identifying said reflection features of said first image (122) and sorting said identified reflection features with respect to brightness, each of said reflection features comprising at least one beam profile, said evaluation device (124) determining at least one vertical coordinate z of each of said reflection features by analysis of said beam profiles; ＤＰＲ at least one evaluation device (124) configured to determine A display device (110) comprising: The evaluation device (124) determines the vertical coordinate z ＤＰＲ and the evaluation device (124) is configured to unambiguously match reflection features to corresponding illumination features by using a function of the vertical coordinate z ＤＰＲ a display device (110) configured to generate a depth map for the true features using

2. The evaluation device (124) uses triangulation and / or depth from defocus and / or structured light techniques to determine at least one second vertical coordinate z for each of the reflective features. ｔｒｉａｎｇ The display device (110) of claim 1, configured to determine:

3. The evaluation device (124) determines the second vertical coordinate z ｔｒｉａｎｇ and the vertical coordinate z ＤＰＲ and the bond vertical coordinate is determined by the second vertical coordinate z ｔｒｉａｎｇ and the vertical coordinate z ＤＰＲ 3. The display device (110) of claim 2, wherein the combined vertical coordinate is an average of .times. ...

4. The display device (110) of any one of claims 1 to 3, wherein the illumination source (114) comprises at least one laser projector, the laser projector including at least one laser source and at least one diffractive optical element (DOE).

5. 5. The display device (110) of claim 1, wherein the illumination source (114) is configured to generate at least one light beam having a beam path passing from the illumination source (114) through the display (112) to the scene, and the display (112) is configured to function as a grating such that the light beam is diffracted by the display (112), thereby producing the illumination pattern.

6. 6. The display device (110) of claim 5, wherein wiring of the display (112) is configured to form gaps and / or slits and ridges of the grating.

7. The display device (110) of any one of claims 1 to 6, wherein the illumination pattern comprises a periodic dot pattern.

8. The display device (110) of any one of claims 1 to 7, wherein the illumination pattern has a low point density, the illumination pattern having no more than 2500 points per field of view.

9. 9. The display device (110) of claim 1, wherein the evaluation device (124) is configured to determine the beam profile information for each of the reflective features by using a depth from photon ratio technique.

10. The display device (110) of any one of claims 1 to 9, wherein the light sensor (118) comprises at least one CMOS sensor.

11. The display device (110) of any one of claims 1 to 10, wherein the display device (110) comprises a further illumination source (138), the further illumination source (138) comprising at least one light emitting diode (LED).

12. 12. The display device (110) of claim 11, wherein the further illumination source (138) is configured to generate light within the visual spectrum.

13. 13. A display device (110) as described in claim 11 or 12, wherein the optical sensor (118) is configured to determine at least one second image comprising at least one two-dimensional image of the scene, and the further illumination source (138) is configured to provide additional illumination for imaging the second image.

14. The evaluation device (124) generates at least one corrected image I by deconvolving the second image I with a diffraction grating function g. ０ where I=I ０ 14. The display device (110) of claim 13, wherein *g.

15. A method for depth measurement through a light-transmitting display (112), wherein at least one display device (110) according to any one of claims 1 to 14 is used, comprising the following steps: a) projecting at least one illumination pattern comprising a plurality of illumination features onto at least one scene by using at least one illumination source (114), the illumination source (114) being positioned in front of a display (112) in a propagation direction of the illumination pattern; b) determining at least one first image (122) comprising a plurality of reflective features produced by the scene in response to illumination by the illumination features by using at least one optical sensor (118), the optical sensor (118) having at least one light-sensitive area (120), the optical sensor (118) being positioned in front of the display (112) in a propagation direction of the illumination pattern, each of the reflective features comprising at least one beam profile; c) evaluating said first image (122) by using at least one evaluation device (124), said evaluation comprising the following sub-steps: C1) identifying the reflective features of the first image (122) and sorting the identified reflective features with respect to brightness; C2) by analyzing the beam profile, at least one vertical coordinate z ＤＰＲ and the sub-step of determining C3) The vertical coordinate z ＤＰＲ a sub-step of unambiguously matching reflective features to corresponding illumination features by using a reflective feature vector, said matching being performed starting from the brightest reflective feature and decreasing the brightness of said reflective features; C4) classifying the reflection features that match the illumination features as true features and the reflection features that do not match the illumination features as false features; C5) Rejecting false features and determining the vertical coordinate z ＤＰＲ and generating a depth map of the true features by using The method includes the steps of:

16. 16. Use of a display device (110) according to any one of claims 1 to 15 for a display device, the use being selected from the group consisting of: position determination in traffic technology; entertainment applications; security applications; surveillance applications; safety applications; human machine interface applications; tracking applications; photography applications; imaging or camera applications; mapping applications for generating at least one map of a space; homing or tracking beacon detectors for vehicles; outdoor applications; mobile applications; communication applications; machine vision applications; robotics applications; quality control applications; manufacturing applications.

Images