Depth measurement by display

Abstract

A display device (110) is proposed. The display device (110) includes: - at least one illumination source (114) configured to project at least one illumination pattern including a plurality of illumination features onto at least one scene; - at least one optical sensor (118) having at least one photosensitive area (120), wherein the optical sensor (118) is configured to determine at least one first image (112), the first image (122) including a plurality of reflection features generated by the scene in response to illumination by the illumination features; - at least one translucent display (112) configured to display information, wherein the illumination source (114) and the optical sensor... (118) The propagation direction of the illumination pattern placed in front of the display (112); - At least one evaluation device (124), wherein the evaluation device (124) is configured to evaluate the first image (122), wherein the evaluation of the first image (122) includes identifying the reflection features of the first image (122) and classifying the identified reflection features with respect to brightness, wherein each of the reflection features includes at least one beam profile, wherein the evaluation device (124) is configured to determine at least one longitudinal coordinate z for each of the reflection features by analyzing their beam profiles. DPR The evaluation device (124) is configured to use the longitudinal coordinate z DPR The reflection features are explicitly matched with corresponding illumination features, wherein the matching is performed by decreasing the brightness of the reflection features starting from the brightest reflection feature, wherein the evaluation device (124) is configured to classify reflection features that match the illumination features as true features and reflection features that do not match the illumination features as false features, wherein the evaluation device (124) is configured to reject the false features and to use the vertical coordinate z DPR To generate a depth map for the real features.

Description

Technical Field

[0001] This invention relates to a display device and a method for depth measurement via a semi-transparent display, as well as various applications of the display device. The device, method, and applications according to the invention can be specifically used, for example, in various fields such as daily life, security technology, gaming, transportation technology, production technology, photography such as digital or video photography for artistic purposes, documentation or technical purposes, security technology, information technology, agriculture, crop protection, maintenance, cosmetics, medical technology, or science. However, other applications are also possible. Background Technology

[0002] Multiple display devices are known. Recent developments in devices with displays show that the display area should cover the entire available space, and the frame around the display should be as small as possible. This means that electronic components and sensors, such as front-facing cameras, flashlights, proximity sensors, and even 3D imaging sensors, can no longer be arranged within the frame but must be placed below the display. However, most common 3D imaging technologies and systems, such as structured light-based or 3D time-of-flight (ToF) based 3D imaging systems, cannot be effortlessly placed below the display.

[0003] Until now, it is unknown whether structured light or 3D-ToF-based 3D imaging systems operate under a display, i.e., without creating empty windows containing no microcircuits and / or microwiring, for placing components or devices of the 3D imaging system so that they can be "seen" through these windows.

[0004] For structured light, the main problem is the microstructure of the microcircuits and / or microwiring in transparent displays, and therefore, the low light transmittance of the display. This microstructure is generated by an electrode matrix used to address individual pixels. Moreover, the pixel itself represents an inverted grating because the metal cathode of a single pixel is opaque. In principle, the display structure can be made transparent or translucent by using specific materials as a whole (including the electrodes), but until now there have been no transparent or translucent displays without a grating-like microstructure.

[0005] Structured light-based 3D imagers rely on projecting a point cloud with thousands of points and a well-known pattern onto a scene. The microstructure of the transparent or semi-transparent display functions similarly to the diffraction grating structure used for lasers. Since most structured light imagers' projectors are based on laser sources that project a sharp point pattern, this pattern undergoes the grating effect of the display, and each individual point in the point pattern will display a higher diffraction order. This has a devastating impact on structured light imagers because the extra and unwanted points introduced by the grating structure make it highly complex for algorithms to retrieve the original intended pattern.

[0006] Furthermore, the number of projection points used in conventional structured light imagers is quite high. Because transparent displays have very low light transmittance, even in the infrared (IR) range (e.g., 850 nm and 940 nm, typical wavelengths for 3D imagers), structured light projectors require very high output power to achieve sufficient power detectable by the imager through the display. The imager must also be located below the display, leading to additional light absorption. This combination of high point count and low transmittance can result in poor ambient light robustness.

[0007] For 3D-ToF sensors, reflections on the display surface cause multiple reflections, and differences in delay occur as light passes through the display. Different display structures have different refractive indices, which hinder robust functionality when used behind the display. Furthermore, 3D-ToF sensors require a large amount of light to illuminate the scene. Additionally, the illumination should be uniform. The low transmittance of the display makes it difficult to provide sufficient light, and the grating structure affects the uniformity of illumination.

[0008] Common 3D sensing systems suffer from the problem of measuring through transparent displays. Current devices use recesses in the display. In that way, the sensor is unaffected by diffraction optical effects.

[0009] DE 20 2018 003 644 U1 describes a portable electronic device comprising: a bottom wall and sidewalls cooperating with the bottom wall to define a cavity, the sidewalls having edges defining an opening to the cavity; a protective layer covering the opening and sealing the cavity; and a vision subsystem disposed within the cavity and between the protective layer and the bottom wall for providing a depth map of an object outside the protective layer, the vision subsystem comprising: a clip assembly for carrying optical components that cooperate to generate information for the depth map, the clip assembly comprising: a first support arranged to support and hold the optical components at fixed distances from each other, and a second support having a body fixed to the first support, wherein the second support has a protrusion extending away from the body.

[0010] US 9,870,024 B2 describes an electronic display comprising multiple layers, such as a capping layer, a color filter layer, a display layer including light-emitting diodes or organic light-emitting diodes, a thin-film transistor layer, etc. In one embodiment, these layers include a substantially transparent region disposed above a camera. The substantially transparent region allows light from the outside to reach the camera, enabling the camera to record an image.

[0011] US 10,057,541 B2 describes an image capturing device and a shooting method. The image capturing device includes: a transparent display panel; and a camera facing the bottom surface of the transparent display panel for synchronizing shutter time with a period of time when the transparent display panel displays a black image, and for capturing an image located in front of the transparent display panel.

[0012] US 10,215,988 B2 describes an optical system for displaying light from a scene, the optical system including an active optical component comprising a first plurality of light guide apertures, an optical detector, a processor, a display, and a second plurality of light guide apertures. The first plurality of light guide apertures are positioned to provide optical input to the optical detector. The optical detector is positioned to receive the optical input and convert the optical input into electrical signals corresponding to intensity and position data. The processor is connected to receive data from the optical detector and process the data for display. The second plurality of light guide apertures are positioned to provide optical output from the display.

[0013] WO 2019 / 042956 A1 describes a detector for determining the position of at least one object. The detector includes: - at least one sensor element having a matrix of optical sensors, each optical sensor having a photosensitive region, wherein each optical sensor is designed to generate at least one sensor signal in response to illumination of its respective photosensitive region by a reflected light beam propagating from the object to the detector, wherein the sensor element is adapted to determine at least one reflection image; - at least one evaluation device, wherein the evaluation device is adapted to select at least one reflection feature of the reflection image, wherein the evaluation device is configured to determine at least one longitudinal region of the selected reflection feature of the reflection image by evaluating a combined signal Q from the sensor signals, wherein the evaluation device is adapted to determine at least one displacement region in at least one reference image corresponding to the longitudinal region, wherein the evaluation device is adapted to match the selected reflection feature with at least one reference feature within the displacement region.

[0014] The problem solved by this invention

[0015] Therefore, the object of the present invention is to provide apparatus and methods that address the aforementioned technical challenges of known devices and methods. Specifically, the object of the present invention is to provide apparatus and methods that allow reliable depth measurement via a display with low technical effort and low requirements in terms of technical resources and cost. Summary of the Invention

[0016] This problem is solved by the present invention, which features independent patent claims. Advantageous developments of the invention, which can be implemented individually or in combination, are presented in the dependent claims and / or in the following description and detailed embodiments.

[0017] As used below, the terms “have,” “include,” or “contain,” or any grammatical variations thereof, are used in a non-exclusive manner. Thus, these terms can refer to a situation where an entity described in this context has no other features besides those introduced by these terms, or to a situation where one or more other features exist. For example, the statements “A has B,” “A includes B,” and “A contains B” can refer to a situation where A has no other elements besides B (i.e., A consists solely and exclusively of B), or to a situation where entity A has one or more other elements besides B (such as element C, elements C and D, or even other elements).

[0018] Furthermore, it should be noted that the terms "at least one," "one or more," or similar expressions indicating that a feature or element may exist once or more will generally be used only once when the corresponding feature or element is introduced. In the following text, in most cases, when referring to the corresponding feature or element, the expression "at least one" or "one or more" will not be repeated, but the fact that the corresponding feature or element may exist once or more will be acknowledged.

[0019] Furthermore, as used below, the terms “preferredly,” “more preferably,” “particularly,” “more particularly,” “specifically,” “more specifically,” or similar terms may be used in combination with optional features without limiting the possibility of substitution. Therefore, features introduced by these terms are optional features and are not intended to limit the scope of the claims in any way. As those skilled in the art will recognize, the invention can be practiced using alternative features. Similarly, features introduced by phrases such as “in embodiments of the invention” are intended to be optional features, without limiting alternative embodiments of the invention, without limiting the scope of the invention, and without limiting the possibility of combining features introduced in this manner with other optional or non-optional features of the invention.

[0020] In a first aspect of the invention, a display device is disclosed. As used herein, the term "display" can refer to any shaped device configured to display information items such as at least one image, at least one chart, at least one histogram, at least one text, or at least one symbol. The display can be at least one monitor or at least one screen. The display can have any shape, preferably a rectangular shape. As used herein, the term "display device" can generally refer to at least one electronic device including at least one display. For example, a display device can be at least one device selected from the group consisting of: a television device, a smartphone, a game console, a personal computer, a laptop computer, a tablet computer, at least one virtual reality device, or a combination thereof.

[0021] The display device includes:

[0022] - At least one illumination source configured to project at least one illumination pattern comprising multiple illumination features onto at least one scene;

[0023] - At least one optical sensor having at least one photosensitive area, wherein the optical sensor is configured to determine at least one first image, the at least one first image including a plurality of reflection features generated by the scene in response to illumination by the illumination features;

[0024] - At least one semi-transparent display configured to display information, wherein the illumination source and the optical sensor are placed in the direction of propagation of the illumination pattern in front of the display;

[0025] - At least one evaluation device, wherein the evaluation device is configured to evaluate the first image, wherein the evaluation of the first image includes identifying the reflection features of the first image and classifying the identified reflection features with respect to brightness, wherein each of the reflection features includes at least one beam profile, and wherein the evaluation device is configured to determine at least one longitudinal coordinate z for each of the reflection features by analyzing their beam profiles. DPR ,

[0026] The evaluation device is configured to use the vertical coordinate z DPR The reflection features are explicitly matched with corresponding illumination features, wherein the matching is performed by decreasing the brightness of the reflection features starting from the brightest reflection feature. The evaluation device is configured to classify reflection features that match the illumination features as true features and reflection features that do not match the illumination features as false features. The evaluation device is configured to reject the false features and to use the vertical coordinate z... DPR To generate a depth map for the real features.

[0027] As used herein, the term "scene" can refer to at least one arbitrary object or area of ​​space. A scene may include at least one object and its surrounding environment.

[0028] An illumination source is configured to project at least one illumination pattern comprising multiple illumination features onto a scene. As used herein, the term "illumination source" can generally refer to at least one arbitrary means adapted to provide at least one illumination beam for illuminating a scene. The illumination source can be adapted to directly or indirectly illuminate the scene, wherein the illumination pattern is reflected or scattered by a surface of the scene and thus guided at least partially toward an optical sensor. The illumination source can be adapted to illuminate the scene, for example, by guiding the beam toward the scene of the reflected beam. The illumination source can be configured to generate an illumination beam for illuminating the scene.

[0029] The illumination source may include at least one light source. The illumination source may include multiple light sources. The illumination source may include artificial illumination sources, particularly at least one laser source and / or at least one incandescent lamp and / or at least one semiconductor light source, such as at least one light-emitting diode, particularly an organic and / or inorganic light-emitting diode. As an example, the light emitted by the illumination source may have a wavelength of 300 to 1100 nm, particularly 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range may be used, such as in the range of 780 nm to 3.0 μm. Specifically, light in a portion of the near-infrared region may be used, in which silicon photodiodes are specifically suited for the range of 700 nm to 1100 nm. The illumination source may be configured to generate at least one illumination pattern in the infrared region. Using light in the near-infrared region allows the light to be detected by the human eye only weakly and not at all, but still detectable by a silicon sensor, particularly a standard silicon sensor.

[0030] As used herein, the term "ray" generally refers to a line of light whose wavefront is perpendicular to the direction of energy flow. As used herein, the term "bundle" generally refers to a collection of rays. Hereinafter, the terms "ray" and "bundle" will be used as synonyms. As further used herein, the term "beam" generally refers to the amount of light, specifically the amount of light traveling substantially in the same direction, including the possibility of a beam having an extension angle or widening angle. A beam may have spatial extension. Specifically, a beam may have a non-Gaussian beam profile. The beam profile may be selected from the group consisting of: trapezoidal beam profile; triangular beam profile; conical beam profile. A trapezoidal beam profile may have a plateau region and at least one edge region. A beam may specifically be a Gaussian beam or a linear combination of Gaussian beams, as will be outlined in more detail below. However, other embodiments are possible.

[0031] The illumination source can be configured to emit light of a single wavelength. Specifically, this wavelength can be in the near-infrared region. In other embodiments, the illumination can be adapted to emit light with multiple wavelengths, thereby allowing for additional measurements in other wavelength channels.

[0032] The illumination source may be or may include at least one multi-beam light source. For example, the illumination source may include at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the illumination source may include at least one laser and / or laser source. Various types of lasers can be used, such as semiconductor lasers, dual heterostructure lasers, external cavity lasers, split-confined heterostructure lasers, quantum cascade lasers, distributed Bragg reflector lasers, polaron lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, indium arsenide lasers, transistor lasers, diode-pumped lasers, distributed feedback lasers, quantum well lasers, interband cascade lasers, gallium arsenide lasers, semiconductor ring lasers, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alternatively, non-laser light sources, such as LEDs and / or bulbs, may be used. The illumination source may include one or more diffractive optical elements (DOEs) adapted to generate an illumination pattern. For example, the illumination source can be adapted to generate and / or project a point cloud. For instance, the illumination source may include one or more of the following: at least one digital light processing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one light-emitting diode array; at least one laser source array. Using at least one laser source as the illumination source is particularly preferred due to its generally defined beam profile and other characteristics of operability. The illumination source can be integrated into the housing of the display device.

[0033] In one embodiment, the illumination source can be a single beam source or multiple beam sources and can be configured to project at least one illumination pattern, such as at least one dot pattern. The illumination pattern can be generated as follows: The illumination source can be configured to generate at least one light beam. The illumination source can be positioned in the direction of propagation of the illumination pattern in front of the display. Therefore, the beam path can travel from the illumination source through the display to the scene. During its passage through the display, the beam may undergo diffraction by the display, which may result in a characteristic illumination pattern, such as a dot pattern. The display in this embodiment can be used as a grating. The wiring of the display, particularly the wiring of the screen, can be configured to form gaps and / or slits and ridges in the grating.

[0034] Furthermore, the illumination source can be configured to emit modulated or unmodulated light. When using multiple illumination sources, the different sources can have different modulation frequencies, as further detailed below, which can then be used to distinguish the light beams.

[0035] One or more light beams generated by an illumination source can typically propagate parallel to or tilted relative to the optical axis, for example, by an angle relative to the optical axis. A display device can be configured such that one or more light beams propagate from the display device toward the scene along the optical axis of the display device. For this purpose, the display device may include at least one reflective element, preferably at least one prism, for deflecting the illumination beam onto the optical axis. As an example, one or more light beams, such as a laser beam, and the optical axis may include an angle of less than 10°, preferably less than 5° or even less than 2°. However, other embodiments are also possible. Moreover, one or more light beams can be on or off the optical axis. As an example, one or more light beams can be parallel to the optical axis, having a distance of less than 10 mm from the optical axis, preferably less than 5 mm or even less than 1 mm from the optical axis, or even coincident with the optical axis.

[0036] As used herein, the term "at least one illumination pattern" means at least one arbitrary pattern comprising at least one illumination feature suitable for at least a portion of the illumination scene. As used herein, the term "illumination feature" means at least one at least partially extended feature of a pattern. An illumination pattern may include a single illumination feature. An illumination pattern may include multiple illumination features. An illumination pattern may be selected from the group consisting of: at least one dot pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; at least one pattern comprising an arrangement of periodic or aperiodic features. An illumination pattern may include regular and / or constant and / or periodic patterns, such as triangular patterns, rectangular patterns, hexagonal patterns, or patterns comprising further convex tiles. An illumination pattern may exhibit at least one illumination feature selected from the group consisting of: at least one dot; at least one line; at least two lines, such as parallel or intersecting lines; at least one dot and one line; at least one arrangement of periodic or aperiodic features; at least one arbitrarily shaped feature. The illumination pattern may include at least one pattern selected from the group consisting of: at least one point pattern, particularly a pseudo-random point pattern; a random point pattern or a quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern including at least one predictable feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern; at least one pattern including convex uniform tiles; at least one line pattern including at least one line; at least one line pattern including at least two lines, such as parallel or intersecting lines. For example, the illumination source may be adapted to generate and / or project a point cloud. The illumination source may include at least one light projector adapted to generate a point cloud such that the illumination pattern may include multiple point patterns. The illumination source may include at least one mask adapted to generate the illumination pattern from at least one light beam generated by the illumination source.

[0037] The distance between two features of the illumination pattern and / or the area of ​​at least one illumination feature may depend on the blurred circle in the image. As outlined above, the illumination source may include at least one light source configured to generate the at least one illumination pattern. Specifically, the illumination source includes at least one light source and / or at least one laser diode designated for generating laser radiation. The illumination source may include at least one diffractive optical element (DOE). The display device may include at least one light projector, such as at least one laser source and DOE, adapted to project at least one periodic dot pattern.

[0038] As used further herein, the term “projecting at least one illumination pattern” means providing at least one illumination pattern for illuminating at least one scene.

[0039] For example, the projected illumination pattern can be a periodic dot pattern. The projected illumination pattern can have a low dot density. For example, the illumination pattern may include at least one periodic dot pattern with a low dot density, wherein the illumination pattern has ≤2500 dots per field of view. Compared to structured light, which typically has a dot density of 10k-30k in a 55x38° field of view, the illumination pattern according to the invention can be sparser. This allows for more power per dot, making the proposed technique less dependent on ambient light compared to structured light.

[0040] The display device may include a single camera, which includes an optical sensor. The display device may also include multiple cameras, each including one or more optical sensors.

[0041] An optical sensor has at least one photosensitive area. As used herein, "optical sensor" generally refers to a photosensitive device for detecting a light beam, such as for detecting illumination and / or a light spot produced by at least one light beam. As further used herein, "photosensitive area" generally refers to a region of an optical sensor that can be illuminated from the outside by at least one light beam, generating at least one sensor signal in response to the illumination. The photosensitive area may be specifically located on the surface of the respective optical sensor. However, other embodiments are also possible. A display device may include multiple optical sensors, each having a photosensitive area. As used herein, the term "each optical sensor has at least one photosensitive area" refers to a configuration in which each of a plurality of individual optical sensors has one photosensitive area, and a configuration in which a combined optical sensor has multiple photosensitive areas. Furthermore, the term "optical sensor" refers to a photosensitive device configured to generate an output signal. In the case where a display device includes multiple optical sensors, it is possible to achieve that each optical sensor has precisely one photosensitive area present in the respective optical sensor, such as by precisely providing one photosensitive area that can be illuminated, generating a precisely uniform sensor signal for the entire optical sensor in response to the illumination. Therefore, each optical sensor may be a single-area optical sensor. However, the use of a single-area optical sensor makes the setup of the display device particularly simple and efficient. Therefore, as an example, commercially available photoelectric sensors, such as commercially available silicon photodiodes, can be used in the setup, each photoelectric sensor having exactly one photosensitive area. However, other embodiments are also possible.

[0042] Preferably, the photosensitive region can be oriented substantially perpendicular to the optical axis of the display device. The optical axis can be a straight optical axis, or it can be curved, or even split, for example by using one or more deflecting elements and / or by using one or more beam splitters, wherein in the latter case, the substantially perpendicular orientation can indicate the local optical axis in the corresponding branch or beam path of the optical setup.

[0043] The optical sensor may specifically be or may include at least one photodetector, preferably an inorganic photodetector, more preferably an inorganic semiconductor photodetector, and most preferably a silicon photodetector. Specifically, the optical sensor may be sensitive in the infrared spectral range. All pixels in the matrix or at least one group of optical sensors in the matrix may specifically be identical. The same group of pixels in the matrix may be specifically provided for different spectral ranges, or all pixels may have the same spectral sensitivity. Furthermore, the pixels may have the same size and / or electronic or optoelectronic properties. Specifically, the optical sensor may be or may include at least one inorganic photodiode sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers. Specifically, the optical sensor may be sensitive in a portion of the near-infrared region, in which the silicon photodiode is specifically suited for the range of 700 nm to 1100 nm. Infrared optical sensors that can be used for the optical sensor are commercially available infrared optical sensors, such as the TrinamiX from Ludwigshafen am Rhein (Germany) D-67056. TM The trademark name launched by GmbH is Hertzstueck TM Commercially available infrared optical sensors are available. Therefore, as an example, the optical sensor may include at least one optical sensor of an inherent photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of: Ge photodiodes, InGaAs photodiodes, extended InGaAs photodiodes, InAs photodiodes, InSb photodiodes, and HgCdTe photodiodes. Additionally or alternatively, the optical sensor may include at least one optical sensor of an inherent photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of: Ge:Au photodiodes, Ge:Hg photodiodes, Ge:Cu photodiodes, Ge:Zn photodiodes, Si:Ga photodiodes, and Si:As photodiodes. Additionally or alternatively, the optical sensor may include at least one photoconductivity sensor, such as a PbS or PbSe sensor, or a radiative thermal meter, preferably selected from V0 radiative thermal meters and amorphous Si radiative thermal meters.

[0044] Optical sensors can be sensitive in one or more of the ultraviolet, visible, or infrared spectral ranges. Specifically, optical sensors can be sensitive in the visible spectral range from 500 nm to 780 nm, most preferably at 650 nm to 750 nm or 690 nm to 700 nm. Specifically, optical sensors can be sensitive in the near-infrared region. Specifically, optical sensors can be sensitive in a portion of the near-infrared region, in which the silicon photodiode is specifically suited for the range of 700 nm to 1000 nm. Optical sensors can be sensitive in the infrared spectral range, specifically in the range of 780 nm to 3.0 micrometers. For example, each optical sensor individually can be or can include at least one element selected from the group consisting of: photodiodes, photovoltaic cells, photoconductors, phototransistors, or any combination thereof. For example, optical sensors can be or can include at least one element selected from the group consisting of: CCD sensor elements, CMOS sensor elements, photodiodes, photovoltaic cells, photoconductors, phototransistors, or any combination thereof. Any other type of photosensitive element can be used. Photosensitive elements can typically be made entirely or partially of inorganic materials and / or entirely or partially of organic materials. Most commonly, one or more photodiodes, such as commercially available photodiodes, for example, inorganic semiconductor photodiodes, can be used.

[0045] An optical sensor may include at least one sensor element comprising a pixel matrix. Therefore, as an example, an optical sensor may be part of or constitute a pixelated optical device. For example, an optical sensor may be and / or may include at least one CCD and / or CMOS device. As an example, an optical sensor may be part of or constitute at least one CCD and / or CMOS device having a pixel matrix, with each pixel forming a photosensitive area.

[0046] As used herein, the term "sensor element" generally refers to a device or combination of devices configured to sense at least one parameter. In this case, the parameter may specifically be an optical parameter, and the sensor element may specifically be an optical sensor element. Sensor elements may be formed as a single device or a combination of several devices. Sensor elements include matrices of optical sensors. Sensor elements may include at least one CMOS sensor. Matrix may include individual pixels, such as independent optical sensors. Thus, a matrix of inorganic photodiodes may be formed. However, alternatively, one or more of commercially available matrices, such as CCD detectors (e.g., CCD detector chips) and / or CMOS detectors (e.g., CMOS detector chips), may be used. Thus, generally, sensor elements may be and / or may include at least one CCD and / or CMOS device, and / or optical sensors may form a sensor array or may be part of a sensor array, such as the matrix mentioned above. Thus, as an example, sensor elements may include pixel arrays, such as a rectangular array with m rows and n columns, where m and n are independently positive integers. Preferably, more than one column and more than one row are given, i.e., n>1, m>1. Therefore, as an example, n can be 2 to 16 or higher, and m can be 2 to 16 or higher. Preferably, the ratio of the number of rows to the number of columns is close to 1. As an example, n and m can be chosen such that 0.3 ≤ m / n ≤ 3, such as by choosing m / n = 1:1, 4:3, 16:9, or similar values. As an example, the array can be a square array with equal numbers of rows and columns, such as by choosing m = 2, n = 2 or m = 3, n = 3, etc.

[0047] The matrix can include individual pixels, such as independent optical sensors. Therefore, a matrix of inorganic photodiodes can be constructed. However, alternatively, commercially available matrices can be used, such as one or more of CCD detectors (such as CCD detector chips) and / or CMOS detectors (such as CMOS detector chips). Thus, typically, the optical sensor can be and / or can include at least one CCD and / or CMOS device, and / or the optical sensor of the display device can form a sensor array or can be part of a sensor array, such as the matrix mentioned above.

[0048] Specifically, the matrix can be a rectangular matrix having at least one row, preferably multiple rows, and multiple columns. As an example, the rows and columns can be substantially vertically oriented. As used herein, the term "substantially vertical" refers to a vertical orientation condition with a tolerance of, for example, ±20° or less, preferably ±10° or less, more preferably ±5° or less. Similarly, the term "substantially parallel" refers to a parallel orientation condition with a tolerance of, for example, ±20° or less, preferably ±10° or less, more preferably ±5° or less. Thus, as an example, tolerances less than 20°, specifically less than 10°, or even less than 5° are acceptable. To provide a wide field of view, the matrix can specifically have at least 10 rows, preferably at least 500 rows, more preferably at least 1000 rows. Similarly, the matrix can have at least 10 columns, preferably at least 500 columns, more preferably at least 1000 columns. The matrix can include at least 50 optical sensors, preferably at least 100,000 optical sensors, more preferably at least 5,000,000 optical sensors. The matrix can include multiple pixels spanning millions of pixels. However, other embodiments are also feasible. Therefore, in a configuration where axial rotational symmetry is desired, a circular or concentric arrangement of the optical sensors (also referred to as pixels) within the matrix is ​​preferred.

[0049] Therefore, as an example, the sensor element may be part of or constitute a pixelated optics. For example, the sensor element may be and / or may include at least one CCD and / or CMOS device. As an example, the sensor element may be part of or constitute at least one CCD and / or CMOS device having a pixel matrix, with each pixel forming a photosensitive area. The sensor element may employ a rolling shutter or global shutter method to read out the matrix of the optical sensor.

[0050] The display device may also include at least one transmission device. The display device may also include one or more additional elements, such as one or more additional optical elements. The display device may include at least one optical element selected from the group consisting of: transmission devices, such as at least one lens and / or at least one lens system, at least one diffractive optical element. The term "transmission device" (also referred to as "transmission system") may generally refer to one or more optical elements adapted to modify a beam, such as by modifying one or more of the beam parameters, beam width, or beam direction. The transmission device may be adapted to guide the beam onto an optical sensor. Specifically, the transmission device may include one or more of the following: at least one lens, for example, at least one lens selected from the group consisting of: at least one focusable lens, at least one aspherical lens, at least one spherical lens, at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflection element, preferably at least one reflector; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitting reflector; at least one multi-lens system. As used herein, the term "focal length" of a transmission device refers to the distance at which collimated incident light rays incident on the transmission device are "focused" (which can also be expressed as "focal point"). Therefore, the focal length constitutes a measure of the transmission device's ability to converge an incident beam. Thus, a transmission device may include one or more imaging elements that can have the effect of a converging lens. For example, a transmission device may have one or more lenses, particularly one or more refracting lenses, and / or one or more convex lenses. In this example, the focal length may be defined as the distance from the center of the thin refracting lens to the principal focal point of the thin lens. For converging thin refracting lenses, such as convex or biconvex thin lenses, the focal length can be considered positive and can provide a distance at which a collimated beam incident on the thin lens acting as a transmission device can be focused into a single spot. Furthermore, the transmission device may include at least one wavelength selection element, such as at least one optical filter. Additionally, the transmission device may be designed to imprint a predefined beam profile onto electromagnetic radiation, for example, at a location in the sensor region and, particularly, the sensor area. In principle, the above-described alternative embodiments of the transmission device can be implemented individually or in any desired combination.

[0051] The transmission device may have an optical axis. In particular, the display device and the transmission device share a common optical axis. As used herein, the term "optical axis of the transmission device" generally refers to the mirror symmetry or rotational symmetry axis of a lens or lens system. The optical axis of the display device may be a line of symmetry of the optical arrangement of the display device. The display device includes at least one transmission device, preferably at least one transmission system having at least one lens. As an example, the transmission system may include at least one beam path, wherein the elements of the transmission system in the beam path are positioned in a rotationally symmetrical manner about the optical axis. However, as will be outlined in more detail below, one or more optical elements located within the beam path may also be off-center or tilted about the optical axis. However, in this case, the optical axis may be defined sequentially, such as by connecting the centers of the optical elements in the beam path to each other, for example by connecting the centers of the lenses, wherein, in this context, the optical sensor is not considered an optical element. The optical axis may generally represent a beam path. The display device may have a single beam path along which a beam travels from an object to an optical sensor, or it may have multiple beam paths. As an example, a single beam path may be given, or the beam path may be divided into two or more partial beam paths. In the latter case, each partial beam path can have its own optical axis. Optical sensors can be located within the same beam path or within a partial beam path. Alternatively, however, optical sensors can also be located in different partial beam paths.

[0052] The transmission device can form a coordinate system, where the longitudinal coordinate is the coordinate along the optical axis, and where d is the spatial offset from the optical axis. The coordinate system can also be a polar coordinate system, where the optical axis of the transmission device forms the z-axis, and where the distance from the z-axis and the polar angle can be used as additional coordinates. Directions parallel or antiparallel to the z-axis can be considered longitudinal directions, and coordinates along the z-axis can be considered longitudinal coordinates. Any direction perpendicular to the z-axis can be considered a transverse direction, and polar coordinates and / or polar angles can be considered transverse coordinates.

[0053] The display device can form a coordinate system, wherein the optical axis of the display device forms the z-axis, and wherein, in addition, x-axis and y-axis perpendicular to the z-axis and perpendicular to each other can be provided. As an example, the display device and / or a portion of the display device can be positioned at a specific point in this coordinate system, such as the origin of the coordinate system. In this coordinate system, directions parallel or antiparallel to the z-axis can be considered longitudinal directions, and coordinates along the z-axis can be considered longitudinal coordinates. Any direction perpendicular to the longitudinal direction can be considered a transverse direction, and x and / or y coordinates can be considered transverse coordinates.

[0054] Alternatively, other types of coordinate systems can be used. Thus, as an example, a polar coordinate system can be used, where the optical axis forms the z-axis, and the distance from the z-axis and the polar angle can be used as additional coordinates. Furthermore, directions parallel or antiparallel to the z-axis can be considered longitudinal directions, and coordinates along the z-axis can be considered longitudinal coordinates. Any direction perpendicular to the z-axis can be considered transverse directions, and polar coordinates and / or polar angles can be considered transverse coordinates.

[0055] An optical sensor is configured to determine at least one first image, the first image including a plurality of reflective features generated by a scene in response to illumination by an illumination feature. As used herein, but not limited to, the term “image” can specifically refer to data recorded using an optical sensor, such as a plurality of electronic readings from an imaging device, such as pixels of a sensor element. Thus, an image itself can include pixels, the pixels of an image being associated with pixels of a matrix of sensor elements. Thus, when “pixel” is referred to, either as a unit of image information generated by a single pixel of a sensor element, or directly as a single pixel of a sensor element. As used herein, the term “two-dimensional image” can generally refer to an image having information about the lateral coordinates, such as dimensions of height and width only. As used herein, the term “three-dimensional image” can generally refer to an image having information about the lateral coordinates and additional information about the longitudinal coordinates, such as dimensions of height, width, and depth. As used herein, the term “reflective feature” can refer to a feature in an image plane generated by a scene in response to illumination specifically using at least one illumination feature.

[0056] The display device includes at least one semi-transparent display configured to display information. As used herein, the term "semi-transparent" can refer to the property of the display to allow light, particularly light of a specific wavelength range, to pass through. An illumination source and optical sensors are positioned in front of the display in the direction of propagation of the illumination pattern. The illumination source and optical sensors can be arranged in fixed positions relative to each other. For example, the display device may include a camera, including optical sensors and a lens system, and a laser projector. The laser projector and camera can be fixed behind the semi-transparent display in the direction of propagation of light reflected from the scene. The laser projector can generate a dot pattern and emit light through the display. The camera can view through the display. However, the arrangement of the illumination source and optical sensors behind the semi-transparent display in the direction of propagation of light reflected from the scene may cause the display's diffraction grating to generate multiple laser dots on the scene and in a first image. Therefore, these multiple dots on the first image may not include any useful distance information. As will be described in detail below, an evaluation device can be configured to find and evaluate the zero-order reflection features of the diffraction grating, i.e., true features, and can ignore higher-order reflection features, i.e., spurious features.

[0057] The display device includes at least one evaluation device. The evaluation device is configured to evaluate a first image. As used further herein, the term "evaluation device" generally refers to any means suitable for performing specified operations, preferably by using at least one data processing means, more preferably by using at least one processor and / or at least one application-specific integrated circuit (ASIC). Thus, as an example, at least one evaluation device may include at least one data processing means on which software code containing a large number of computer commands is stored. The evaluation device may provide one or more hardware elements for performing one or more of the specified operations, and / or may provide software to one or more processors to run thereon to perform one or more of the specified operations, including evaluating the image. Specifically, instructions for determining the bundle profile and surface may be performed by at least one evaluation device. Thus, as an example, one or more instructions may be implemented in software and / or hardware. Thus, as an example, the evaluation device may include one or more programmable means configured to perform the above evaluation, such as one or more computers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), or field-programmable gate arrays (FPGAs). However, additionally or alternatively, the evaluation device may also be embodied entirely or partially in hardware.

[0058] The evaluation device and the display device can be fully or partially integrated into a single device. Therefore, the evaluation device can often also form part of the display device. Alternatively, the evaluation device and the display device can be implemented as separate devices, either fully or partially. The display device may include further components.

[0059] The evaluation apparatus may be or may include one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs), and / or one or more data processing devices, such as one or more computers, preferably one or more microcomputers and / or microcontrollers, field-programmable arrays, or digital signal processors. Additional components may be included, such as one or more preprocessing devices and / or data acquisition devices, such as one or more devices for receiving and / or preprocessing sensor signals, such as one or more analog-to-digital converters and / or one or more filters. Further, the evaluation apparatus may include one or more measuring devices, such as one or more measuring devices for measuring current and / or voltage. Further, the evaluation apparatus may include one or more data storage devices. Further, the evaluation apparatus may include one or more interfaces, such as one or more wireless interfaces and / or one or more wired interfaces.

[0060] The evaluation device may be connected to or may include at least one further data processing device, which may be used for one or more of the following: display, visualization, analysis, distribution, communication, or further processing of information (such as information obtained by optical sensors and / or by the evaluation device). As an example, the data processing device may be connected to or include at least one of the following: a display, projector, monitor, LCD, TFT, speaker, multi-channel sound system, LED pattern, or further visualization device. The data processing device may also be connected to or include at least one of the following: a communication device or communication interface, connector, or port, capable of sending encrypted or unencrypted information using one or more of the following: email, text message, telephone, Bluetooth, Wi-Fi, infrared, or Internet interface, port, or connection. The data processing device may also be connected to or include at least one of the following: a processor; a graphics processor; a CPU; an Open Multimedia Application Platform (OMAP). TM Integrated circuits; systems-on-a-chip such as those from Apple's A-series or Samsung's S3C2 series; microcontrollers or microprocessors; one or more blocks of memory such as ROM, RAM, EEPROM, or flash memory; timing sources such as oscillators or phase-locked loops, counter timers, real-time timers, or power-on reset generators; voltage regulators; power management circuits; or DMA controllers. Individual units can also be connected by a bus (such as an AMBA bus) or integrated into IoT or Industry 4.0 type networks.

[0061] The evaluation device and / or data processing device may be connected to or have further external interfaces or ports, such as serial or parallel interfaces or ports, USB, parallel ports, FireWire, HDMI, Ethernet, Bluetooth, RFID, Wi-Fi, USART, or SPI, or analog interfaces or ports, such as ADCs, DACs, or standardized interfaces or ports to further devices (such as 2D camera devices using RGB interfaces, such as CameraLink). The evaluation device and / or data processing device may also be connected by one or more of the following: inter-processor interfaces or ports, FPGA-FPGA interfaces, or serial or parallel interface ports. The evaluation device and data processing device may also be connected to one or more of the following: optical disc drives, CD-RW drives, DVD+RW drives, flash drives, memory cards, disk drives, hard disk drives, solid-state drives, or solid-state drives.

[0062] The evaluation device and / or data processing device may be connected by one or more further external connectors or have one or more further external connectors, such as one or more of the following: telephone connector, RCA connector, VGA connector, male and female connector, USB connector, HDMI connector, 8P8C connector, BCN connector, IEC 60320C14 connector, fiber optic connector, D miniature connector, RF connector, coaxial connector, SCART connector, XLR connector, and / or may include at least one suitable receptacle for one or more of these connectors.

[0063] An evaluation device is configured to evaluate a first image. The evaluation of the first image includes identifying reflection features of the first image. The evaluation device can be configured to perform at least one image analysis and / or image processing to identify reflection features. The image analysis and / or image processing can use at least one feature detection algorithm. The image analysis and / or image processing can include one or more of the following: filtering; selection of at least one region of interest; forming a difference image between an image created by a sensor signal and at least one offset; inverting a sensor signal by inverting an image created by a sensor signal; forming a difference image between images created by a sensor signal at different times; background correction; decomposition into color channels; decomposition into hue, saturation, and luminance channels; frequency decomposition; singular value decomposition; application of a droplet detector; application of a corner detector; application of a Hessian filter determinant; application of a curvature-based region detector; application of a maximum stable extremum region detector; application of a generalized Hough transform; application of a ridge detector; application of an affine invariant feature detector; application of an affine adaptive interest point operator; application of a Harris affine region detector; application of a Hessian affine... Region of interest detector; application of scale-invariant feature transform; application of scale-space extremum detector; application of local feature detector; application of accelerated robust feature algorithm; application of gradient localization and orientation histogram algorithm; application of histogram with orientation gradient descriptor; application of Deriche edge detector; application of differential edge detector; application of spatiotemporal interest point detector; application of Moravec corner detector; application of Canny edge detector; application of Laplacian Gaussian filter; application of difference Gaussian filter; application of Sobel operator; application of Laplacian operator; application of Scharr operator; application of Prewitt operator; application of Roberts operator; application of Kirsch operator; application of high-pass filter; application of low-pass filter; application of Fourier transform; application of Radon transform; application of Hough transform; application of wavelet transform; thresholding; creation of binary image. Regions of interest can be determined manually by the user or automatically, such as by identifying features within an image generated by an optical sensor.

[0064] For example, the illumination source can be configured to generate and / or project a point cloud, such that multiple illuminated areas are generated on an optical sensor (e.g., a CMOS detector). Furthermore, interference can exist on the optical sensor, such as interference attributed to speckles and / or external light and / or multiple reflections. The evaluation apparatus can be adapted to determine at least one region of interest, for example, one or more pixels illuminated by the beam for determining the longitudinal coordinates of an object. For example, the evaluation apparatus can be adapted to perform filtering methods, such as speckle analysis and / or edge filtering and / or object recognition methods.

[0065] The evaluation apparatus can be configured to perform at least one image correction. Image correction may include at least one background subtraction. The evaluation apparatus may be adapted to remove the effects of background light from the beam profile, for example, through imaging without further illumination.

[0066] Each of the reflection features includes at least one beam profile. As used herein, the term "beam profile" for a reflection feature can generally refer to at least one intensity distribution of the reflection feature as a function of pixels, such as the intensity distribution of a light spot on an optical sensor. The beam profile can be selected from a linear combination of trapezoidal beam profiles, triangular beam profiles, conical beam profiles, and Gaussian beam profiles. The evaluation apparatus is configured to determine beam profile information for each of the reflection features by analyzing their beam profiles.

[0067] The evaluation apparatus is configured to determine at least one longitudinal coordinate z for each of the reflection features by analyzing their beam profiles. DPR As used herein, the term "beam profile analysis" can generally refer to the evaluation of the beam profile and may include at least one mathematical operation and / or at least one comparison and / or at least one symmetry and / or at least one filtering and / or at least one normalization. For example, beam profile analysis may include at least one of histogram analysis steps, calculation of difference metrics, application of neural networks, and application of machine learning algorithms. The evaluation apparatus can be configured to symmetricize and / or normalize and / or filter the beam profile, particularly to remove noise or asymmetry from recordings at large angles, recording edges, etc. The evaluation apparatus can filter the beam profile by removing high spatial frequencies, such as through spatial frequency analysis and / or median filtering. The summarization can be performed by averaging the intensity center of the spot and all intensities at the same distance from the center. The evaluation apparatus can be configured to normalize the beam profile to maximum intensity, particularly taking into account intensity differences attributable to recording distances. The evaluation apparatus can be configured to remove the effects of background light from the beam profile, for example, through imaging without illumination.

[0068] A reflection feature may cover at least one pixel of an image or may extend over at least one pixel of an image. For example, a reflection feature may cover multiple pixels or may extend over multiple pixels. An evaluation device may be configured to determine and / or select all pixels connected to and / or belonging to a reflection feature (e.g., a light spot). The evaluation device may be configured to determine the intensity center using the following formula:

[0069]

[0070] Where R coi It is the location of the intensity center, r pixel It is the pixel position, and Where j is the number of pixels connected to and / or belonging to the reflection feature, and I total It is the total intensity.

[0071] The evaluation device can be configured to determine the longitudinal coordinate z for each of the reflection features using a depth-from-photon-ratio technique (also known as beam profile analysis). DPR Regarding photon ratio ranging (DPR) technology, see WO 2018 / 091649 A1, WO 2018 / 091638 A1 and WO 2018 / 091640 A1, the entire contents of which are incorporated herein by reference.

[0072] The evaluation apparatus can be configured to determine the beam profile of each of the reflection features. As used herein, the term "determine beam profile" refers to identifying at least one reflection feature provided by an optical sensor and / or selecting at least one reflection feature provided by an optical sensor and evaluating at least one intensity distribution of the reflection feature. As an example, a region of a matrix can be used and evaluated to determine the intensity distribution, such as a three-dimensional intensity distribution or a two-dimensional intensity distribution, such as along an axis or line through the matrix. As an example, the illumination center of the beam can be determined, such as by determining at least one pixel with the highest illumination, and a cross-sectional axis through the illumination center can be selected. The intensity distribution can be an intensity distribution as a function of coordinates along this cross-sectional axis through the illumination center. Other evaluation algorithms are feasible.

[0073] Analysis of the beam profile, one of the reflection characteristics, may include determining at least one first region and at least one second region of the beam profile. The first region of the beam profile may be region A1, and the second region of the beam profile may be region A2. An evaluation device may be configured to integrate the first and second regions. The evaluation device may be configured to derive a combined signal, particularly a quotient Q, by one or more of the following: division of the integrated first and second regions, division of multiples of the integrated first and second regions, and division of a linear combination of the integrated first and second regions. The evaluation device may be configured to determine at least two regions of the beam profile and / or segment the beam profile into at least two segments comprising different regions of the beam profile, wherein overlap of regions may be possible, provided the regions are not congruent. For example, the evaluation device may be configured to determine multiple regions, such as two, three, four, five, or up to ten regions. The evaluation device may be configured to segment the beam spot into at least two regions of the beam profile and / or segment the beam profile into at least two segments comprising different regions of the beam profile. The evaluation device may be configured to determine the integration of the beam profile over the respective regions for at least two of the regions. The evaluation apparatus can be configured to compare at least two of the determined integrals. Specifically, the evaluation apparatus can be configured to determine at least one first region and at least one second region of the beam profile. As used herein, the term "region of beam profile" generally refers to any region of the beam profile at the location of the optical sensor used to determine the quotient Q. The first region and the second region of the beam profile can be one or both of adjacent or overlapping regions. The first region and the second region of the beam profile may not be congruent in area. For example, the evaluation apparatus can be configured to divide the sensor region of the CMOS sensor into at least two sub-regions, wherein the evaluation apparatus can be configured to divide the sensor region of the CMOS sensor into at least one left portion and / or at least one right portion and / or at least one upper portion and at least one lower portion and / or at least one inner portion and at least one outer portion. Additionally or alternatively, the display device may include at least two optical sensors, wherein the photosensitive areas of the first optical sensor and the second optical sensor can be arranged such that the first optical sensor is adapted to determine the first region of the beam profile of the reflection characteristics, and the second optical sensor is adapted to determine the second region of the beam profile of the reflection characteristics. The evaluation apparatus can be adapted to integrate the first and second regions. The evaluation apparatus can be configured to determine the vertical coordinate using at least one predetermined relationship between the quotient Q and the vertical coordinate. This predetermined relationship can be one or more of empirical, semi-empirical, and analytically derived relationships. The evaluation apparatus may include at least one data storage device for storing the predetermined relationship, such as a lookup list or lookup table.

[0074] The first region of the beam profile may include substantially edge information of the beam profile, and the second region of the beam profile includes substantially center information of the beam profile, and / or the first region of the beam profile may include information substantially about the left portion of the beam profile, and the second region of the beam profile includes information substantially about the right portion of the beam profile. The beam profile may have a center, i.e., the maximum value of the beam profile and / or the center point of the beam profile's flat top and / or the geometric center of the spot, and a descending edge extending from the center. The second region may include an inner region of the cross-section, and the first region may include an outer region of the cross-section. As used herein, the term "substantially center information" generally refers to a lower proportion of edge information (i.e., a lower proportion of the intensity distribution corresponding to the center) compared to the proportion of center information (i.e., the proportion of the intensity distribution corresponding to the center). Preferably, the center information has a proportion of less than 10%, more preferably less than 5%, of edge information, and most preferably, the center information does not include edge content. As used herein, the term "substantially edge information" generally refers to a lower proportion of center information compared to the proportion of edge information. Edge information may include information about the entire beam profile, particularly information from the center region and the edge region. The proportion of center information in the edge information is less than 10%, preferably less than 5%, and more preferably, the edge information does not include any center content. If at least one region of the bundle profile is close to or surrounds the center and includes substantially center information, that region can be determined and / or selected as a second region of the bundle profile. If at least one region of the bundle profile includes at least some portion of the descending edge of the cross section, that region can be determined and / or selected as a first region of the bundle profile. For example, the entire region of the cross section can be determined as the first region.

[0075] Other options for the first region A1 and the second region A2 are also possible. For example, the first region may include a substantially outer region of the bundle profile, and the second region may include a substantially inner region of the bundle profile. For example, in the case of a two-dimensional bundle profile, the bundle profile may be divided into a left portion and a right portion, wherein the first region may include a region of the substantially left portion of the bundle profile, and the second region may include a region of the substantially right portion of the bundle profile.

[0076] Edge information may include information relating to the number of photons in a first region of the beam profile, while center information may include information relating to the number of photons in a second region of the beam profile. An evaluation device may be configured to determine the area integral of the beam profile. The evaluation device may be configured to determine the edge information by integrating and / or summing the first region. The evaluation device may be configured to determine the center information by integrating and / or summing the second region. For example, the beam profile may be a trapezoidal beam profile, and the evaluation device may be configured to determine the integral of the trapezoid. Furthermore, when assuming a trapezoidal beam profile, the determination of the edge and center signals can be replaced by deriving the edge and center signals through geometric considerations using an equivalent evaluation of the characteristics of the trapezoidal beam profile (e.g., determining the slope and position of the edges and the height of the center flat top).

[0077] In one embodiment, A1 may correspond to the entire or complete area of ​​the feature point on the optical sensor. A2 may be the central area of ​​the feature point on the optical sensor. The central area may be a constant value. Compared to the entire area of ​​the feature point, the central area may be smaller. For example, in the case of a circular feature point, the central area may have a radius from 0.1 to 0.9 of the full radius of the feature point, preferably from 0.4 to 0.6 of the full radius.

[0078] In one embodiment, the illumination pattern may include at least one line pattern. A1 may correspond to a region having the full line width of the line pattern on the optical sensor (particularly the photosensitive area of ​​the optical sensor). Compared to the line pattern of the illumination pattern, the line pattern on the optical sensor may be widened and / or shifted to increase the line width on the optical sensor. In particular, in the case of a matrix of optical sensors, the line width of the line pattern on the optical sensor may vary from one column to another. A2 may be the central region of the line pattern on the optical sensor. The line width of the central region may be a constant value and may specifically correspond to the line width in the illumination pattern. The central region may have a smaller line width compared to the full line width. For example, the central region may have a line width from 0.1 to 0.9 of the full line width, preferably from 0.4 to 0.6 of the full line width. The line pattern may be segmented on the optical sensor. Each column of the matrix of optical sensors may include center information of the intensity in the central region of the line pattern and edge information of the intensity from the region extending further outward from the central region of the line pattern to the edge region.

[0079] In one embodiment, the illumination pattern may include at least a dot pattern. A1 may correspond to the region of the full radius of the dots in the dot pattern on the optical sensor. A2 may be the central region of the dots in the dot pattern on the optical sensor. The central region may be a constant value. The central region may have a radius compared to the full radius. For example, the central region may have a radius from 0.1 to 0.9 of the full radius, preferably from 0.4 to 0.6 of the full radius.

[0080] The illumination pattern may include at least one dot pattern and at least one line pattern. Other embodiments that are added to or replace the line and dot patterns are possible.

[0081] The evaluation device can be configured to derive the quotient Q by one or more of the following: division of the first and second regions, division of multiples of the first and second regions, and division of linear combinations of the first and second regions. The evaluation device can also be configured to derive the quotient Q using the following formula:

[0082]

[0083] Where x and y are the lateral coordinates, A1 and A2 are the first and second regions of the bundle profile, respectively, and E(x,y) represents the bundle profile.

[0084] Additionally or alternatively, the evaluation device may be adapted to determine one or both of center information and edge information from at least one slice or cut of the light spot. This can be achieved, for example, by replacing the area integral in the quotient Q with a line integral along the slice or cut. To improve accuracy, several slices or cuts of the light spot can be used and averaged. In the case of an elliptical light spot profile, averaging over several slices or cuts improves the distance information.

[0085] For example, in the case of an optical sensor having a pixel matrix, the evaluation device can be configured to evaluate the beam profile through the following steps:

[0086] - Identify the pixel with the highest sensor signal and form at least one center signal;

[0087] - Evaluate the sensor signals of the matrix and form at least one sum signal;

[0088] - The quotient Q is determined by combining the center signal and the signal; and

[0089] - Determine at least one longitudinal coordinate z of the object by evaluating the quotient Q.

[0090] As used herein, "sensor signal" generally refers to a signal generated by an optical sensor and / or at least one pixel of an optical sensor in response to illumination. Specifically, a sensor signal can be or may include at least one electrical signal, such as at least one analog electrical signal and / or at least one digital electrical signal. More specifically, a sensor signal can be or may include at least one voltage signal and / or at least one current signal. More specifically, a sensor signal may include at least one photocurrent. Further, the original sensor signal can be used, or the display device, optical sensor, or any other element can be adapted to process or preprocess the sensor signal to generate a secondary sensor signal that can also be used as the sensor signal, such as through preprocessing like filtering. The term "center signal" generally refers to at least one sensor signal that includes substantially center information of the bundle profile. As used herein, the term "maximum sensor signal" refers to one or both of a local maximum value or a maximum value in the region of interest. For example, a center signal can be the signal of a pixel having the highest sensor signal among a plurality of sensor signals generated by pixels of the entire matrix or a region of interest within the matrix, wherein the region of interest can be predetermined or determinable within an image generated by the pixels of the matrix. The center signal can originate from a single pixel or a group of optical sensors. In the latter case, for example, the sensor signals of the group of pixels can be added, integrated, or averaged to determine the center signal. The group of pixels from which the center signal originates can be a group of adjacent pixels, such as pixels having a distance less than a predetermined distance from the actual pixel having the highest sensor signal, or it can be a group of pixels generating sensor signals within a predetermined range from the highest sensor signal. The group of pixels from which the center signal originates can be selected to be as large as possible to allow for the maximum dynamic range. The evaluation apparatus can be adapted to determine the center signal by integrating multiple sensor signals, for example, multiple pixels surrounding the pixel having the highest sensor signal. For example, the beam profile can be a trapezoidal beam profile, and the evaluation apparatus can be adapted to determine the integral of the trapezoid (especially the flat top of the trapezoid).

[0091] As outlined above, the center signal can typically be a single sensor signal, such as a sensor signal from a pixel at the center of the spot, or it can be a combination of multiple sensor signals, such as a combination of sensor signals originating from a pixel at the center of the spot, or a secondary sensor signal derived by processing one or more of the aforementioned possibilities. The determination of the center signal can be performed electronically, as the comparison of sensor signals can be implemented fairly easily using conventional electronics, or it can be performed entirely or partially by software. Specifically, the center signal can be selected from the group including the following: the highest sensor signal; the average of a group of sensor signals within a predetermined tolerance range from the highest sensor signal; the average of sensor signals from a group of pixels including the pixel with the highest sensor signal and a predetermined adjacent pixel group; the sum of sensor signals from a group of pixels including the pixel with the highest sensor signal and a predetermined adjacent pixel group; the sum of a group of sensor signals within a predetermined tolerance range from the highest sensor signal; the average of a group of sensor signals greater than a predetermined threshold; the sum of a group of sensor signals greater than a predetermined threshold; the integral of sensor signals from a group of optical sensors including the optical sensor with the highest sensor signal and a predetermined adjacent pixel group; the integral of a group of sensor signals within a predetermined tolerance range from the highest sensor signal; and the integral of a group of sensor signals greater than a predetermined threshold.

[0092] Similarly, the term "sum signal" generally refers to a signal that includes essentially edge information of the bundle profile. For example, the sum signal can be derived by adding, integrating, or averaging sensor signals over a region of interest (ROI) within the entire matrix or the matrix, where the ROI can be predetermined or determinable within an image generated by the matrix's optical sensors. When adding, integrating, or averaging sensor signals, the actual optical sensors that generate the signals can be excluded from the addition, integration, or averaging, or can be included in the addition, integration, or averaging. The evaluation apparatus can be adapted to determine the sum signal by integrating the signal over the entire matrix or the ROI within the matrix. For example, the bundle profile can be a trapezoidal bundle profile, and the evaluation apparatus can be adapted to determine the integral over the entire trapezoid. Furthermore, when assuming a trapezoidal bundle profile, the determination of the edge and center signals can be substituted by deriving the edge and center signals through geometric considerations using an equivalent evaluation of the characteristics of the trapezoidal bundle profile (e.g., determining the slope and position of the edges and the height of the center flat top).

[0093] Similarly, center and edge signals can be determined by segmenting the bundle profile (such as circular segments). For example, the bundle profile can be divided into two segments by a secant or chord that does not pass through the center of the bundle profile. Thus, one segment essentially contains edge information, while the other essentially contains center information. For example, to further reduce the amount of edge information in the center signal, the edge signal can be further subtracted from the center signal.

[0094] The quotient Q can be a signal generated by combining a center signal and a sum signal. Specifically, the determination can include one or more of the following: forming a quotient of the center signal and the sum signal, or vice versa; forming a quotient of a multiple of the center signal and a multiple of the sum signal, or vice versa; forming a quotient of a linear combination of the center signals and a linear combination of the sum signals, or vice versa. Additionally or alternatively, the quotient Q can include any signal or combination of signals containing at least one piece of information regarding the comparison between the center signal and the sum signal.

[0095] As used herein, the term "longitudinal coordinate of an object" can refer to the distance between the optical sensor and the object. The evaluation apparatus can be configured to determine the longitudinal coordinate using at least one predetermined relationship between the quotient Q and the longitudinal coordinate. This predetermined relationship can be one or more of empirical, semi-empirical, and analytically derived relationships. The evaluation apparatus may include at least one data storage device for storing the predetermined relationship, such as a lookup list or lookup table.

[0096] The evaluation device can be configured to perform at least one photon ratio ranging algorithm that calculates the distances to all zero-order and higher-order reflectivity features.

[0097] The evaluation of the first image includes classifying the identified reflective features based on brightness. As used herein, the term "classification" can refer to assigning a sequence of reflective features based on brightness for further evaluation, specifically starting with reflective features having the highest brightness and subsequently decreasing brightness. As used herein, the term "brightness" can refer to the magnitude and / or intensity of reflective features in the first image. Brightness can refer to a defined passband, such as within the visible or infrared spectral range, or it can be wavelength-independent. Classification with decreasing brightness can refer to classification based on decreasing brightness and / or classification about decreasing brightness. If the brightest reflective feature is preferred for DPR calculation, the determination of the longitudinal coordinate z can be added. DPR The robustness of the reflection feature is mainly due to the fact that the reflection feature with a zero-order diffraction grating is always brighter than that with a higher-order spurious feature.

[0098] The evaluation device is configured to use the longitudinal coordinate z DPRThe reflection features are explicitly matched with their corresponding illumination features. The longitudinal coordinates determined using photon ratio ranging techniques can be used to solve the so-called correspondence problem. In that way, distance information for each reflection feature can be used to find correspondences within a known laser projector grid. As used herein, the term "matching" refers to identifying and / or determining and / or evaluating corresponding illumination and reflection features. As used herein, the term "corresponding illumination and reflection features" can refer to the fact that each of the illumination features in an illumination pattern produces a reflection feature at a scene location, wherein the produced reflection feature is assigned to the illumination feature that has already produced the reflection feature.

[0099] As used herein, the term “explicit match” can mean that only one reflection feature is assigned to a irradiation feature and / or no other reflection feature can be assigned to the same matched irradiation feature.

[0100] Irradiation features corresponding to reflection characteristics can be determined using epipolar geometry. For a description of epipolar geometry, see, for example, X. Jiang, H. Bunke: "Dreidimensionales Computersehen", Springer, Berlin Heidelberg. Chapter 2, 1997. Epipolar geometry can assume an irradiated image, i.e., an image of a non-distorted irradiated pattern, and the first image can be an image determined at different spatial locations and / or spatial orientations with fixed distances. The distance can be a relative distance, also represented as a baseline. The irradiated image can also be represented as a reference image. The evaluation apparatus can be adapted to determine epipolar lines in the reference image. The relative positions of the reference image and the first image can be known. For example, the relative positions of the reference image and the first image can be stored in at least one storage unit of the evaluation apparatus. The evaluation apparatus can be adapted to determine a straight line extending from a selected reflective feature of the first image to a real-world feature of its origin. Thus, the straight line can include possible object features corresponding to the selected reflective feature. The straight line and the baseline cross the epipolar plane. Since the reference image is determined at a different relative constellation than the first image, the corresponding possible object features can be imaged on the straight line (referred to as an epipolar line) in the reference image. The epipolar line can be the intersection of the epipolar plane and the reference image. Therefore, the features of the reference image corresponding to the selected features of the first image lie on the epipolar line.

[0101] Depending on the distance to the object in the scene that has reflected the illumination feature, the reflection feature corresponding to the illumination feature can be displaced within the first image. A reference image may include at least one displaced region where the illumination feature corresponding to the selected reflection feature will be imaged. The displaced region may include only one illumination feature. The displaced region may also include more than one illumination feature. The displaced region may include a epipolar line or a portion of an epipolar line. The displaced region may include one or more epipolar lines or multiple portions of one or more epipolar lines. The displaced region may extend along an epipolar line, orthogonally to an epipolar line, or both. The evaluation apparatus may be adapted to determine the illumination feature along an epipolar line. The evaluation apparatus may be adapted to determine the longitudinal coordinate z for the reflection feature and the error interval ±ε of the combined signal Q to determine the displaced region along an epipolar line corresponding to z±ε or orthogonally to an epipolar line. Measurement uncertainty using the distance measurement of the combined signal Q may result in a non-circular displaced region in the second image because the measurement uncertainty may be different for different directions. Specifically, the measurement uncertainty along one or more epipolar lines may be greater than the measurement uncertainty in the orthogonal direction of one or more epipolar lines. The displaced region may include a range in the orthogonal direction of one or more epipolar lines. The evaluation apparatus can be adapted to match a selected reflection feature with at least one illumination feature within the displacement region. The evaluation apparatus can be adapted to use a determined ordinate z. DPR At least one evaluation algorithm matches selected features of a first image with illumination features within a displacement region. The evaluation algorithm may be a linear scaling algorithm. The evaluation device may be adapted to determine the epipolar line closest to the displacement region and / or within the displacement region. The evaluation device may be adapted to determine the epipolar line closest to the image location of the reflection feature. The extent of the displacement region along the epipolar line may be greater than the extent of the displacement region orthogonal to the epipolar line. The evaluation device may be adapted to determine the epipolar line before determining the corresponding illumination feature. The evaluation device may determine the displacement region around the image location of each reflection feature. The evaluation device may be adapted to assign an epipolar line to each displacement region of each image location of the reflection feature, such as by assigning an epipolar line closest to the displacement region and / or closest to the displacement region within the displacement region and / or closest to the displacement region along the direction orthogonal to the epipolar line. The evaluation device may be adapted to determine the illumination feature corresponding to the reflection feature by determining the illumination feature closest to the assigned displacement region and / or closest to the assigned displacement region within the assigned displacement region and / or closest to the assigned displacement region along the assigned epipolar line and / or within the assigned displacement region along the assigned epipolar line.

[0102] Alternatively or additionally, the evaluation device may be configured to perform the following steps:

[0103] - Determine the displacement region for the image location of each reflection feature;

[0104] - Assign epipolar lines to the displacement regions of each reflection feature, such as by assigning epipolar lines that are closest to the displacement region and / or closest to the displacement region within the displacement region and / or along the direction orthogonal to the epipolar lines;

[0105] - Assign and / or determine at least one irradiation feature for each reflection feature, such as by assigning an irradiation feature that is closest to the assigned displacement region and / or within the assigned displacement region and / or closest to the assigned displacement region along the assigned nucleus line and / or within the assigned displacement region along the assigned nucleus line.

[0106] Additionally or alternatively, the evaluation device may be configured to make a decision among more than one epipolar line and / or illumination feature to assign to a reflection feature, such as by comparing the distances of the epipolar line and / or reflection features within the illumination image and / or by comparing the error-weighted distances of the epipolar line and / or illumination features within the illumination image, such as ε-weighted distances, and assigning the epipolar line and / or illumination feature within the shorter distance and / or ε-weighted distance to the illumination feature and / or reflection feature.

[0107] As outlined above, due to the diffraction grating, multiple reflection features are generated; for example, for each illumination feature, there is one true feature and multiple spurious features. Matching is performed by decreasing the brightness of the reflection features, starting with the brightest one. No other reflection feature can be assigned to the same matched illumination feature. Due to display artifacts, the resulting spurious features are typically darker than the true features. By classifying the reflection features by brightness, brighter reflection features are preferred for correspondence matching. If a correspondence for an illumination feature has already been used, a spurious feature cannot be assigned to an already used (i.e., matched) illumination feature.

[0108] The evaluation apparatus is configured to classify reflection features that match the illumination features as true features and reflection features that do not match the illumination features as false features. As used herein, the term "classification" can refer to assigning a reflection feature to at least one category. As used herein, the term "true feature" can refer to a zero-order reflection feature of a diffraction grating. As used herein, the term "false feature" can refer to a higher-order reflection feature of a diffraction grating, i.e., one with an order ≥ 1. A zero-order diffraction grating is always brighter than a false feature with a higher order.

[0109] The evaluation device is configured to reject false features and to use the vertical coordinate z DPR This is used to generate depth maps for realistic features. As used herein, the term "depth" can refer to the distance between an object and an optical sensor and can be given by a longitudinal coordinate. As used herein, the term "depth map" can refer to the spatial distribution of depth. Display devices can be used to generate 3D maps from scenes, such as faces.

[0110] Structured light methods typically use cameras and projectors with a fine grid of points (e.g., thousands of points). Well-known projector patterns are used to find correspondences between point patches on a scene. If the point correspondences are resolved, distance information is obtained through triangulation. If the camera is behind a display, diffraction spatially distorts the image. Therefore, finding point patterns on a distorted image is a challenging task. In contrast to structured light methods, this invention proposes using photon ratio ranging to evaluate the beam profile, which is not directly affected by the display's diffraction grating. Distortion does not affect the beam profile.

[0111] The depth map can be further refined using further depth measurement techniques, such as triangulation and / or defocus depth and / or structured light. The evaluation device can be configured to determine at least one second longitudinal coordinate z for each of the reflection features using triangulation and / or defocus depth and / or structured light techniques. triang .

[0112] The evaluation apparatus can be adapted to determine the displacement of the illumination feature and the reflection feature. The evaluation apparatus can be adapted to determine the displacement of the matching illumination feature and the selected reflection feature. The evaluation apparatus, such as at least one data processing unit of the evaluation apparatus, can be configured to determine the displacement of the illumination feature and the reflection feature, particularly by comparing the corresponding image positions of the illumination image and the first image. As used herein, the term "displacement" refers to the difference between the image position in the illumination image and the image position in the first image. The evaluation apparatus can be adapted to determine the second longitudinal coordinate of the matching feature using a predetermined relationship between the second longitudinal coordinate and the displacement. The evaluation apparatus can be adapted to determine the predetermined relationship using a triangulation method. Given that the position of the selected reflection feature and the position of the matching illumination feature in the first image and / or the relative displacement of the selected reflection feature and the matching illumination feature are known, the longitudinal coordinate of the corresponding object feature can be determined by triangulation. Therefore, the evaluation apparatus can be adapted, for example, to subsequently and / or sequentially select reflection features and use triangulation to determine the corresponding distance value for each potential position of the illumination feature. The displacement and the corresponding distance value can be stored in at least one storage device of the evaluation apparatus. As an example, the evaluation apparatus may include at least one data processing unit, such as at least one processor, at least one DSP, at least one FPGA, and / or at least one ASIC. Furthermore, to store at least one predetermined or determinable relationship between the second longitudinal coordinate z and the displacement, at least one data storage device may be provided, such as for providing one or more lookup tables for storing the predetermined relationship. The evaluation device may be adapted to store parameters for internal and / or external calibration of the camera and / or display device. The evaluation device may be adapted to generate parameters for internal and / or external calibration of the camera and / or display device, such as by performing Tsai camera calibration. The evaluation device may be adapted to calculate and / or estimate parameters such as the focal length of the transmission device, the radial lens distortion coefficient, the coordinates of the center of the radial lens distortion, the scaling factor to explain any uncertainties attributed to defects in the hardware timing used for scanning and digitization, the rotation angle of the transformation between world coordinates and camera coordinates, the translation component of the transformation between world coordinates and camera coordinates, the aperture angle, the image sensor format, the principal point, the skew coefficient, the camera center, the camera heading, the baseline, the rotation or translation parameters between the camera and / or the illumination source, the aperture, the focal length, etc.

[0113] The evaluation device can be configured to determine the second longitudinal coordinate z. triang and vertical coordinate z DPR The combined vertical coordinates. The combined vertical coordinates can be a second vertical coordinate z. triang and vertical coordinate z DPR The average value. Combining the longitudinal coordinates can be used to determine the depth map.

[0114] The display device may include a further illumination source. The further illumination source may include at least one light-emitting diode (LED). The further illumination source may be configured to generate light in the visible spectrum. An optical sensor may be configured to determine at least one second image comprising at least one two-dimensional image of the scene. The further illumination source may be configured to provide additional illumination for imaging the second image. For example, the setup of the display device may be extended by adding floodlight LEDs. The further illumination source may illuminate the scene, such as a face, with LEDs and, in particular, without an illumination pattern, and the optical sensor may be configured to capture the two-dimensional image. The 2D image may be used for face detection and verification algorithms. If the impulse response of the display is known, the distorted image captured by the optical sensor can be repaired. An evaluation device may be configured to determine at least one corrected image I0 by deconvolving the second image I with a raster function g, where I = I0*g. The raster function is also expressed as an impulse response. The undistorted image may be recovered using deconvolution methods, such as Van-Cittert or Wiener deconvolution. The display device may be configured to determine the raster function g. For example, the display device may be configured to illuminate a black scene with an illumination pattern comprising small, individual bright spots. The captured image can be a raster function. This process can be performed only once, such as during calibration. To determine a corrected image, even for imaging via a display, the display device can be configured to capture the image and use a deconvolution method on the captured impulse response g. The resulting image can be a reconstructed image with fewer display artifacts and can be used for a variety of applications, such as face recognition.

[0115] The evaluation apparatus can be configured to determine at least one material property m of an object by evaluating the beam profile of at least one of the reflection features, preferably the beam profiles of multiple reflection features. For details regarding the determination of at least one material property by evaluating the beam profile, refer to WO 2020 / 187719, the contents of which are incorporated herein by reference.

[0116] As used herein, the term "material property" refers to at least one arbitrary property of a material configured for characterizing and / or identifying and / or classifying the material. For example, a material property can be a property selected from: roughness, depth of light penetration through the material, properties characterizing the material as biological or non-biological, reflectivity, specular reflectivity, diffuse reflectivity, surface properties, translucency, scattering, specifically backscattering behavior, etc. At least one material property can be a property selected from: scattering coefficient, translucency, transparency, deviation from Lambertian surface reflection, speckle, etc. As used herein, the term "identifying at least one material property" means one or more of the following: determining the material property and assigning the material property to an object. The evaluation apparatus may include at least one database comprising lists and / or tables of predefined and / or predetermined material properties, such as lookup lists or lookup tables. The list and / or table of material properties can be determined and / or generated by performing at least one test measurement using the display device according to the invention, for example, by performing material testing using a sample having known material properties. The list and / or table of material properties can be determined and / or generated at the manufacturer's site and / or by the user of the display device. Material properties can be additionally assigned to a material classifier, such as one or more of the following: material name, material group, such as biological or non-biological material, translucent or non-translucent material, metallic or non-metallic, skin or non-skin, fur or non-fur, carpet or non-carpet, reflective or non-reflective, specular or non-specular, foam or non-foam, hair or non-hair, roughness group, etc. The evaluation apparatus may include at least one database comprising lists and / or tables that include material properties and associated material names and / or material groups.

[0117] For example, to avoid being bound by this theory, human skin may have a reflective profile, also represented as a backscattering profile, comprising the portion produced by back reflection from the surface, represented as surface reflection, and the portion produced by diffuse reflection of light penetrating the skin, represented as the diffuse reflection portion of back reflection. For a description of the reflective profile of human skin, see “Lasertechnik in der Medizin: Grundlagen, Systeme, Anwendungen”, “Wirkung von Laserstrahlung auf Gewebe”, 1991, pp. 10171-266, Jürgen Eichler, Theo Seiler, Springer Verlag, ISBN 0939-0979. The surface reflectance of skin may increase with increasing wavelength towards the near-infrared. Furthermore, the penetration depth may increase with increasing wavelength from visible light to near-infrared. The diffuse reflection portion of back reflection may increase with the penetration depth of light. These properties can be used to distinguish skin from other materials by analyzing the backscattering profile.

[0118] Specifically, the evaluation device can be configured to compare the beam profile of a reflection feature (also referred to as a reflection beam profile) with at least one predetermined and / or pre-recorded and / or predefined beam profile. The predetermined and / or pre-recorded and / or predefined beam profile can be stored in a table or lookup table and can be determined empirically, for example, and, as an example, can be stored in at least one data storage device of the display device. For example, the predetermined and / or pre-recorded and / or predefined beam profile can be determined during the initial startup of a mobile device including the display device. For example, the predetermined and / or pre-recorded and / or predefined beam profile can be stored in at least one data storage device of the mobile device, for example, via software, specifically via an application downloaded from an app store, etc. If the reflection beam profile and the predetermined and / or pre-recorded and / or predefined beam profile are identical, the reflection feature can be identified as being generated by biological tissue. The comparison may include overlaying the reflection beam profile and the predetermined or predefined beam profile such that their intensity centers match. The comparison may include determining the deviation between the reflection beam profile and the predetermined and / or pre-recorded and / or predefined beam profile, such as the sum of squares of point-to-point distances. The evaluation device can be configured to compare a determined deviation with at least one threshold, wherein if the determined deviation is less than and / or equal to the threshold, the surface is indicated as biological tissue and / or the detection of biological tissue is confirmed. The threshold can be stored in a table or lookup table and can be determined empirically, for example, and, as an example, can be stored in at least one data storage device of the display device.

[0119] Additionally or alternatively, to identify whether reflective features are generated by biological tissue, the evaluation device can be configured to apply at least one image filter to the image of the region. As further used herein, "image" refers to a two-dimensional function f(x,y), where a brightness and / or color value is given for any x,y location in the image. This can correspond to discretizing the location in terms of recorded pixels. It can correspond to discretizing the brightness and / or color in terms of the bit depth of an optical sensor. As used herein, the term "image filter" refers to at least one mathematical operation applied to at least one specific region of the beam profile and / or beam profile. Specifically, the image filter Ф maps the image f or the region of interest in the image to a real number, Ф(f(x,y)) = φ, where φ represents a feature, particularly a material feature. The image may be subject to noise, and the same applies to the feature. Therefore, the feature can be a random variable. The feature can be normally distributed. If the feature is not normally distributed, it can be converted to a normal distribution, such as through a Box-Cox transform.

[0120] The evaluation device can be configured to determine at least one material feature φ by applying at least one material-related image filter Ф2 to an image. 2m As used herein, the term "material-correlated" image filter refers to an image with material-correlated output. The output of a material-correlated image filter in this paper represents "material feature φ". 2m "or "material-related characteristics φ" 2m The material characteristics may be, or may include, at least one piece of information about at least one material property of the surface of the area where the reflective characteristics have been generated.

[0121] Material-related image filters can be at least one filter selected from the following: brightness filter; speckle shape filter; square norm gradient; standard deviation; smoothness filter, such as a Gaussian filter or median filter; contrast filter based on gray-level occurrence; energy filter based on gray-level occurrence; homogeneity filter based on gray-level occurrence; dissimilarity filter based on gray-level occurrence; law-based energy filter; threshold area filter; or linear combinations thereof; or further material-related image filters. 2other The further material-related image filter Ф 2other pass It is related to one or more of the following, or linear combinations thereof: brightness filter, speckle shape filter, square norm gradient, standard deviation, smoothness filter, energy filter based on gray-level occurrence, homogeneity filter based on gray-level occurrence, dissimilarity filter based on gray-level occurrence, law-based energy filter, or threshold region filter, where Ф mIt is one of the following: a brightness filter, a speckle shape filter, a square norm gradient filter, a standard deviation filter, a smoothness filter, an energy filter based on gray-level occurrence, a homogeneity filter based on gray-level occurrence, a dissimilarity filter based on gray-level occurrence, a law-based energy filter, or a threshold region filter, or a linear combination thereof. This further describes the material-related image filter Ф. 2other It can be done Preferably through One or more of the material-related image filters Фm are associated.

[0122] The material-dependent image filter can be at least one arbitrary filter φ that passes the hypothesis test. As used herein, the term "passes the hypothesis test" means the fact that the null hypothesis H0 is rejected and the alternative hypothesis H1 is accepted. The hypothesis test may include testing the material-dependent nature of the image filter by applying it to a predefined dataset. The dataset may include multiple beam profile images. As used herein, the term "beam profile image" refers to... The sum of Gaussian radial basis functions,

[0123]

[0124]

[0125] in, Each of the Gaussian radial basis functions is derived from the center. Prefactor and exponential factors Definition. The exponential factor is the same for all Gaussian functions across all images. Center position. , For all images They are all the same. Each beam profile image in the dataset corresponds to a material classifier and a distance. The material classifier can be a label, such as "material A", "material B", etc. This can be achieved by using the above-mentioned methods... The formula, combined with the following parameter table, generates the beam profile image:

[0126]

[0127] , The value is an integer corresponding to a pixel, where The image can have a pixel size of 32x32. A dataset of beam contour images can be obtained by using the methods described above. The formula is generated by combining the parameter set to obtain A continuous description. The value of each pixel in a 32x32 image can be obtained by... The values ​​for 𝑥 and 𝑦 are obtained by inserting integer values ​​from 0, …, 31. For example, for pixel (6, 9), the value can be calculated. .

[0128] Then, for each image It is possible to calculate the eigenvalues ​​corresponding to filter Φ. , , where z k It corresponds to an image from a predefined dataset. The distance value. This produces the corresponding eigenvalues. The dataset. Hypothesis testing can use a null hypothesis that the filter does not distinguish between material classifiers. The null hypothesis can be H0: Given, where, It corresponds to the eigenvalue Expected value for each material group. Index Represents a group of materials. The hypothesis test can be used as an alternative hypothesis for the filter to distinguish between at least two material classifiers. The alternative hypothesis can be derived from H1: As used herein, the term "not distinguishable between material classifiers" means that the expected values ​​of the material classifiers are the same. As used herein, the term "distinguishing material classifiers" means that at least two of the expected values ​​of the material classifiers are different. As used herein, "distinguishing at least two material classifiers" is used synonymously with "suitable material classifier". Hypothesis testing may include at least one analysis of variance (ANOVA) on the generated eigenvalues. In particular, hypothesis testing may include determining each The average value of the material's characteristic values, i.e., the total average value, For , in, Give the number of eigenvalues ​​for each material J in a predefined dataset. Hypothesis testing may include determining all... The average of the eigenvalues Hypothesis testing may include determining the mean sum of squares within the following scope:

[0129] .

[0130] Assume the test may include the average sum of squares among the following:

[0131] .

[0132] Assume that the test may include performing the F test:

[0133] o

[0134] o

[0135] o

[0136] In this article, It is a regularized incomplete beta function. The Euler-Beta function ,as well as It is an incomplete beta function. The image filter passes the hypothesis test if the p-value is less than or equal to a predefined significance level. The filter passes the hypothesis test if p ≤ 0.075, preferably p ≤ 0.05, more preferably p ≤ 0.025, and most preferably p ≤ 0.01. For example, with a predefined significance level of α = 0.075, if the p-value is less than α = 0.075, the image filter passes the hypothesis test. In this case, the null hypothesis H0 can be rejected, and the alternative hypothesis H1 can be accepted. Therefore, the image filter distinguishes at least two material classifiers. Thus, the image filter passes the hypothesis test.

[0137] In the following description, we assume that the reflection image includes at least one reflection feature, particularly a blob image, and then describe the image filter. Blob image Can be derived from a function Given, where the background of image f may have been subtracted. However, other reflection features may be possible.

[0138] For example, a material-related image filter can be a brightness filter. A brightness filter can return a measure of the brightness of a spot as a material feature. The material feature can be determined by the following formula:

[0139] ,

[0140] Where f is the speckle image. The distance between the specks is represented by z, where z can be obtained, for example, by using defocus ranging or photon ratio ranging techniques and / or by using triangulation techniques. The surface normal of the material is... Given and obtainable as the normal to the surface spanned by at least three measurement points. Vector It is the direction vector of the light source. Since the position of the spot is known using defocus ranging or photon ratio ranging techniques and / or using triangulation techniques, where the position of the light source is known as a parameter of the display device, therefore... It is the difference vector between the spot and the position of the light source.

[0141] For example, a material-related image filter could be a filter with an output that depends on the shape of the speckle. This material-related image filter might return a value related to the translucency of the material as a material feature. The translucency of the material affects the shape of the speckle. The material feature can be given by the following formula:

[0142] ,

[0143] Where 0 < α, β < 1 are the weights of the speckle height h, and H represents the symmetry function, i.e., H(x) = 1 : x ≥ 0, H(x) = 0 : x < 0. The speckle height h can be determined by the following formula:

[0144] ,

[0145] Among them, B r It is the inner circle of a spot with radius r.

[0146] For example, a material-related image filter can be a squared norm gradient. This material-related image filter can return values ​​as material features related to measures of the soft and hard transitions and / or roughness of the speckle. Material features can be defined by the following formula:

[0147] .

[0148] For example, a material-related image filter can be a standard deviation. The standard deviation of a blob can be determined by the following formula:

[0149] ,

[0150] Where μ is composed of The given average value.

[0151] For example, a material-related image filter can be a smoothness filter, such as a Gaussian filter or a median filter. In one embodiment of the smoothness filter, the image filter may reference observations of volume scattering exhibiting less speckle contrast compared to diffuse scattering materials. This image filter can quantify the smoothness of the speckle corresponding to the speckle contrast as a material feature. The material feature can be determined by the following formula:

[0152] ,

[0153] in, This is a smoothness function, such as a median filter or a Gaussian filter. The image filter can include division by a distance z, as described in the formula above. The distance z can be determined, for example, using defocus ranging or photon ratio ranging techniques and / or by using triangulation techniques. This allows the filter to be distance-sensitive. In one embodiment of the smoothness filter, the smoothness filter can be based on the standard deviation of the extracted speckle noise pattern. The speckle noise pattern N can be empirically described by the following formula:

[0154] ,

[0155] in, It is an image with speckled spots removed. This is the noise term used to model the speckle pattern. Computing a despeckle image can be computationally difficult. Therefore, a despeckle image can be approximated using a smoothed version of f, i.e. ,in, It is a smoothing operator similar to a Gaussian filter or a median filter. Therefore, the approximation of the blob pattern can be given by the following equation:

[0156] .

[0157] The material characteristics of the filter can be determined by the following formula:

[0158] ,

[0159] Where Var represents the variance function.

[0160] For example, an image filter could be a contrast filter based on grayscale occurrence. This material filter could be based on a grayscale occurrence matrix. ,and It is the occurrence rate of grayscale combinations. Furthermore, the relation ρ defines the distance between (x1,y1) and (x2,y2) as ρ(x,y)=(x+a,y+b), where a and b are selected from 0 and 1.

[0161] The material characteristics of a contrast filter based on grayscale appearance can be given by the following formula:

[0162] .

[0163] For example, an image filter can be an energy filter based on grayscale occurrence. This material filter is based on the grayscale occurrence matrix defined above.

[0164] The material characteristics of an energy filter based on grayscale appearance can be given by the following formula:

[0165] .

[0166] For example, an image filter can be a homogeneous filter based on grayscale occurrence. This material filter is based on the grayscale occurrence matrix defined above.

[0167] The material characteristics of a homogeneous filter based on grayscale appearance can be given by the following formula:

[0168] .

[0169] For example, an image filter can be a dissimilarity filter based on grayscale occurrence. This material filter is based on the grayscale occurrence matrix defined above.

[0170] The material characteristics of the phase filter based on the appearance of gray levels can be given by the following formula:

[0171] .

[0172] For example, an image filter could be a law-based energy filter. This material filter could be based on the law vectors L5=[1,4,6,4,1] and E5=[-1,-2,0,-2,-1], and the matrix L5(E5). T And E5 (L5) T .

[0173] Image f k Convolution with these matrices:

[0174]

[0175] as well as

[0176] .

[0177]

[0178] The material characteristics of the energy filter according to the law can be determined by the following formula:

[0179] .

[0180] For example, a material-related image filter can be a threshold region filter. The material feature may be associated with two regions in the image plane. The first region Ω1 can be a region where the function f is greater than a multiple of the maximum value f by α. The second region Ω2 can be a region where the function f is less than a multiple of the maximum value f by α but greater than a threshold ε of the maximum value f. Preferably, α can be 0.5, and ε can be 0.05. Due to speckle or noise, the region may not only correspond to an inner and outer circle around the center of the light spot. As an example, Ω1 may include speckle or unconnected regions in the outer circle. The material feature can be determined by the following formula:

[0181] ,

[0182] in, and .

[0183] The evaluation device can be configured to use material characteristic φ 2m The material properties of the surface with the reflective features are determined by at least one predetermined relationship between the material properties and the surface with the reflective features. This predetermined relationship can be one or more of empirical, semi-empirical, and analytically derived relationships. The evaluation apparatus may include at least one data storage device, such as a lookup list or lookup table, for storing the predetermined relationships.

[0184] The evaluation device is configured to identify a reflective feature as being generated by irradiating biological tissue if its corresponding material properties meet at least one predetermined or predefined criterion. If the material properties indicate "biological tissue," the reflective feature can be identified as being generated by biological tissue. If the material properties are below or equal to at least one threshold or range, the reflective feature can be identified as being generated by biological tissue, wherein the reflective feature is identified as being generated by biological tissue and / or the detection of biological tissue is confirmed if a determined deviation is below and / or equal to the threshold. At least one threshold and / or range can be stored in a table or lookup table and can be determined empirically, for example, and, as an example, can be stored in at least one data storage device of the display device. The evaluation device is configured to otherwise identify the reflective feature as background. Therefore, the evaluation device can be configured to assign depth information and material properties, such as whether skin is present or not, to each projected light spot.

[0185] After determining the vertical coordinate z, φ can be evaluated subsequently. 2m To determine material properties, information about the longitudinal coordinate z can be considered for evaluating φ. 2m .

[0186] In another aspect, the present invention discloses a method for depth measurement using a semi-transparent display, wherein a display device according to the present invention is used. The method includes the following steps:

[0187] a) Projecting at least one illumination pattern comprising multiple illumination features onto at least one scene using at least one illumination source, wherein the illumination source is placed in the propagation direction of the illumination pattern in front of the display;

[0188] b) By using at least one optical sensor, in response to illumination by the illumination features, determining at least one first image comprising a plurality of reflection features generated by the scene, wherein the optical sensor has at least one photosensitive area, wherein the optical sensor is placed in the propagation direction of the illumination pattern in front of the display, wherein each of the reflection features comprises at least one beam profile.

[0189] c) Evaluating the first image using at least one evaluation device, wherein the evaluation includes the following sub-steps:

[0190] C1) Identify the reflection features of the first image and classify the identified reflection features according to brightness.

[0191] C2) Determine at least one longitudinal coordinate z for each of the reflection features by analyzing their beam profiles. DPR ;

[0192] C3) By using the vertical coordinate z DPR The reflection features are explicitly matched with their corresponding illumination features, wherein the matching is performed by decreasing the brightness of the reflection features, starting with the brightest ones.

[0193] C4) Classify reflection features that match the illumination features as true features, and classify reflection features that do not match the illumination features as false features;

[0194] C5) Reject the false feature and use the vertical coordinate z DPR To generate a depth map for the real features.

[0195] The method steps can be performed in a given order or in a different order. Furthermore, one or more additional method steps not listed may be present. Additionally, one, more than one, or even all method steps may be performed repeatedly. For details, options, and definitions, refer to the display device discussed above. Therefore, specifically, as described above, the method may include the use of a display device according to the invention (such as according to one or more embodiments given above or given in more detail below).

[0196] The at least one evaluation device can be configured to execute at least one computer program, such as being configured to perform or support one or more, or even all, of the method steps according to the method of the invention. As an example, one or more algorithms that can determine the location of an object can be implemented.

[0197] In a further aspect of the invention, uses of the detector according to the invention are proposed, such as one or more of the embodiments given above or given in more detail below, selected for the purpose of use from the group consisting of: location measurement in traffic technology; entertainment applications; security applications; surveillance applications; safety applications; human-machine interface applications; tracking applications; photographic applications; imaging applications or camera applications; map-building applications for generating maps of at least one space; homing or tracking beacon detectors for vehicles; outdoor applications; mobile applications; communication applications; machine vision applications; robotic applications; quality control applications; manufacturing applications.

[0198] For further uses of the display device and apparatus of the present invention, see WO 2018 / 091649 A1, WO2018 / 091638 A1 and WO 2018 / 091640 A1, the contents of which are incorporated herein by reference.

[0199] In general, the following embodiments are considered preferred in the context of this invention:

[0200] Example 1: A display device, comprising:

[0201] - At least one illumination source configured to project at least one illumination pattern comprising multiple illumination features onto at least one scene;

[0202] - At least one optical sensor having at least one photosensitive area, wherein the optical sensor is configured to determine at least one first image, the first image including a plurality of reflection features generated by the scene in response to illumination by the illumination features;

[0203] - At least one semi-transparent display configured to display information, wherein the illumination source and the optical sensor are placed in front of the display in the direction of propagation of the illumination pattern;

[0204] - At least one evaluation device, wherein the evaluation device is configured to evaluate the first image, wherein the evaluation of the first image includes identifying the reflection features of the first image and classifying the identified reflection features with respect to brightness, wherein each of the reflection features includes at least one beam profile, and wherein the evaluation device is configured to determine at least one longitudinal coordinate z for each of the reflection features by analyzing their beam profiles. DPR ,

[0205] The evaluation device is configured to use the longitudinal coordinate z DPR The reflection features are explicitly matched with corresponding illumination features, wherein the matching is performed by decreasing the brightness of the reflection features starting from the brightest reflection feature. The evaluation device is configured to classify reflection features that match the illumination features as true features and reflection features that do not match the illumination features as false features. The evaluation device is configured to reject the false features and to use the vertical coordinate z... DPR To generate a depth map for the real features.

[0206] Example 2: A display device according to the previous embodiment, wherein the evaluation device is configured to determine at least one second longitudinal coordinate z for each of the reflective features using triangulation and / or defocus depth and / or structured light techniques. triang .

[0207] Example 3: The display device according to the previous embodiment, wherein the evaluation device is configured to determine the second longitudinal coordinate z. triang and the vertical coordinate z DPR The combined vertical coordinates, wherein the combined vertical coordinates are the second vertical coordinate z triangand the vertical coordinate z DPR The average value of the combined longitudinal coordinates is used to determine the depth map.

[0208] Example 4: A display device according to any of the foregoing embodiments, wherein the illumination source includes at least one laser projector, wherein the laser projector includes at least one laser source and at least one diffractive optical element (DOE).

[0209] Example 5: A display device according to any of the foregoing embodiments, wherein the illumination source is configured to generate at least one light beam having a beam path from the illumination source through the display to the scene, wherein the display is configured to function as a grating such that the light beam undergoes diffraction by the display resulting in the dot pattern.

[0210] Example 6: A display device according to the previous embodiment, wherein the wiring of the display is configured to form gaps and / or slits and ridges of the grating.

[0211] Example 7: A display device according to any of the foregoing embodiments, wherein the illumination pattern comprises a periodic dot pattern.

[0212] Example 8: A display device according to any of the foregoing embodiments, wherein the illumination pattern has a low dot density, wherein the illumination pattern has ≤ 2500 dots per field of view.

[0213] Example 9: A display device according to any of the foregoing embodiments, wherein the evaluation device is configured to determine beam profile information for each of the reflection features by using photon ratio ranging technology.

[0214] Example 10: A display device according to any of the foregoing embodiments, wherein the optical sensor includes at least one CMOS sensor.

[0215] Example 11: A display device according to any of the foregoing embodiments, wherein the display device includes a further illumination source, wherein the further illumination source includes at least one light-emitting diode (LED).

[0216] Example 12: A display device according to the previous embodiment, wherein the further irradiation source is configured to generate light in the visible spectrum range.

[0217] Example 13: A display device according to any one of the foregoing two embodiments, wherein the optical sensor is configured to determine at least one second image including at least one two-dimensional image of the scene, wherein the further illumination source is configured to provide additional illumination for imaging the second image.

[0218] Example 14: A display device according to the previous embodiment, wherein the evaluation device is configured to determine at least one corrected image I0 by deconvolving the second image I with a raster function g, wherein I = I0*g.

[0219] Example 15: A method for depth measurement via a semi-transparent display, wherein at least one display device according to any of the foregoing embodiments is used, wherein the method includes the following steps:

[0220] a) Projecting at least one illumination pattern comprising multiple illumination features onto at least one scene using at least one illumination source, wherein the illumination source is positioned in the direction of propagation of the illumination pattern in front of the display;

[0221] b) Determine at least one first image by using at least one optical sensor, the first image including a plurality of reflection features generated by the scene in response to illumination by the illumination features, wherein the optical sensor has at least one photosensitive area, wherein the optical sensor is placed in the propagation direction of the illumination pattern in front of the display, wherein each of the reflection features includes at least one beam profile.

[0222] c) Evaluating the first image using at least one evaluation device, wherein the evaluation includes the following sub-steps:

[0223] C1) Identify the reflection features of the first image and classify the identified reflection features according to brightness.

[0224] C2) Determine at least one longitudinal coordinate z for each of the reflection features by analyzing their beam profiles. DPR ;

[0225] C3) By using the vertical coordinate z DPR The reflection features are explicitly matched with their corresponding illumination features, wherein the matching is performed by decreasing the brightness of the reflection features, starting with the brightest ones.

[0226] C4) Classify reflection features that match the illumination features as true features, and classify reflection features that do not match the illumination features as false features;

[0227] C5) Reject the false feature and use the vertical coordinate z DPR To generate a depth map for the real features.

[0228] Example 16: An application of the display device according to any of the foregoing embodiments relating to a display device, for the purpose of use, selected from the group consisting of: location measurement in traffic technology; entertainment applications; security applications; surveillance applications; safety applications; human-machine interface applications; tracking applications; photography applications; imaging applications or camera applications; map building applications for generating maps of at least one space; homing or tracking beacon detectors for vehicles; outdoor applications; mobile applications; communication applications; machine vision applications; robotic applications; quality control applications; manufacturing applications. Attached Figure Description

[0229] Other optional details and features of the invention will become apparent from the following description of preferred exemplary embodiments in conjunction with the dependent claims. In such cases, a particular feature may be implemented separately or in combination with other features. The invention is not limited to exemplary embodiments. Exemplary embodiments are schematically illustrated in the accompanying drawings. The same reference numerals in the various drawings refer to the same elements or elements having the same function, or elements that correspond to each other in terms of their function.

[0230] Specifically, in the diagram:

[0231] Figure 1A and Figure 1B An embodiment of the display device according to the present invention is shown;

[0232] Figures 2A to 2C An embodiment is shown that utilizes at least one optical sensor of a display device to determine a first image;

[0233] Figures 3A to 3C A further embodiment of a first image determined using at least one optical sensor of a display device is shown;

[0234] Figure 4 The diagram shows a 2D image with correction determined using a display device; and

[0235] Figures 5A to 5C The images shown are distorted 2D images captured with a monitor, 2D images captured without a monitor, and corrected 2D images. Detailed Implementation

[0236] Figure 1AAn embodiment of the display device 110 according to the present invention is shown in a highly schematic manner. The display device 110 includes at least one semi-transparent display 112 configured to display information. The display 112 can be any shaped device configured to display information items such as at least one image, at least one chart, at least one histogram, at least one text, or at least one symbol. The display 112 can be at least one monitor or at least one screen. The display 112 can have any shape, preferably a rectangular shape. For example, the display device 110 can be at least one device selected from the group consisting of: a television device, a smartphone, a game console, a personal computer, a laptop computer, a tablet computer, at least one virtual reality device, or a combination thereof.

[0237] The display device 110 includes at least one illumination source 114 configured to project at least one illumination pattern comprising multiple illumination features onto at least one scene. The scene may be an object or a spatial region, such as a face. The scene may include at least one object and its surrounding environment.

[0238] Illumination source 114 can be adapted to directly or indirectly illuminate a scene, wherein the illumination pattern is reflected or scattered by the surface of the scene and is therefore at least partially guided toward an optical sensor. Illumination source 114 can be adapted to illuminate the scene, for example, by guiding a beam toward the scene that reflects the beam. Illumination source 114 can be configured to generate an illumination beam for illuminating the scene.

[0239] Illumination source 114 may include at least one light source. Illumination source 114 may include multiple light sources. Illumination source 114 may include artificial illumination sources, particularly at least one laser source and / or at least one incandescent lamp and / or at least one semiconductor light source, such as at least one light-emitting diode, particularly organic and / or inorganic light-emitting diodes. As an example, the light emitted by the illumination source may have a wavelength of 300 to 1000 nm (especially 500 to 1000 nm). Additionally or alternatively, light in the infrared spectral range may be used, such as in the range of 780 nm to 3.0 μm. Specifically, light in a portion of the near-infrared region may be used, in which silicon photodiodes are specifically suited for the range of 700 nm to 1100 nm. Illumination source 114 may be configured to generate at least one illumination pattern in the infrared region. Using light in the near-infrared region allows the light to be detected by the human eye only weakly and still detectable by silicon sensors, particularly standard silicon sensors. Illumination source 114 may be configured to emit light at a single wavelength. Specifically, the wavelength may be in the near-infrared region. In other embodiments, the illumination may be adapted to emit light with multiple wavelengths, thereby allowing additional measurements to be performed in other wavelength channels.

[0240] Irradiation source 114 may be or may include at least one multi-beam light source. For example, irradiation source 114 may include at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the irradiation source may include at least one laser and / or laser source. Various types of lasers can be used, such as semiconductor lasers, dual heterostructure lasers, external cavity lasers, split-confined heterostructure lasers, quantum cascade lasers, distributed Bragg reflector lasers, polaron lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, indium arsenide lasers, transistor lasers, diode-pumped lasers, distributed feedback lasers, quantum well lasers, interband cascade lasers, gallium arsenide lasers, semiconductor ring lasers, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alternatively, non-laser light sources, such as LEDs and / or bulbs, may be used. The irradiation source may include one or more diffractive optical elements (DOEs) adapted to generate an irradiation pattern. For example, the illumination source 114 may be adapted to generate and / or project a point cloud. For instance, the illumination source may include one or more of the following: at least one digital light processing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one light-emitting diode array; at least one laser source array. Using at least one laser source as the illumination source 114 is particularly preferred due to its generally defined beam profile and other characteristics of operability. The illumination source 114 may be integrated into the housing 116 of the display device 110.

[0241] Furthermore, the illumination source 114 can be configured to emit modulated or unmodulated light. When using multiple illumination sources 114, the different sources can have different modulation frequencies, as further detailed below, which can then be used to distinguish the light beams.

[0242] The illumination pattern can be at least one arbitrary pattern, comprising at least one illumination feature suitable for at least a portion of the illumination scene. The illumination pattern may include a single illumination feature. The illumination pattern may include multiple illumination features. The illumination pattern may be selected from the group consisting of: at least one dot pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; at least one pattern comprising an arrangement of periodic or aperiodic features. The illumination pattern may include regular and / or constant and / or periodic patterns, such as triangular patterns, rectangular patterns, hexagonal patterns, or patterns comprising further convex tiles. The illumination pattern may exhibit at least one illumination feature selected from the group consisting of: at least one dot; at least one line; at least two lines, such as parallel or intersecting lines; at least one dot and one line; at least one arrangement of periodic or aperiodic features; at least one arbitrarily shaped feature. The illumination pattern may include at least one pattern selected from the group consisting of: at least one point pattern, particularly a pseudo-random point pattern; a random point pattern or a quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern including at least one predictable feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern; at least one pattern including convex uniform tiles; at least one line pattern including at least one line; at least one line pattern including at least two lines, such as parallel or intersecting lines. For example, the illumination source may be adapted to generate and / or project a point cloud. Illumination source 114 may include at least one light projector adapted to generate a point cloud such that the illumination pattern may include multiple point patterns. Illumination source 114 may include at least one mask adapted to generate the illumination pattern from at least one light beam generated by illumination source 114.

[0243] The distance between two features of the illumination pattern and / or the area of ​​at least one illumination feature may depend on the blurring circle in the image. As outlined above, the illumination source may include at least one light source configured to generate at least one illumination pattern. Specifically, illumination source 114 includes at least one light source and / or at least one laser diode designated for generating laser radiation. Illumination source 114 may include at least one diffractive optical element (DOE). Display device 110 may include at least one dot projector, such as at least one laser source and DOE, adapted to project at least one periodic dot pattern.

[0244] For example, the projected illumination pattern can be a periodic dot pattern. The projected illumination pattern can have a low dot density. For example, the illumination pattern may include at least one periodic dot pattern with a low dot density, wherein the illumination pattern has ≤2500 dots per field of view. Compared to structured light, which typically has a dot density of 10k-30k in a 55x38° field of view, the illumination pattern according to the invention can be sparser. This allows for more power per dot, making the proposed technique less dependent on ambient light compared to structured light.

[0245] The display device 110 includes at least one optical sensor 118 having at least one photosensitive region 120. The optical sensor 118 is configured to determine, for example... Figures 2A to 2C and Figures 3A to 3C The at least one first image shown includes a plurality of reflection features generated by the scene in response to illumination by illumination features. The display device 110 may include a single camera, which includes an optical sensor 118. The display device 110 may include a plurality of cameras, each camera including an optical sensor 118 or a plurality of optical sensors 118.

[0246] The optical sensor 118 may specifically be or may include at least one photodetector, preferably an inorganic photodetector, more preferably an inorganic semiconductor photodetector, and most preferably a silicon photodetector. Specifically, the optical sensor 118 may be sensitive in the infrared spectral range. All pixels in the matrix or at least one group of optical sensors in the matrix may specifically be identical. Specifically, groups of identical pixels in the matrix may be provided for different spectral ranges, or all pixels may have the same spectral sensitivity. Furthermore, the pixels may have the same size and / or be identical in their electronic or optoelectronic properties. Specifically, the optical sensor 118 may be or may include at least one inorganic photodiode that is sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers. Specifically, the optical sensor 118 may be sensitive in a portion of the near-infrared region, in which the silicon photodiode is specifically suited for the range of 700 nm to 1100 nm. Infrared optical sensors that can be used for the optical sensor are commercially available infrared optical sensors, such as the TrinamiX from Ludwigshafenam Rhein (Germany) D-67056. TM The trademark name launched by GmbH is Hertzstueck TMCommercially available infrared optical sensors are available. Therefore, as an example, optical sensor 118 may include at least one optical sensor of an inherent photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of: Ge photodiodes, InGaAs photodiodes, extended InGaAs photodiodes, InAs photodiodes, InSb photodiodes, and HgCdTe photodiodes. Additionally or alternatively, optical sensor 118 may include at least one optical sensor of an inherent photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of: Ge:Au photodiodes, Ge:Hg photodiodes, Ge:Cu photodiodes, Ge:Zn photodiodes, Si:Ga photodiodes, and Si:As photodiodes. Additionally or alternatively, optical sensor may include at least one photoconductivity sensor, such as a PbS or PbSe sensor, or a radiative thermal meter, preferably selected from V0 radiative thermal meters and amorphous Si radiative thermal meters.

[0247] Optical sensor 118 can be sensitive in one or more of the ultraviolet, visible, or infrared spectral ranges. Specifically, the optical sensor can be sensitive in the visible spectral range from 500 nm to 780 nm, most preferably in the range of 650 nm to 750 nm or 690 nm to 700 nm. Specifically, optical sensor 118 can be sensitive in the near-infrared region. Specifically, optical sensor 118 can be sensitive in a portion of the near-infrared region, in which the silicon photodiode is specifically suited for the range of 700 nm to 1000 nm. Optical sensor 118 can be sensitive in the infrared spectral range, specifically in the range of 780 nm to 3.0 micrometers. For example, each optical sensor individually can be or can include at least one element selected from the group consisting of: photodiode, photovoltaic cell, photoconductor, phototransistor, or any combination thereof. For example, optical sensor 118 can be or can include at least one element selected from the group consisting of: CCD sensor element, CMOS sensor element, photodiode, photovoltaic cell, photoconductor, phototransistor, or any combination thereof. Any other type of photosensitive element can be used. Photosensitive elements can typically be made entirely or partially of inorganic materials and / or entirely or partially of organic materials. Most commonly, one or more photodiodes, such as commercially available photodiodes, for example, inorganic semiconductor photodiodes, can be used.

[0248] Optical sensor 118 may include at least one sensor element comprising a pixel matrix. Therefore, by way of example, optical sensor 118 may be part of or constitute a pixelated optics device. For example, optical sensor 118 may be and / or may include at least one CCD and / or CMOS device. By way of example, optical sensor 118 may be part of or constitute at least one CCD and / or CMOS device having a pixel matrix, with each pixel forming a photosensitive area. Sensor elements may be formed as a single device or a combination of multiple devices. The matrix may specifically be or may include a rectangular matrix having one or more rows and one or more columns. Rows and columns may specifically be arranged in a rectangular manner. However, other arrangements are possible, such as non-rectangular arrangements. By way of example, a circular arrangement is also possible, wherein the elements are arranged in concentric circles or ellipses about a center point. For example, the matrix may be a single row of pixels. Other arrangements are possible.

[0249] The pixels of the matrix can specifically be equal in one or more aspects of size, sensitivity, and other optical, electrical, and mechanical properties. The photosensitive areas 120 of all the optical sensors 118 in the matrix can specifically be located in a common plane, preferably facing the object 112, such that a light beam propagating from the object to the display device 110 can generate a light spot on the common plane. The photosensitive areas 120 can specifically be located on the surface of the respective optical sensor 118. However, other embodiments are also possible. The optical sensor 118 may include, for example, at least one CCD and / or CMOS device. As an example, the optical sensor 118 may be part of or constitute a pixelated optics. As an example, the optical sensor 118 may be part of or constitute at least one CCD and / or CMOS device having a pixel matrix, with each pixel forming a photosensitive area 120.

[0250] Display device 110 includes at least one translucent display 112 configured to display information. An illumination source 114 and an optical sensor 118 are positioned in front of the display 112 in the direction of propagation of the illumination pattern. The illumination source 114 and the optical sensor 118 may be arranged in fixed positions relative to each other. For example, the setup of display device 110 may include a camera, which includes an optical sensor 118 and a lens system; and a laser projector as the illumination source 114. The laser projector and camera may be fixed behind the translucent display in the direction of propagation of light reflected from the scene. The laser projector may generate a dot pattern and emit light through the display 112. The camera can view through the display. However, the arrangement of the illumination source 114 and the optical sensor 118 behind the translucent display in the direction of propagation of light reflected from the scene may cause the diffraction grating of the display 112 to generate multiple laser dots on the scene and also in the first image. Therefore, these multiple dots on the first image may not include any useful distance information. Display device 110 includes at least one evaluation device 124. The evaluation device 124 can be configured to discover and evaluate the zero-order reflection features of the diffraction grating, i.e. the true features, and can ignore higher-order reflection features, i.e. the false features.

[0251] Evaluation device 124 is configured to evaluate a first image. Evaluation device 124 may include at least one data processing device, and more preferably, by using at least one processor and / or at least one application-specific integrated circuit (ASIC). Thus, by example, at least one evaluation device 124 may include at least one data processing device on which software code comprising a large number of computer commands is stored. Evaluation device 124 may provide one or more hardware elements for performing one or more specified operations, and / or may provide software to one or more processors to run thereon to perform one or more specified operations, including evaluating the image. Specifically, instructions for determining the bundle profile and surface may be performed by at least one evaluation device. Thus, by example, one or more instructions may be implemented in software and / or hardware. Thus, by example, evaluation device 124 may include one or more programmable devices configured to perform the above evaluation, such as one or more computers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), or field-programmable gate arrays (FPGAs). However, additionally or alternatively, the evaluation device may also be embodied entirely or partially in hardware.

[0252] The evaluation of the first image includes identifying the reflection features of the first image. The evaluation device 124 can be configured to perform at least one image analysis and / or image processing to identify the reflection features. The image analysis and / or image processing may use at least one feature detection algorithm. The image analysis and / or image processing may include one or more of the following: filtering; selection of at least one region of interest; forming a difference image between an image created by a sensor signal and at least one offset; inverting a sensor signal by inverting an image created by a sensor signal; forming a difference image between images created by a sensor signal at different times; background correction; decomposition into color channels; decomposition into hue, saturation, and luminance channels; frequency decomposition; singular value decomposition; application of a droplet detector; application of a corner detector; application of a Hessian filter determinant; application of a curvature-based region detector; application of a maximum stable extremum region detector; application of a generalized Hough transform; application of a ridge detector; application of an affine invariant feature detector; application of an affine adaptive interest point operator; application of a Harris affine region detector; application of a Hessian affine transform. Region of interest detector; application of scale-invariant feature transform; application of scale-space extremum detector; application of local feature detector; application of accelerated robust feature algorithm; application of gradient localization and orientation histogram algorithm; application of histogram with orientation gradient descriptor; application of Deriche edge detector; application of differential edge detector; application of spatiotemporal interest point detector; application of Moravec corner detector; application of Canny edge detector; application of Laplacian Gaussian filter; application of difference Gaussian filter; application of Sobel operator; application of Laplacian operator; application of Scharr operator; application of Prewitt operator; application of Roberts operator; application of Kirsch operator; application of high-pass filter; application of low-pass filter; application of Fourier transform; application of Radon transform; application of Hough transform; application of wavelet transform; thresholding; creation of binary image. Regions of interest can be determined manually by the user or automatically, such as by identifying features within an image generated by an optical sensor.

[0253] For example, illumination source 114 can be configured to generate and / or project a point cloud, such that multiple illumination regions are generated on optical sensor 118 (e.g., a CMOS detector). Furthermore, interference may exist on optical sensor 118, such as interference attributed to speckles and / or external light and / or multiple reflections. Evaluation device 124 can be adapted to determine at least one region of interest, for example, one or more pixels illuminated by the light beam for determining the ordinate of an object. For example, evaluation device 124 can be adapted to perform filtering methods, such as speckle analysis and / or edge filtering and / or object recognition methods.

[0254] The evaluation device 124 can be configured to perform at least one image correction. Image correction may include at least one background subtraction. The evaluation device 124 may be adapted to remove the effects of background light from the beam profile, for example, by imaging without further illumination.

[0255] Each of the reflection features includes at least one beam profile. The beam profile may be selected from a linear combination of a trapezoidal beam profile, a triangular beam profile, a conical beam profile, and a Gaussian beam profile. The evaluation device is configured to determine beam profile information for each of the reflection features by analyzing their beam profiles.

[0256] Evaluation device 124 is configured to determine at least one longitudinal coordinate z for each of the reflection features by analyzing their beam profiles. DPR For example, beam profile analysis may include at least one of histogram analysis steps, calculation of difference measures, application of neural networks, and application of machine learning algorithms. Evaluation device 124 may be configured to symmetrize and / or normalize and / or filter the beam profile, particularly removing noise or asymmetry from recordings at larger angles, recording edges, etc. Evaluation device 124 may filter the beam profile by removing high spatial frequencies, such as through spatial frequency analysis and / or median filtering. Summarization may be performed by averaging the intensity center of the spot and all intensities at the same distance from the center. Evaluation device 124 may be configured to normalize the beam profile to maximum intensity, particularly taking into account intensity differences attributable to recording distances. Evaluation device 124 may be configured to remove the effects of background light from the beam profile, for example, through imaging without illumination.

[0257] Evaluation device 124 can be configured to determine the longitudinal coordinate z for each of the reflection features using photon ratio ranging technology. DPR Regarding the photon ratio ranging (DPR) technique, see WO 2018 / 091649 A1, WO2018 / 091638 A1 and WO 2018 / 091640 A1, the entire contents of which are incorporated herein by reference.

[0258] Evaluation device 124 can be configured to determine a beam profile for each of the reflection features. Determining the beam profile may include identifying at least one reflection feature provided by optical sensor 118 and / or selecting at least one reflection feature provided by optical sensor 118 and evaluating at least one intensity distribution of the reflection feature. As an example, a region of an image may be used and evaluated to determine the intensity distribution, such as a three-dimensional intensity distribution or a two-dimensional intensity distribution, such as along an axis or line through the image. As an example, the illumination center of the beam may be determined, such as by determining at least one pixel with the highest illumination, and a cross-sectional axis through the illumination center may be selected. The intensity distribution may be an intensity distribution as a function of coordinates along the cross-sectional axis through the illumination center. Other evaluation algorithms are feasible.

[0259] Analysis of the beam profile, one of the reflection characteristics, may include determining at least one first region and at least one second region of the beam profile. The first region of the beam profile may be region A1, and the second region of the beam profile may be region A2. Evaluation device 124 may be configured to integrate the first and second regions. Evaluation device 123 may be configured to derive a combined signal, particularly a quotient Q, by one or more of the following: division of the integrated first region and the integrated second region; division of multiples of the integrated first region and the integrated second region; and division of a linear combination of the integrated first region and the integrated second region. Evaluation device 124 may be configured to determine at least two regions of the beam profile and / or segment the beam profile into at least two segments comprising different regions of the beam profile, wherein overlap of regions may be possible, provided that the regions are not congruent. For example, evaluation device 124 may be configured to determine multiple regions, such as two, three, four, five, or up to ten regions. Evaluation device 124 may be configured to segment the beam spot into at least two regions of the beam profile and / or segment the beam profile into at least two segments comprising different regions of the beam profile. Evaluation device 124 can be configured to determine the integral of the beam profile over corresponding regions for at least two regions. Evaluation device 124 can be configured to compare at least two of the determined integrals. Specifically, evaluation device 124 can be configured to determine at least one first region and at least one second region of the beam profile. The first region and the second region of the beam profile can be one or both of adjacent or overlapping regions. The first region and the second region of the beam profile may not be congruent in area. For example, evaluation device 124 can be configured to divide the sensor region of a CMOS sensor into at least two sub-regions, wherein the evaluation device can be configured to divide the sensor region of the CMOS sensor into at least one left portion and / or at least one right portion and / or at least one upper portion and at least one lower portion and / or at least one inner portion and at least one outer portion.

[0260] Additionally or alternatively, the display device 110 may include at least two optical sensors 118, wherein the photosensitive areas of the first and second optical sensors may be arranged such that the first optical sensor is adapted to determine a first area of ​​the beam profile of the reflection feature, and the second optical sensor is adapted to determine a second area of ​​the beam profile of the reflection feature. The evaluation device 124 may be adapted to integrate the first and second areas.

[0261] In one embodiment, A1 may correspond to the entire or complete area of ​​the feature point on the optical sensor. A2 may be the central area of ​​the feature point on the optical sensor. The central area may be a constant value. Compared to the entire area of ​​the feature point, the central area may be smaller. For example, in the case of a circular feature point, the central area may have a radius from 0.1 to 0.9 of the full radius of the feature point, preferably from 0.4 to 0.6 of the full radius.

[0262] The evaluation device 124 can be configured to derive the quotient Q by one or more of the following: performing division on the first and second regions, performing division on multiples of the first and second regions, and performing division on linear combinations of the first and second regions. The evaluation device 124 can also be configured to derive the quotient Q using the following formula:

[0263]

[0264] Where x and y are the lateral coordinates, A1 and A2 are the first and second regions of the bundle profile, respectively, and E(x,y) represents the bundle profile.

[0265] The evaluation device 124 can be configured to determine the vertical coordinate using at least one predetermined relationship between the quotient Q and the vertical coordinate. This predetermined relationship can be one or more of empirical, semi-empirical, and analytically derived relationships. The evaluation device may include at least one data storage device for storing the predetermined relationship, such as a lookup list or lookup table.

[0266] The evaluation device 124 can be configured to perform at least one photon ratio ranging algorithm that calculates the distance with all reflection characteristics of zero order and higher.

[0267] The evaluation of the first image includes classifying the identified reflective features based on brightness. Classification may include assigning a sequence of reflective features based on brightness for further evaluation, specifically starting with the reflective feature having the highest brightness and subsequently decreasing in brightness. If the brightest reflective feature is preferred for DPR calculation, the vertical coordinate z can be determined. DPR The robustness of the diffraction grating is mainly due to the fact that the zero-order reflection feature is always brighter than the spurious feature with a higher order.

[0268] The evaluation device 124 is configured to use the longitudinal coordinate z DPR The reflection features are explicitly matched with their corresponding illumination features. The longitudinal coordinates determined using photon ratio ranging technology can be used to solve the so-called correspondence problem. In that way, the distance information for each reflection feature can be used to find the correspondence for a known laser projector grid.

[0269] Irradiation features corresponding to reflection characteristics can be determined using epipolar geometry. For a description of epipolar geometry, see, for example, X. Jiang, H. Bunke: "Dreidimensionales Computersehen", Springer, Berlin Heidelberg. Chapter 2 of 1997. Epipolar geometry can assume an illuminated image, i.e., an image of a non-distorted illuminated pattern, and a first image that can be an image determined at different spatial locations and / or spatial orientations with fixed distances. The distance can be a relative distance, also represented as a baseline. The illuminated image can also be represented as a reference image. Evaluation device 124 can be configured to determine epipolar lines in the reference image. The relative positions of the reference image and the first image can be known. For example, the relative positions of the reference image and the first image can be stored in at least one storage unit of the evaluation device. Evaluation device 124 can be adapted to determine a straight line extending from a selected reflective feature of the first image to a real-world feature of its origin. Thus, the straight line can include possible object features corresponding to the selected reflective feature. The straight line and the baseline cross the epipolar plane. Since the reference image is determined at a different relative constellation than the first image, the corresponding possible object features can be imaged on the straight line (referred to as an epipolar line) in the reference image. The epipolar line can be the intersection of the epipolar plane and the reference image. Therefore, the features of the reference image corresponding to the selected features of the first image lie on the epipolar line.

[0270] Depending on the distance to the object in the scene that has reflected the illumination feature, the reflection feature corresponding to the illumination feature can be displaced within the first image 122. A reference image may include at least one displaced region where the illumination feature corresponding to the selected reflection feature will be imaged. The displaced region may include only one illumination feature. The displaced region may also include more than one illumination feature. The displaced region may include a epipolar line or a portion of an epipolar line. The displaced region may include one or more epipolar lines or multiple portions of one or more epipolar lines. The displaced region may extend along an epipolar line, or orthogonally to an epipolar line, or both. The evaluation device 124 may be adapted to determine the illumination feature along an epipolar line. The evaluation device 124 may be adapted to determine the longitudinal coordinate z for the reflection feature and the error interval ±ε of the distance to the combined signal Q to determine the displaced region along an epipolar line corresponding to z±ε or orthogonally to an epipolar line. Measurement uncertainty using the distance measurement of the combined signal Q may result in non-circular displaced regions in the second image because the measurement uncertainty may be different for different directions. Specifically, the measurement uncertainty along one or more epipolar lines may be greater than the measurement uncertainty with respect to the orthogonal directions of one or more epipolar lines. The displacement region may include a range in orthogonal directions about one or more epipolar lines. The evaluation device 124 may be adapted to match a selected reflection feature with at least one illumination feature within the displacement region. The evaluation device 124 may be adapted to use a determined longitudinal coordinate z. DPR At least one evaluation algorithm matches selected features of the first image with illumination features within the displacement region. The evaluation algorithm may be a linear scaling algorithm. The evaluation device 124 may be adapted to determine the epipolar line closest to the displacement region and / or within the displacement region. The evaluation device may be adapted to determine the epipolar line closest to the image location of the reflection feature. The extent of the displacement region along the epipolar line may be greater than the extent of the displacement region orthogonal to the epipolar line. The evaluation device 124 may be adapted to determine the epipolar line before determining the corresponding illumination feature. The evaluation device 124 may determine the displacement region around the image location of each reflection feature. The evaluation device 124 may be adapted to assign an epipolar line to each displacement region of each image location of the reflection feature, such as by assigning an epipolar line closest to the displacement region and / or closest to the displacement region within the displacement region and / or closest to the displacement region along the direction orthogonal to the epipolar line. The evaluation device 124 may be adapted to determine the illumination feature corresponding to the reflection feature by determining the illumination feature closest to the assigned displacement region and / or closest to the assigned displacement region within the assigned displacement region and / or closest to the assigned displacement region along the assigned epipolar line and / or within the assigned displacement region along the assigned epipolar line.

[0271] Alternatively or additionally, the evaluation device 124 may be configured to perform the following steps:

[0272] - Determine the displacement region for the image location of each reflection feature;

[0273] - Assign epipolar lines to the displacement regions of each reflection feature, such as by assigning epipolar lines that are closest to the displacement region and / or closest to the displacement region within the displacement region and / or along the direction orthogonal to the epipolar lines;

[0274] - Assign and / or determine at least one irradiation feature for each reflection feature, such as by assigning an irradiation feature that is closest to the assigned displacement region and / or within the assigned displacement region and / or closest to the assigned displacement region along the assigned nucleus line and / or within the assigned displacement region along the assigned nucleus line.

[0275] Additionally or alternatively, the evaluation device 124 may be configured to make a decision among more than one epipolar line and / or illumination feature to assign to a reflection feature, such as by comparing the distances of the epipolar line and / or reflection features within the illumination image and / or by comparing the error-weighted distances of the epipolar line and / or illumination features within the illumination image, such as ε-weighted distances, and assigning the epipolar line and / or illumination feature within the shorter distance and / or ε-weighted distance to the illumination feature and / or reflection feature.

[0276] As outlined above, due to the diffraction grating, multiple reflection features are generated; for example, for each illumination feature, there is one true feature and multiple spurious features. Matching is performed by decreasing the brightness of the reflection features, starting with the brightest one. No other reflection feature can be assigned to the same matched illumination feature. Due to display artifacts, the resulting spurious features are typically darker than the true features. By classifying the reflection features by brightness, brighter reflection features are preferred for correspondence matching. If a correspondence for an illumination feature has already been used, a spurious feature cannot be assigned to an already used (i.e., matched) illumination feature.

[0277] Figure 2A A first image 122 of a simulation of an illumination pattern including a single spot of light without a display 112 is shown. Figure 2B A first image 122 captured by an optical sensor 118 behind the display 112 is shown. Multiple light spots are observed to be produced by the diffraction grating. Figure 2B In the diagram, the real feature is shown as reference 126, and the exemplary spurious feature is shown as reference 128. Figure 2C A further example of a first image 122 captured by an optical sensor 118 behind a display 112 is shown, where, in this case, the illumination pattern is a projected laser grid. Multiple light spots appear due to the diffraction grating.

[0278] Figure 3A A further exemplary first image 122 of a scene with a projected laser spot is shown. The reflection characteristics of the zero-order diffraction grating 130 and the higher-order grating 132 are shown. Figure 3B and Figure 3C The matching of reflection and illumination features is shown. In Figure 3B and 3C The left portion shows the first image 122, and the right portion shows the corresponding illumination pattern, including two illumination features. The first image 122 may include six reflection features. The evaluation device 124 can be configured to identify the reflection features in the first image 122 and classify them with respect to their brightness. Figure 3B As shown, compared to other reflection features, the two reflection features may be brighter. Evaluation device 124 can begin beam profilometry and match the illumination feature with one of the two brighter reflection features (denoted by circles 134). Each of the two brighter reflection features can be matched with one illumination feature, indicated by an arrow. Evaluation device 124 can classify the matched feature as a true feature. Figure 3C In the depicted illumination pattern, two illumination features have been matched with a brighter reflective feature. No other reflective features can be assigned to the same matching illumination features. By classifying reflective features by brightness, the brighter reflective feature is preferred for correspondence matching. If a correspondence for illumination features has already been used, a false feature cannot be assigned to the used (i.e., matched) illumination feature. Therefore, the two remaining reflective features, represented by circle 136, have no corresponding illumination features and cannot be assigned to any point in the pattern. These remaining reflective features are classified as false features by the evaluation device 124.

[0279] The evaluation device 124 is configured to reject false features and to use the longitudinal coordinate z DPR This generates depth maps for realistic features. The display device 110 can be used to generate 3D maps from scenes, such as faces.

[0280] The depth map can be further refined using further depth measurement techniques, such as triangulation and / or defocus depth and / or structured light. The evaluation device can be configured to determine at least one second longitudinal coordinate z for each of the reflection features using triangulation and / or defocus depth and / or structured light techniques. triang The evaluation device 124 can be configured to determine the second longitudinal coordinate z. triang and vertical coordinate z DPR The combined vertical coordinates. The combined vertical coordinates can be a second vertical coordinate z. triang and vertical coordinate z DPR The average value. Combining the longitudinal coordinates can be used to determine the depth map.

[0281] like Figure 1BAs shown, the display device 110 may include a further illumination source 138. The further illumination source 138 may include at least one light-emitting diode (LED). The further illumination source 138 may be configured to generate light in the visible spectrum. The optical sensor 118 may be configured to determine at least one second image comprising at least one two-dimensional image of the scene. The further illumination source 138 may be configured to provide additional illumination for the imaging of the second image. For example, the setup of the display device 110 may be extended by adding floodlight LEDs. The further illumination source 138 may illuminate the scene, such as a face, using LEDs and, in particular, without an illumination pattern, and the optical sensor 118 may be configured to capture the two-dimensional image. The 2D image may be used for face detection and verification algorithms.

[0282] If the impulse response of display 112 is known, the distorted image captured by optical sensor 118 can be repaired. Evaluation device 124 can be configured to determine at least one corrected image I0 by deconvolving a second image I with a raster function g, where I = I0*g. The raster function is also expressed as an impulse response. The undistorted image can be recovered using deconvolution methods, such as Van-Cittert or Wiener deconvolution.

[0283] like Figure 4 As shown, display device 110 can be configured to determine a raster function g. Display device 110 can be configured to illuminate a black scene with an illumination pattern comprising small, individual bright spots (denoted by reference numeral 140). The captured image 142 can be the raster function. This process can be performed only once, such as during calibration. To determine even a corrected image for imaging via display 112, display device 110 can be configured to capture an image and use a deconvolution method on the captured impulse response g. The resulting image can be a reconstructed image with fewer display artifacts and can be used for various applications, such as face recognition. Figures 5A to 5C An example of a two-dimensional image captured using optical sensor 118 is shown. Figure 5A In this example, the scene is captured using an optical sensor 118 located behind the display 112. Figure 5B In this example, the scene is captured using an optical sensor 118 without a display 112. Figure 5C The image reconstructed using the deconvolution method is shown.

[0284] Reference tag list

[0285] 110 display device

[0286] 112 monitors

[0287] 114 Irradiation Source

[0288] 116 Casing

[0289] 118 Optical Sensors

[0290] 120 photosensitive area

[0291] 122 First Image

[0292] 124 Evaluation Device

[0293] 126 True Characteristics

[0294] 128 False Features

[0295] 130 zero-order diffraction grating

[0296] 132 Advanced

[0297] 134 Circle

[0298] 136 Circle

[0299] 138 Further sources of irradiation

[0300] 140 is used to illuminate black scenes.

[0301] 142. Capture image.

[0302] References

[0303] DE 20 2018 003 644 U1

[0304] US 9,870,024 B2

[0305] US 10,057,541 B2

[0306] US 10,215,988 B2

[0307] WO 2018 / 091649 A1

[0308] WO 2018 / 091638 A1

[0309] WO 2018 / 091640 A1

[0310] WO 2019 / 042956 A1.

Claims

1. A display device (110), comprising: - At least one illumination source (114) configured to project at least one illumination pattern comprising multiple illumination features onto at least one scene; - At least one optical sensor (118) having at least one photosensitive area (120), wherein the optical sensor (118) is configured to determine at least one first image (122), the first image (122) including a plurality of reflection features generated by the scene in response to illumination by the illumination features; - At least one semi-transparent display (112) configured to display information, wherein the illumination source (114) and the optical sensor (118) are placed in the direction of propagation of the illumination pattern in front of the display (112); - At least one evaluation device (124), wherein the evaluation device (124) is configured to evaluate the first image (122), wherein the evaluation of the first image (122) includes identifying the reflection features of the first image (122) and classifying the identified reflection features with respect to brightness, wherein each of the reflection features includes at least one beam profile, wherein the evaluation device (124) is configured to determine at least one longitudinal coordinate z for each of the reflection features by analyzing their beam profiles. DPR Wherein, the vertical coordinate z DPR Let be the coordinates in the coordinate system formed by the display device, which are parallel or antiparallel to the optical axis of the display device. The evaluation device (124) is configured to use the longitudinal coordinate z DPR The reflection features are explicitly matched with the corresponding illumination features, wherein the matching is performed by decreasing the brightness of the reflection features starting from the brightest reflection feature, wherein the evaluation device (124) is configured to classify reflection features that match the illumination features as true features and reflection features that do not match the illumination features as false features, wherein the evaluation device (124) is configured to reject the false features and to use the vertical coordinate z DPR To generate a depth map for the real features.

2. The display device (110) according to claim 1, wherein, The evaluation device (124) is configured to determine at least one second longitudinal coordinate z for each of the reflection features using triangulation and / or defocus depth and / or structured light techniques. triang .

3. The display device (110) according to claim 2, wherein, The evaluation device (124) is configured to determine the second longitudinal coordinate z. triang and the vertical coordinate z DPR The combined vertical coordinates, wherein the combined vertical coordinates are the second vertical coordinate z triang and the vertical coordinate z DPR The average value of the combined longitudinal coordinates is used to generate the depth map.

4. The display device (110) according to any one of claims 1 to 3, wherein, The irradiation source (114) includes at least one laser projector, wherein the laser projector includes at least one laser source and at least one diffractive optical element (DOE).

5. The display device (110) according to any one of claims 1 to 3, wherein, The illumination source (114) is configured to generate at least one light beam having a beam path from the illumination source (114) through the display (112) to the scene, wherein the display (112) is configured to function as a grating such that the light beam undergoes diffraction by the display resulting in the illumination pattern.

6. The display device (110) according to claim 5, wherein, The wiring of the display (112) is configured to form gaps and / or slits and ridges of the grating.

7. The display device (110) according to any one of claims 1 to 3, wherein, The illumination pattern includes a periodic dot pattern.

8. The display device (110) according to any one of claims 1 to 3, wherein, The illumination pattern has a low dot density, wherein the illumination pattern has ≤ 2500 dots per field of view.

9. The display device (110) according to any one of claims 1 to 3, wherein, The evaluation device (124) is configured to determine beam profile information for each of the reflection features by using photon ratio ranging technology.

10. The display device (110) according to any one of claims 1 to 3, wherein, The optical sensor (118) includes at least one CMOS sensor.

11. The display device (110) according to any one of claims 1 to 3, wherein, The display device (110) includes a further illumination source (138), wherein the further illumination source (138) includes at least one light-emitting diode (LED).

12. The display device (110) according to claim 11, wherein, The further irradiation source (138) is configured to generate light in the visible spectrum.

13. The display device (110) according to claim 11, wherein, The optical sensor (118) is configured to determine at least one second image including at least one two-dimensional image of the scene, wherein the further illumination source (138) is configured to provide additional illumination for imaging the second image.

14. The display device (110) according to claim 13, wherein, The evaluation device (124) is configured to determine at least one corrected image by deconvolving the second image with a raster function, where I = I0*g, and where I represents the second image, I0 represents the corrected image, and g represents the raster function.

15. A method for depth measurement via a semi-transparent display (112), wherein, Using at least one display device (110) according to any one of claims 1 to 14, wherein the method comprises the following steps: a) Projecting at least one illumination pattern comprising multiple illumination features onto at least one scene using at least one illumination source (114), wherein the illumination source (114) is placed in the propagation direction of the illumination pattern in front of the display (112); b) Determine at least one first image (122) by using at least one optical sensor (118), the first image (122) including a plurality of reflection features generated by the scene in response to illumination by the illumination features, wherein the optical sensor (118) has at least one photosensitive area (120), wherein the optical sensor (118) is placed in the direction of propagation of the illumination pattern in front of the display (112), wherein each of the reflection features includes at least one beam profile; c) Evaluate the first image (122) using at least one evaluation device (124), wherein the evaluation includes the following sub-steps: C1) Identify the reflection features of the first image (122) and classify the identified reflection features with respect to brightness. C2) Determine at least one longitudinal coordinate z for each of the reflection features by analyzing their beam profiles. DPR Wherein, the vertical coordinate z DPR The coordinates are those parallel or antiparallel to the optical axis of the display device in the coordinate system formed by the display device; C3) By using the vertical coordinate z DPR The reflection features are explicitly matched with their corresponding illumination features, wherein the matching is performed by decreasing the brightness of the reflection features, starting with the brightest ones. C4) Classify reflection features that match the illumination features as true features, and classify reflection features that do not match the illumination features as false features; C5) Reject the false feature and use the vertical coordinate z DPR To generate a depth map for the real features.

16. The use of the display device (110) according to any one of claims 1 to 14 relating to a display device, for the purpose of use, is selected from the group consisting of: location measurement in traffic technology; entertainment applications; security applications; surveillance applications; safety applications; human-machine interface applications; tracking applications; photographic applications; imaging applications or camera applications; map building applications for generating maps of at least one space; homing or tracking beacon detectors for vehicles; outdoor applications; mobile applications; communication applications; machine vision applications; robotic applications; quality control applications; manufacturing applications.

Images