Depth measurement via display

JP2023552974A5Inactive Publication Date: 2024-11-20TRINAMIX GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023528715
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-08-23
Filing Date
2021-11-12
Publication Date
2024-11-20
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing 3D imaging technologies face challenges in integrating components like front cameras and sensors under transparent or translucent displays due to microcircuits and microwirings, leading to low light transmission, diffraction, and interference, which complicates depth measurement and requires additional light sources, notches, or edge areas.

Method used

A display device with an illumination source and light sensor positioned in front of a translucent display, controlled by a unit to turn off the display during measurement, allowing for depth measurement via structured light or ToF techniques without notches, using a control unit to synchronize illumination and sensing, and an evaluation device to process images for accurate depth mapping.

Benefits of technology

Enables reliable depth measurement with reduced technical effort and costs, maintaining full-size displays without notches, improving robustness to ambient light and reducing interference from display diffraction.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

A display device (1) is disclosed. The display device (1) - at least one illumination source (5) configured to project at least one illumination beam onto at least one scene; - at least one light sensor (4) having at least one light-sensitive area, said light sensor (4) being configured to measure at least one reflected light beam generated by the scene in response to illumination by the illumination beam; - at least one light-transmitting display (2) configured to display information, wherein the illumination source (5) and the light sensor (4) are arranged in front of the display (2) in the direction of propagation of an illumination light beam; - at least one control unit (8) configured to switch off the display (2) in the area of ​​the irradiation source (5) during irradiation and / or in the area of ​​the light sensor (4) during measurement; Equipped with.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] The present invention relates to a display device, a method for measuring via a translucent display and various uses of the display device. The device, method and use according to the invention can in particular be employed in various fields, for example in everyday life, security technology, gaming, traffic technology, production technology, art, photography, such as digital photography or videography for documentation or technical purposes, safety technology, information technology, agriculture, crop protection, maintenance, cosmetics, medical technology or science, although other applications are also possible. [Background technology]

[0002] Several display devices are known. Recent developments in devices with displays have shown that the display area should cover the entire available space and the frame surrounding the display should be as small as possible. As a result, electronic components and sensors (e.g., front-facing cameras, flashlights, proximity sensors, and even 3D image sensors) can no longer be located within the frame but must be placed below the display. However, most common 3D imaging technologies and systems, such as 3D imaging systems based on structured light or 3D time-of-flight (ToF), cannot be placed below the display without difficulty.

[0003] To date, it has not been known for 3D imaging systems based on structured light or 3D-ToF to operate underneath a display, i.e., without creating empty windows, free of microcircuits and / or microwiring, into which to place the components or devices of the 3D imaging system and "see" through these windows.

[0004] In the case of structured light, the main problem is the fine structure of the microcircuits and / or microwiring in transparent displays and the resulting low light transmittance through the display. This fine structure is due to the electrode matrix used to address a single pixel. Furthermore, the single pixel's metal cathode is not transparent, so the pixel itself exhibits a reciprocal lattice. In principle, the display structure as a whole, including the electrodes, can be made transparent or translucent by using certain materials. However, to date, no transparent or translucent display exists that does not have a lattice-like microstructure while maintaining high display quality and stability.

[0005] Structured light-based 3D imaging systems are based on projecting a point cloud with thousands of points and a known pattern into a scene. The microstructure of a transparent or translucent display acts like a diffraction grating structure for laser light. Because most projectors for structured light imaging systems are based on laser sources that project a well-defined dot pattern, this pattern suffers from the display's diffraction grating effect, causing all spots in the dot pattern to exhibit high diffraction orders. This has a devastating effect on structured light imaging systems, as the additional unwanted points caused by the diffraction grating structure greatly complicate the algorithm for recovering the original expected pattern.

[0006] Furthermore, the number of projection points used in conventional structured light imaging devices is quite large. Because translucent displays have very low light transmittance (e.g., even at infrared (IR) wavelengths of 850 nm and 940 nm, which are typical wavelengths for 3D imaging devices), very high output power is required for the structured light projector to have enough power through the display to be detected by the imaging device, and the structured light projector must be installed below the display, which results in additional light absorption. The combination of a large number of points and low light transmittance can result in low robustness to ambient light.

[0007] In the case of 3D-ToF sensors, due to the reflections of the display surface, which leads to multiple reflections, as well as differences in the delay of light as it passes through the display, different display structures have different refractive indices, preventing robust function when used behind a display. Furthermore, 3D-ToF sensors also require a large amount of light to illuminate the scene. Furthermore, the illumination must be uniform. The low light transmittance of the display makes it difficult to provide sufficient light, and the grating structure affects the uniformity of the illumination.

[0008] Typical 3D sensing systems have trouble measuring through a transparent display, and current devices use a notch in the display so that the sensor is not obstructed by diffractive optical effects.

[0009] DE202018003644U1 describes a portable electronic device comprising: a bottom wall and side walls that cooperate with the bottom wall to define a cavity, the side walls having edges that define an opening leading to the cavity; a protective layer that covers the opening and surrounds the cavity; and a vision subsystem that is disposed within the cavity and between the protective layer and the bottom wall and serves to provide a depth map of objects outside the protective layer, the vision subsystem comprising a clip assembly for carrying optical components that cooperate to generate depth map information, the clip assembly comprising a first bracket arranged to support and hold the optical components at a fixed distance from each other, and a second bracket having a body fixed to the first bracket (the second bracket having a protrusion extending away from the body).

[0010] US9,870,024B2 describes an electronic display including multiple layers, such as a cover layer, a color filter layer, a display layer including light-emitting diodes or organic light-emitting diodes, and a thin-film transistor layer. In one embodiment, the layers include a substantially transparent region disposed above a camera. The substantially transparent region allows external light to reach the camera, enabling the camera to record an image.

[0011] US 10,057,541 B2 describes an imaging device and an imaging method, which includes a transparent display panel and a camera facing the bottom surface of the transparent display panel, for capturing an image located in front of the transparent display panel by synchronizing a shutter time when the transparent display panel displays a black image.

[0012] US 10,215,988 B2 describes an optical system for displaying light from a scene, including an active optical component including a first plurality of light-directing apertures, a photodetector, a processor, a display, and a second plurality of light-directing apertures. The first plurality of light-directing apertures are arranged to provide optical input to the photodetector. The photodetector is arranged to receive the optical input and convert the optical input into electrical signals corresponding to intensity data and position data. The processor is connected to receive data from the photodetector and process the data for display. The second plurality of light-directing apertures are arranged to provide optical output from the display.

[0013] Mobile devices, such as smartphones and tablets, typically have a front-facing display, such as an organic light-emitting diode (OLED) area. However, the mobile device requires several sensors on its front side, such as for fingerprint recognition, one or more self-portrait cameras, and a 3D sensor. To reduce the possibility of interference with measurements using the sensors due to the presence of the front-facing display, it is known to have a special area where the sensors are located and where the display is not present or obstructed. Such a special area is called a notch. Measurements using an active light source with an activated OLED often suffer from artifacts due to interference, such as flicker and color shifts, and can still be a technical challenge. Furthermore, hiding sensor technology behind the OLED prevents direct feedback to the consumer when something is happening, such as secure authentication. [Prior art documents] [Patent documents]

[0014] [Patent Document 1] DE202018003644U1 [Patent Document 2] US9,870,024B2 [Patent Document 3] US10,057,541B2 [Patent Document 4] US10,215,988B2 Summary of the Invention [Problem to be solved by the invention]

[0015] It is therefore an object of the present invention to provide an apparatus and method that addresses the above-mentioned technical problems of known apparatus and methods, in particular to provide an apparatus and method that allows reliable depth measurement via a display with low technical effort and low demands in terms of technical resources and costs, and at the same time has a full-size display without notches or edge areas. [Means for solving the problem]

[0016] This problem is solved by the invention with the features of the independent patent claims. Advantageous developments of the invention, which can be realized individually or in combination, are set out in the dependent claims and / or in the following description and detailed embodiments.

[0017] When used below, the terms "have," "comprise," or "include," or any grammatical variations thereof, are used in a non-exclusive manner. Thus, these terms can refer both to a situation in which, besides the features introduced by these terms, no further features are present in the entity described in this context, and to a situation in which one or more further features are present. As an example, the expressions "A has B," "A comprises B," and "A includes B" can refer both to a situation in which no other elements are present in A besides B (i.e., a situation in which A is solely and exclusively composed of B), and to a situation in which, in addition to B, one or more elements are present in entity A, such as element C, elements C and D, or further elements.

[0018] Furthermore, it should be noted that the terms "at least one," "one or more," or similar expressions indicating that a feature or element may be present more than one time are typically used only once when introducing each feature or element. Note that in most cases hereinafter, when referring to each feature or element, the expressions "at least one" or "one or more" will not be repeated, despite the fact that the feature or element may appear more than one time.

[0019] Furthermore, when used hereinafter, the terms "preferably," "more preferably," "particularly," "more particularly," "particularly," "more particularly," or similar terms are used in connection with any feature without limiting its alternative possibilities. Features introduced by these terms are therefore optional features and are not intended to limit the scope of the claims in any way. The present invention can be practiced using alternative features, as will be recognized by those skilled in the art. Similarly, features introduced by "in one embodiment of the present invention" or similar phrases are intended to be optional features, without any limitations regarding alternative embodiments of the invention, without any limitations regarding the scope of the invention, and without any limitations regarding the possibility of combining features introduced in this way with other optional or non-optional features of the invention.

[0020] In a first aspect of the present invention, a display device is disclosed. As used herein, the term "display" may refer to a device of any shape configured to display information items, such as at least one image, at least one diagram, at least one histogram, at least one text, or at least one symbol. The display may be at least one monitor or at least one screen. The display may have any shape, preferably a rectangular shape. As used herein, the term "display device" may generally refer to at least one electronic device including at least one display. For example, the display device may be a mobile device selected from the group consisting of a television, a mobile phone, a smartphone, a game console, a tablet computer, a personal computer, a laptop, a tablet, a virtual reality device, or another type of portable computer.

[0021] The display device - at least one illumination source configured to project at least one illumination beam onto at least one scene; - at least one light sensor having at least one light-sensitive area, said light sensor configured to measure at least one reflected light beam generated by said scene in response to illumination by an illumination beam; - at least one light-transmitting display configured to display information, wherein the illumination source and the light sensor are arranged in front of the display in a direction of propagation of an illumination light beam; - at least one control unit configured to switch off the display in the area of ​​the illumination source during illumination and / or in the area of ​​the light sensor during measurements; Equipped with.

[0022] As used herein, the term "scene" may refer to any at least one object or region of space. A scene may include at least one object and its surrounding environment.

[0023] The illumination source is configured to project an illumination beam, in particular at least one illumination pattern including a plurality of illumination features, onto the scene. As used herein, the term "illumination source" may generally refer to at least one device adapted to provide at least one illumination beam for illuminating a scene. The illumination source may be adapted to directly or indirectly illuminate the scene, where the illumination beam is reflected or scattered by a surface of the scene and thereby directed at least in part to the light sensor. The illumination source may be adapted, for example, to illuminate the scene by directing a light beam toward the scene, thereby illuminating the scene that reflects the light beam.

[0024] The irradiation source may include at least one light source. The irradiation source may also include multiple light sources. The irradiation source may include an artificial irradiation source, in particular at least one laser source, and / or at least one incandescent lamp, and / or at least one semiconductor light source, such as at least one light-emitting diode, in particular an organic and / or inorganic light-emitting diode. By way of example, the light emitted by the irradiation source may have a wavelength of 300 to 1100 nm, in particular 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range, such as the range of 780 nm to 3.0 μm, may be used. The irradiation source may comprise at least one infrared light source. In particular, light in the near-infrared range, in particular the range of 700 nm to 1100 nm, to which silicon photodiodes are applicable, may be used. The irradiation source may be configured to generate at least one irradiation light beam, in particular at least one irradiation pattern, in the infrared range. The use of light in the near infrared range allows light that is not detected or only weakly detected by the human eye, but is detectable by silicon sensors, especially standard silicon sensors.

[0025] As used herein, the term "light ray" generally refers to a line perpendicular to the wavefront of light, indicating the direction of energy flow. As used herein, the term "beam" generally refers to a collection of light rays. Hereinafter, the terms "light ray" and "beam" are used synonymously. As further used herein, the term "light beam" generally refers to a quantity of light, specifically a quantity of light traveling in essentially the same direction, including the possibility that the light beam has an expansion or divergence angle. The light beam can have a spatial extent. Specifically, the light beam can have a non-Gaussian beam profile. The beam profile may be selected from the group consisting of a trapezoidal beam profile; a triangular beam profile; and a conical beam. The trapezoidal beam profile can have a plateau region and at least one edge region. Specifically, the light beam can be a Gaussian light beam or a linear combination of Gaussian light beams, as outlined in more detail below. However, other embodiments are possible.

[0026] The illumination source can be configured to emit light at a single wavelength. In particular, the wavelength may be in the near-infrared region. In other embodiments, the illumination may be adapted to emit light having multiple wavelengths, allowing for additional measurements in other wavelength channels.

[0027] The illumination source may comprise at least one laser projector. The laser projector may be a vertical cavity surface emitting laser (VCSEL) projector combined with refractive optics. However, other embodiments are possible. The laser projector may comprise at least one laser source and at least one diffractive optical element (DOE). The illumination source may be or include at least one multiple beam light source. For example, the illumination source may include at least one laser source and one or more diffractive optical elements (DOE). In particular, the illumination source may comprise at least one laser and / or laser source. Various types of lasers may be used, such as semiconductor lasers, double heterostructure lasers, external cavity lasers, separate confinement heterostructure lasers, quantum cascade lasers, distributed Bragg reflector lasers, polariton lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, indium arsenide lasers, transistor lasers, diode-pumped lasers, distributed feedback lasers, quantum well lasers, interband cascade lasers, gallium arsenide lasers, semiconductor ring lasers, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alternatively, non-laser light sources, such as LEDs and / or light bulbs, may be used. The illumination source may include one or more diffractive optical elements (DOEs) adapted to generate an illumination pattern. For example, the illumination source may be adapted to generate and / or project a point cloud, and may include, for example, one or more of: at least one digital light processing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one array of light-emitting diodes; or at least one array of laser light sources. Considering their generally defined beam profile and other characteristics of ease of handling, the use of at least one laser source as the illumination source is particularly preferred. The illumination source may be integrated into the housing of the display device.

[0028] Furthermore, the illumination sources may be configured to emit modulated or unmodulated light. When multiple illumination sources are used, different illumination sources may have different modulation frequencies, which may then be used to distinguish between the light beams, as outlined in more detail below.

[0029] The one or more light beams generated by the illumination source may generally propagate parallel to the optical axis or at an angle relative to the optical axis, e.g., at an angle with the optical axis. The display device may be configured such that the one or more light beams propagate from the display device toward the scene along the optical axis of the display device. To this end, the display device may include at least one reflective element, preferably at least one prism, for deflecting the illumination light beam onto the optical axis. As an example, the one or more light beams, such as laser light beams, and the optical axis may have an angle of less than 10°, preferably less than 5°, or even less than 2°. However, other embodiments are possible. Furthermore, the one or more light beams may be on the optical axis or off the optical axis. As an example, the one or more light beams may be parallel to the optical axis or even coincident with the optical axis at a distance of less than 10 mm, preferably less than 5 mm, or even less than 1 mm from the optical axis.

[0030] The illumination source is configured to generate at least one illumination pattern. The illumination pattern may include a periodic dot pattern. As used herein, the term "at least one illumination pattern" refers to at least one pattern including at least one illumination feature adapted to illuminate at least a portion of a scene. As used herein, the term "illumination feature" refers to at least one feature extending over at least a portion of a pattern. The illumination pattern may include a single illumination feature. The illumination pattern may include multiple illumination features. The illumination pattern may be selected from the group consisting of at least one dot pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; and at least one pattern including a periodic or aperiodic arrangement of features. The illumination pattern may include a regular and / or regular and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or even a convex tiling pattern. The illumination pattern may exhibit at least one illumination feature selected from the group consisting of at least one point; at least one line; at least two lines, such as parallel or intersecting lines; at least one point and one line; at least one arrangement of periodic or aperiodic features; and at least one feature of any shape. The illumination pattern may include at least one pattern selected from the group consisting of: at least one point pattern, in particular a pseudorandom point pattern; a random point pattern or a quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern including at least one known feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern; at least one pattern including a convex uniform tiling; at least one line pattern including at least one line; and at least one line pattern including at least two lines, such as parallel or intersecting lines. For example, the illumination source may be adapted to generate and / or project a point cloud.The illumination source may include at least one light projector adapted to generate a point cloud such that the illumination pattern can include a plurality of point patterns. The illumination source may include at least one mask adapted to generate the illumination pattern from the at least one light beam generated by the illumination source.

[0031] The distance between two features of the illumination pattern and / or the area of ​​at least one illumination feature may depend on the circle of confusion in the image. As outlined above, the illumination source may include at least one light source configured to generate at least one illumination pattern. Specifically, the illumination source includes at least one laser source and / or at least one laser diode assigned to generate laser radiation. The illumination source may include at least one diffractive optical element (DOE). The display device may include at least one laser source and at least one point projector, such as a DOE, adapted to project at least one periodic point pattern. As further used herein, the term "projecting at least one illumination pattern" refers to providing at least one illumination pattern for illuminating at least one scene.

[0032] For example, the projected illumination pattern may be a periodic dot pattern. The projected illumination pattern may have a low dot density. For example, the illumination pattern may include at least one periodic dot pattern with a low dot density, such that the illumination pattern has 2500 or fewer dots per field of view. Compared to structured light, which typically has a dot density of 10k to 30k in a 55 x 38° field of view, the illumination pattern according to the present invention may have a lower density. This allows the proposed technology to increase the power per dot so that it is less dependent on ambient light compared to structured light.

[0033] As used herein, the term "optical sensor" generally refers to a photosensitive device for detecting a light beam, such as for detecting illumination and / or a light spot generated by at least one light beam. As further used herein, the term "photosensitive area" generally refers to an area of ​​a light sensor that is externally illuminated by at least one light beam and generates at least one sensor signal in response to the illumination. The photosensitive area may specifically be located on the surface of each light sensor. However, other embodiments are possible. A display device may include a single camera with a light sensor. A display device may include multiple cameras, each with one or more light sensors. A display device may include multiple light sensors, each with a light sensitive area. As used herein, the term "light sensor each with at least one light sensitive area" refers to a configuration including multiple single light sensors, each with a light sensitive area, and a configuration including a combined light sensor with multiple light sensitive areas. The term "light sensor" further refers to a photosensitive device configured to generate an output signal. When the display device includes multiple light sensors, each light sensor may be implemented by providing, for example, exactly one light-sensitive area that can be illuminated, such that exactly one light-sensitive area is present within each light sensor, and generating exactly one uniform sensor signal for the entire light sensor in response to illumination of the light-sensitive area. Thus, each light sensor may be a single-area light sensor. The use of single-area light sensors, however, makes the construction of the display device particularly simple and efficient. Thus, by way of example, commercially available light sensors, such as commercially available silicon photodiodes, each having exactly one light-sensitive area, may be used in the construction. However, other embodiments are possible.

[0034] The display device may be configured to perform at least one distance measurement, such as based on a time-of-flight (ToF) technique, and / or based on a depth from defocus technique, and / or based on a depth from photon ratio technique, also called beam profile analysis. Regarding the depth from photon ratio (DPR) technique, reference is made to WO2018 / 091649A1, WO2018 / 091638A1 and WO2018 / 091640A1, the entire contents of which are incorporated by reference. The optical sensor may be or comprise at least one distance sensor.

[0035] Preferably, the photosensitive area may be oriented substantially perpendicular to the optical axis of the display device, which may be a straight optical axis or may even be refracted or split, such as by using one or more deflection elements and / or one or more beam splitters, in the latter case the substantially perpendicular orientation refers to the local optical axis of each branch or beam path of the optical arrangement.

[0036] The optical sensor may be or include at least one photodetector, preferably an inorganic photodetector, more preferably an inorganic semiconductor photodetector, and most preferably a silicon photodetector. Specifically, the optical sensor may be sensitive in the infrared spectral range. All pixels of the matrix, or at least a group of optical sensors of the matrix, may be identical. Groups of identical pixels of the matrix may be provided for different spectral ranges, or all pixels may be identical in terms of spectral sensitivity. Furthermore, the pixels may be identical in terms of size and / or their electronic or optoelectronic properties. Specifically, the optical sensor may be or include at least one inorganic photodiode sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers. Specifically, the optical sensor may be sensitive in the part of the near-infrared range, particularly in the range of 700 nm to 1100 nm, where silicon photodiodes are applicable. Infrared light sensors that can be used in the optical sensor may be commercially available infrared light sensors, such as those sold under the brand name Hertzstueck® by trinamiX GmbH, D-67056 Ludwigshafen am Rhein, Germany. Thus, by way of example, the optical sensor may include at least one optical sensor of an intrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of a Ge photodiode, an InGaAs photodiode, an extended InGaAs photodiode, an InAs photodiode, an InSb photodiode, and an HgCdTe photodiode. Additionally or alternatively, the optical sensor may include at least one optical sensor of an extrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of a Ge:Au photodiode, a Ge:Hg photodiode, a Ge:Cu photodiode, a Ge:Zn photodiode, a Si:Ga photodiode, and a Si:As photodiode.Additionally or alternatively, the optical sensor may include at least one photoconductive sensor, such as a PbS or PbSe sensor, a bolometer, preferably a bolometer selected from the group consisting of a VO bolometer and an amorphous Si bolometer.

[0037] The optical sensors may be sensitive in one or more of the ultraviolet, visible, or infrared spectral ranges. Specifically, the optical sensors may be sensitive in the visible spectral range of 500 nm to 780 nm, most preferably 650 nm to 750 nm, or 690 nm to 700 nm. Specifically, the optical sensors may be sensitive in the near-infrared range. Specifically, the optical sensors may be sensitive in the near-infrared range, particularly in the range of 700 nm to 1000 nm, where silicon photodiodes are applicable. Specifically, the optical sensors may be sensitive in the infrared spectral range, specifically in the range of 780 nm to 3.0 μm. For example, the optical sensors may each independently be or include at least one element selected from the group consisting of a photodiode, a photocell, a photoconductor, a phototransistor, or any combination thereof. For example, the optical sensor may be or include at least one element selected from the group consisting of a CCD sensor element, a CMOS sensor element, a photodiode, a photocell, a photoconductor, a phototransistor, or any combination thereof. Any other type of photosensitive element may also be used. The photosensitive element may generally be made entirely or partially of inorganic materials and / or entirely or partially of organic materials. Most commonly, one or more photodiodes, such as commercially available photodiodes, e.g., inorganic semiconductor photodiodes, may be used.

[0038] The optical sensor may include at least one sensor element including a matrix of pixels. Thus, by way of example, the optical sensor may be part of or constitute a pixelated optical device. For example, the optical sensor may be and / or include at least one CCD device and / or CMOS device. As an example, the optical sensor may be part of or constitute at least one CCD device and / or CMOS device having a matrix of pixels, each pixel forming a light-sensitive area. As used herein, the term "sensor element" generally refers to a device or a combination of devices configured to sense at least one parameter. In this case, the parameter may specifically be an optical parameter, and the sensor element may specifically be an optical sensor element. The sensor element may be formed as a single, integrated device or as a combination of several devices. The sensor element includes a matrix of optical sensors. The sensor element may include at least one CMOS sensor. The matrix may be composed of individual pixels, such as individual optical sensors. Thus, a matrix of inorganic photodiodes may be constituted. However, alternatively, commercially available matrices, e.g., one or more CCD detectors, such as CCD detector chips, and / or CMOS detectors, such as CMOS detector chips, may be used. Thus, in general, the sensor element may be and / or include at least one CCD and / or CMOS device, and / or the photosensor may form a sensor array or be part of a sensor array, such as the matrix described above. Thus, by way of example, the sensor element may comprise an array of pixels, e.g., a rectangular array having m rows and n columns, where m and n are independently positive integers. Preferably, a plurality of columns and a plurality of rows are provided, i.e., n>1 and m>1. Thus, by way of example, n may be 2 to 16 or more, and m may be 2 to 16 or more. Preferably, the ratio of the number of rows to the number of columns is close to 1. By way of example, n and m may be selected such that 0.3≦m / n≦3, such as by selecting m / n=1:1, 4:3, 16:9, or the like.As an example, the array may be a square array with equal numbers of rows and columns, such as by selecting m=2, n=2 or m=3, n=3, etc.

[0039] The matrix may be made up of individual pixels, such as individual photosensors, and thus may constitute a matrix of inorganic photodiodes. However, alternatively, one or more commercially available matrices, for example CCD detectors, such as CCD detector chips, and / or CMOS detectors, such as CMOS detector chips, may be used. Thus, in general, the photosensor may be and / or include at least one CCD and / or CMOS device, and / or the photosensor of the display device may form a sensor array or be part of a sensor array, such as the matrix described above. The matrix may be specifically a rectangular matrix having at least one row, preferably multiple rows and multiple columns. As an example, the rows and columns may be oriented in substantially perpendicular directions. As used herein, the term "substantially perpendicular" refers to a perpendicular orientation with a tolerance of, for example, ±20° or less, preferably ±10° or less, and more preferably ±5° or less. Similarly, the term "substantially parallel" refers to a parallel orientation with a tolerance of, for example, ±20° or less, preferably ±10° or less, and more preferably ±5° or less. Thus, as an example, a tolerance of less than 20°, specifically less than 10°, or even less than 5° may be tolerated. To provide a wide field of view, the matrix may specifically have at least 10 rows, preferably at least 500 rows, and more preferably at least 1,000 rows. Similarly, the matrix may have at least 10 columns, preferably at least 500 columns, and more preferably at least 1,000 columns. The matrix may include at least 50 optical sensors, preferably at least 100,000 optical sensors, and more preferably at least 5,000,000 optical sensors. The matrix may contain a number of pixels in the range of several megapixels. However, other embodiments are possible. Thus, in configurations where axial rotational symmetry is expected, a circular or concentric arrangement of the photosensors of the matrix, which may also be called pixels, may be preferred.

[0040] Thus, by way of example, the sensor element may be part of or constitute a pixelated optical device. For example, the sensor element may be and / or include at least one CCD and / or CMOS device. By way of example, the sensor element may be part of or constitute at least one CCD and / or CMOS device having a matrix of pixels, each pixel forming a light-sensitive area. The sensor element may employ a rolling shutter or a global shutter scheme for reading the matrix of optical sensors.

[0041] The display device may further include at least one transfer device. The display device may further include one or more additional elements, such as one or more additional optical elements. The display device may include at least one optical element selected from the group consisting of a transfer device, such as at least one lens and / or at least one lens system, and at least one diffractive optical element. The term "transfer device," also referred to as a "transfer system," may generally refer to one or more optical elements adapted to modify a light beam, such as by changing one or more of the beam parameters of the light beam, the width of the light beam, or the direction of the light beam. The transfer device may be adapted to direct the light beam to a light sensor. The transfer device may specifically include one or more of the following: at least one lens, for example, at least one lens selected from the group consisting of at least one adjustable-focus lens, at least one aspherical lens, at least one spherical lens, and at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflecting element, preferably at least one mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitting mirror; or at least one multi-lens system. As used herein, the term "focal length" of a transfer device refers to the distance at which incident parallel light rays that may impinge on the transfer device are focused to a "focus," also called a "focal point." Thus, the focal length constitutes an indicator of the transfer device's ability to converge an incident light beam. Therefore, the transfer device may include one or more imaging elements that may have the effect of a focusing lens. For example, the transfer device may include one or more lenses, in particular one or more refractive lenses, and / or one or more convex mirrors. In this example, focal length can be defined as the distance from the center of the thin refractive lens to the principal focus of the thin lens. For a converging thin refractive lens, such as a convex or biconvex thin lens, focal length can be considered positive and give the distance over which collimated light striking the thin lens as a translator can be focused to a single spot.Furthermore, the transfer device can include at least one wavelength-selective element, for example at least one optical filter. Furthermore, the transfer device can be designed to impart a predefined beam profile to the electromagnetic radiation, for example at the location of the sensor region, specifically at the sensor area. In principle, any of the above-described embodiments of the transfer device can be realized individually or in any desired combination.

[0042] The transfer device may have an optical axis. Specifically, the display device and the transfer device share a common optical axis. As used herein, the term "optical axis of the transfer device" generally refers to an axis of mirror symmetry or rotational symmetry of a lens or lens system. The optical axis of the display device may be a line of symmetry of the optical configuration of the display device. The display device may include at least one transfer device, preferably at least one transfer system having at least one lens. The transfer system may, for example, include at least one beam path, in which the elements of the transfer system in the beam path are arranged rotationally symmetrically with respect to the optical axis. Furthermore, as will be explained in more detail below, one or more optical elements arranged in the beam path may be off-centered or tilted with respect to the optical axis. However, in this case, the optical axis may be subsequently defined by interconnecting the centers of the optical elements in the beam path, such as by interconnecting the centers of the lenses, and in this context, the optical sensor is not considered an optical element. The optical axis may generally refer to the beam path. There, the display device may have a single beam path along which the light beam travels from the object to the light sensor, or it may have multiple beam paths. As an example, a single beam path may be provided, or the beam path may be divided into two or more partial beam paths. In the latter case, each partial beam path may have its own optical axis. The light sensors may be arranged in one and the same beam path or partial beam paths. Alternatively, however, the light sensors may also be arranged in different partial beam paths.

[0043] The transfer device may form a coordinate system where the longitudinal coordinate is the coordinate along the optical axis and d is the spatial offset from the optical axis. The coordinate system may be a polar coordinate system where the optical axis of the transfer device forms the z-axis and distance and polar angle from the z-axis can be used as additional coordinates. Directions parallel or anti-parallel to the z-axis can be considered longitudinal directions, and coordinates along the z-axis can be considered longitudinal coordinates. Any direction perpendicular to the z-axis can be considered horizontal directions, and polar coordinates and / or polar angles can be considered horizontal coordinates.

[0044] A display device may constitute a coordinate system in which the optical axis of the display device forms the z-axis, and in which additional x- and y-axes may be provided that are orthogonal to the z-axis and orthogonal to each other. As an example, the display device and / or a portion of the display device may be located at a particular point in this coordinate system, such as the origin of the coordinate system. In this coordinate system, a direction parallel or anti-parallel to the z-axis may be considered a vertical direction, and a coordinate along the z-axis may be considered a vertical coordinate. Any direction perpendicular to the vertical direction may be considered a horizontal direction, and the x- and / or y-coordinates may be considered horizontal coordinates. Alternatively, other types of coordinate systems may be used. Thus, as an example, a polar coordinate system may be used in which the optical axis forms the z-axis, and the distance from the z-axis and the polar angle may be used as additional coordinates. Similarly, a direction parallel or anti-parallel to the z-axis may be considered a vertical direction, and a coordinate along the z-axis may be considered a vertical coordinate. Any direction perpendicular to the z-axis may be considered a horizontal direction, and the polar coordinate and / or polar angle may be considered horizontal coordinates.

[0045] The display device comprises at least one translucent display configured to display information. The translucent display may be or may comprise at least one screen. The screen may have any shape, preferably a rectangular shape. As used herein, the term "translucent" may refer to the property of a display that allows light to pass through, particularly light in a specific wavelength range, preferably light in the infrared range. An illumination source and a light sensor are arranged in front of the translucent display in the direction of propagation of the illumination pattern. The information may be any information, such as at least one image, at least one diagram, at least one histogram, at least one graphic, text, numbers, at least one symbol, an operation menu, etc.

[0046] The light-transmitting display may be a full-size display having a display material that extends across the full size of the display. The term "size" of the display may refer to the surface area of ​​the light-transmitting display. The term "full-size display" may refer to one or more of the following: the light-transmitting display being recess-free or cutout-free, the light-transmitting display having an entire active display area, or the entire display area being activatable. The light-transmitting display may have a continuous distribution of display material. The light-transmitting display may be designed without recesses or cutouts. For example, the display device may include a front side having a display area, such as a rectangular display area, in which the light-transmitting display is disposed. The display area may be completely covered by the light-transmitting display, particularly the display material, particularly without recesses or notches. This may allow for a larger display size, particularly an area of ​​the display device configured to display information. For example, the entire and / or full front size of the display device may be covered by the display material, although a frame surrounding the display may be possible.

[0047] The light-transmitting display may be or may include at least one organic light-emitting diode (OLED) display. As used herein, the term "organic light-emitting diode" is a broad term and should be given its ordinary and customary meaning to those skilled in the art and should not be limited to any special or customized meaning. The term may specifically refer, without limitation, to a light-emitting diode (LED), which is a film of organic compounds in which a light-emitting electroluminescent layer is configured to emit light in response to an electric current. The OLED may be configured to emit visible light.

[0048] The display device comprises at least one control unit. The control unit is configured to turn off the display in the area of ​​the illumination source during illumination and / or in the area of ​​the light sensor during measurement. As further used herein, the term "control unit" generally refers to any device configured to control at least one further component of the display, such as the illumination source and / or the light sensor and / or the display device, in particular by using at least one processor and / or at least one application-specific integrated circuit. Thus, by way of example, the control unit may comprise at least one data processing device having stored therein software code including a number of computer commands. The control unit may provide one or more hardware elements for performing one or more specified operations and / or one or more processors having software running thereon for performing one or more specified operations. Thus, by way of example, the control unit may comprise one or more programmable devices, such as one or more computers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), or field-programmable gate arrays (FPGAs), configured to perform the above-mentioned control. Additionally or alternatively, however, the control unit may also be embodied fully or partly in hardware.

[0049] The control unit is configured to turn off the display in the area of ​​the illumination source during illumination and / or in the area of ​​the light sensor during measurement. As used herein, the term "during illumination" may refer to a time span during which the illumination source is active, e.g., switched on and / or ready to generate at least one illumination beam and / or actively illuminating a scene, etc. As used herein, the term "during measurement" may refer to a time span during which the light sensor is active, e.g., switched on and / or ready to measure and / or actively measuring, etc. As used herein, the term "turning off the display in an area" may refer to adjusting, in particular, to prevent and / or interrupt and / or stop power supply to a particular area of ​​the display. As outlined above, the display may comprise at least one OLED display. When the control unit turns off the display in the area of ​​the illumination source, the OLED display may be inactive in the area of ​​the illumination source. When the control unit turns off the display in the area of ​​the light sensor, the OLED display may be inactive in the area of ​​the light sensor. The control unit may be configured to turn off areas of the display while measurements are active.

[0050] The illumination source may comprise an emission area from which an illumination beam, in particular an illumination pattern, is emitted towards the scene. The emission area may be defined by an aperture angle of the illumination source. The term "display in the area of ​​the illumination source" may refer to the part of the translucent display covered by the emission area. The term "display in the area of ​​the light sensor" may refer to the part of the display covered by the light-sensitive area of ​​the light sensor. The illumination source and the light sensor may be arranged in a defined area. The illumination source and the light sensor may be arranged in a fixed position relative to each other. For example, the illumination source and the light sensor may be arranged adjacent to each other, in particular at a fixed distance. The illumination source and the light sensor may be arranged such that the area of ​​the translucent display covered by the emission area and the light-sensitive area is minimized.

[0051] The display may be configured to show a black area in the illumination source area and / or the light sensor area when the control unit turns off the display in the illumination source area during illumination and / or in the light sensor area during measurement. The black area may be an area that does not emit light and / or an area that has a reduced amount of light compared to other areas of the display. For example, the black area may be a darkened area. Specifically, the control unit is configured to turn off the display in the illumination source area so that the display in the illumination source area functions as an adjustable notch, and / or to turn off the display in the light sensor area so that the display in the light sensor area functions as an adjustable notch. The adjustable notch may be configured to be active during illumination and / or measurement and inactive otherwise. The adjustable notch may function as a virtual notch that is active when the display device is not in use, such as during measurement, such as during face unlock, and inactive when the light sensor, particularly the front sensor, is not needed. In the case of an OLED display used, this may mean that the display is not activated at all. This can make it possible to ensure that the color of any pixel is not changed by IR light. Furthermore, the display device, in particular the control unit and / or the further processing device and / or the further optical elements, can be configured to perceive, for example, IR laser flicker and correct the color of the display.

[0052] For example, a display device configuration may comprise a camera with a light sensor and lens system, and a projector, in particular a laser projector. The projector and camera may be fixed behind a translucent display in the direction of propagation of light reflected by the scene. The projector generates a dot pattern that can shine through the display. During the measurement, the translucent display can be turned off in the area of ​​the projector so that the translucent display displays a black area during this time. The camera can see through the display.

[0053] The adjustable notch may include a sharp edge. However, in other embodiments, the adjustable notch may be implemented with a luminance gradient to avoid any sharp fringes. The display device may comprise a luminance reduction element configured to introduce a luminance gradient at the edge of the display where the light sensor is normally located to avoid sharp fringes. This may make it possible to provide reduced luminance in the area of ​​the adjustable notch.

[0054] The control unit can be configured to synchronize the display and the illumination source in a way that they do not interfere with each other, in a so-called toggle mode.

[0055] For example, a display device may include at least one projector configured to generate at least one illumination pattern, additional flood illumination for illuminating a scene, and a light sensor. The display device may be configured such that these components are arranged in front of the display in the propagation direction of the illumination light beam. The light-transmitting display may be at least one OLED display. The OLED display may have a transmittance of about 25% or more. However, even embodiments of OLED displays with lower transmittance may be possible. The OLED display may include multiple pixels arranged in a matrix configuration. The OLED may update and / or refresh content row by row from top to bottom of the matrix. The control unit may be configured to synchronize the display, the projector, the flood illumination, and the light sensor. The display, particularly the display driver, may be configured to emit at least one signal indicating that the update and / or refresh wraps around from the last row to the first row. The illumination device and the light sensor may be arranged in the first row or one of the first rows. The display driver may be part of the control unit. The display driver may be an element of the display. Additionally or alternatively, the signal can be issued by an external control unit. For example, a vertical synchronization (VSYNC) signal (also referred to as display VSYNC) may be issued by the display when updating and / or refreshing wraps around from the last row to the first row. The display can be operated in two operating modes: "video mode" or "command mode." In video mode, the VSYNC signal can be issued from the display by the display driver. In command mode, the VSYNC signal can be generated and issued by an external control unit (e.g., a system-on-chip), as described below. The exposure of the photosensor can occur just before the display VSYNC, i.e., just before the refresh of the first line in which the illumination device and photosensor are located.The emission of light through an OLED display can be timed to occur slightly before the content of the display is updated and / or refreshed, particularly before it is overwritten, thereby minimizing visible distortion.

[0056] The light sensor, e.g., at least one IR-camera, may be synchronized with the projector and the flood illumination. The light sensor may be active during illumination, i.e., in a mode for capturing images and / or in a mode for detecting light. For example, synchronization of the light sensor and the illumination source may be achieved by the light sensor issuing a VSYNC signal, also referred to as camera VSYNC, to the control unit and a strobe signal to the illumination source, and the control unit issues a trigger signal to the illumination source in response to the camera VSYNC to activate the illumination source. When the trigger signal and the strobe signal are received by the illumination source, the illumination source starts illumination(s). However, other embodiments for synchronizing the light sensor and the illumination source are also possible.

[0057] For example, the control unit may include a system-on-chip (SoC). The SoC may include a display interface. The SoC may include at least one application programming interface (API) connected to at least one application. The SoC may further include at least one image signal processor (ISP). The light sensor may be connected to the SoC, particularly the ISP and / or the API, via at least one connection. The connection may be configured for one or more of power control, providing a clock signal (CLK), and transferring an image signal. Additionally or alternatively, the connection may be embodied as an Inter-Integrated Circuit (I2C). Additionally or alternatively, the connection may be embodied as an image data interface, such as a MIP. The application may request illumination by one or more of the illumination sources. The SoC may provide power to the light sensor via the connection via the API. The light sensor may issue a VSYNC signal to the SoC and a strobe signal to the illumination source. The SoC may issue trigger signals to the illumination sources in response to the camera VSYNC via the API to activate the illumination sources, respectively. When the respective trigger and strobe signals are received by the respective illumination sources, specifically by AND logic gates, the respective drivers of the illumination sources drive the illumination. The light sensor signals are forwarded via connections to the SoC, e.g., API and ISP, and may be provided to an application for further evaluation, e.g., along with metadata.

[0058] Furthermore, the optical sensor and the display can be synchronized. The display device can be configured to pass the display VSYNC to the optical sensor as a trigger signal to synchronize the display VSYNC with the exposure edge of the camera frame. Depending on the trigger requirement of the optical sensor, the display VSYNC can be adapted and specifically adjusted to fully meet the requirement before being passed to the optical sensor. For example, the frequency of the display VSYNC can be adjusted by half.

[0059] The control unit may be configured to emit an indication when the light sensor and / or the illumination source are active. The light-transmissive display may be configured to display the indication when the light sensor and / or the illumination source are active. For example, the display device may be configured to perform facial recognition using the illumination source and the light sensor. Methods and technologies for facial recognition are generally known to those skilled in the art. The control unit may be configured to emit an indication indicating that facial recognition is active while facial recognition is being performed. The light-transmissive display may be configured to display the indication while facial recognition is being performed. For example, the indication may be at least one warning element. The indication may be one or more of an icon and / or logo and / or symbol and / or animation indicating that the light sensor and / or the illumination source, particularly facial recognition, is active. For example, the black area may include an identifying mark indicating that secure authentication is active. This allows a user to recognize that they are in a secure environment, for example, for payment, signing, etc. For example, the warning element may change color and / or appearance to indicate that facial recognition is active. This indication also allows the user to know that the light sensors, in particular the camera, are on to avoid espionage. The control unit and / or the further security zone may be configured to issue at least one command to display at least one watermark in the black area. The watermark may be a symbol that cannot be imitated by low-security apps, for example from a store.

[0060] However, positioning the illumination source and light sensor in the direction of propagation of light reflected by the scene behind the light-transmissive display may result in the display's diffraction grating generating multiple laser points on the scene and also in the image captured by the light sensor. Thus, these multiple spots on the image may not contain useful distance information. As described in more detail below, the display device may include at least one evaluation device configured to find and evaluate the zero-order reflection features of the diffraction grating, i.e., real features, and can ignore higher-order reflection features, i.e., spurious features.

[0061] The illumination source may be configured to project at least one illumination pattern including a plurality of illumination features onto at least one scene. The optical sensor may be configured to determine at least one first image including a plurality of reflective features produced by the scene in response to illumination by the illumination features. The display device may further include at least one evaluation device. The evaluation device may be configured to evaluate the first image, where evaluation of the first image includes identifying reflective features of the first image and sorting the identified reflective features with respect to brightness. Each of the reflective features may include at least one beam profile. The evaluation device may determine at least one vertical coordinate z for each of the reflective features by analyzing the beam profile. DPR The evaluation device can be configured to determine the vertical coordinate z DPR The evaluation device may be configured to unambiguously match a reflection feature with a corresponding illumination feature using the vertical coordinate z. The matching may be performed starting with the brightest reflection feature and decreasing the brightness of the reflection feature. The evaluation device may be configured to classify reflection features that match the illumination feature as real features and classify reflection features that do not match the illumination feature as false features. The evaluation device may reject false features and classify reflection features that do not match the illumination feature as false features. DPR can be configured to generate a depth map of the real features by using

[0062] The optical sensor may be configured to determine at least one first image including a plurality of reflective features produced by the scene in response to illumination by the illumination feature. As used herein, without limitation, the term "image" may refer to data recorded by using the optical sensor, specifically, a plurality of electronic readings from an imaging device, such as pixels of a sensor element. The image itself may therefore include pixels, and the pixels of the image correlate to pixels of the matrix of sensor elements. Thus, a reference to a "pixel" may refer either to a unit of image information produced by a single pixel of a sensor element or directly to a single pixel of a sensor element. As used herein, the term "two-dimensional image" may generally refer to an image having information about lateral coordinates, such as height and width dimensions. As used herein, the term "three-dimensional image" may generally refer to an image having information about lateral coordinates and, additionally, information about vertical coordinates, such as height, width, and depth dimensions. As used herein, the term "reflective feature" may refer to a feature in the image plane produced by the scene in response to illumination, specifically, at least one illumination feature.

[0063] The evaluation device may be configured to evaluate the first image. As further used herein, the term "evaluation device" generally refers to any device adapted to perform specified operations, preferably by using at least one data processing device, more preferably by using at least one processor and / or at least one application-specific integrated circuit. Thus, by way of example, the at least one evaluation device may include at least one data processing device having software code stored thereon, including a number of computer commands. The evaluation device may provide one or more hardware elements for performing one or more of the specified operations and / or one or more processors having software executing thereon for performing one or more of the specified operations. The operations include evaluating the image. Specifically, determining the beam profile and displaying the surface may be performed by the at least one evaluation device. Thus, by way of example, the one or more instructions can be implemented in software and / or hardware. Thus, by way of example, the evaluation device may comprise one or more computers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), or one or more programmable devices, such as field-programmable gate arrays (FPGAs), configured to perform the evaluation. However, additionally or alternatively, the evaluation device may also be embodied completely or partly in hardware.

[0064] The evaluation device and the display device may be fully or partially integrated into a single device. Thus, in general, the evaluation device may also form part of the display device. Alternatively, the evaluation device and the display device may be fully or partially embodied as separate devices. The display device may include further components. The evaluation device may be connected to and / or be part of the control unit.

[0065] The evaluation device may be or include one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs), and / or one or more computers, preferably one or more microcomputers and / or one or more data processing devices, such as microcontrollers, field programmable gate arrays, or digital signal processors. Additional components may be included, for example, data acquisition devices, such as one or more preprocessing devices and / or one or more devices for receiving and / or preprocessing the sensor signals, such as one or more analog-to-digital converters and / or one or more filters. Furthermore, the evaluation device may comprise one or more measurement devices, such as one or more measurement devices for measuring current and / or voltage. Furthermore, the evaluation device may include one or more data storage devices. Furthermore, the evaluation device may include one or more interfaces, for example one or more wireless interfaces and / or one or more wired interfaces.

[0066] The evaluation device can be connected to or include at least one further data processing device that can be used for one or more of displaying, visualizing, analyzing, disseminating, communicating, or further processing information, such as information obtained by the optical sensor and / or the evaluation device. The data processing device can be connected to or incorporate at least one of a display, projector, monitor, LCD, TFT, loudspeaker, multi-channel sound system, LED pattern, or further visualization device, by way of example. It can further be connected to or incorporate at least one communication device or communication interface, connector, or port that can send encrypted or unencrypted information using one or more of email, text message, telephone, Bluetooth, Wi-Fi, infrared, or internet interfaces, ports, or connections. It may further be connected to or incorporate at least one of the following: a processor, a graphics processor, a CPU, an Open Multimedia Applications Platform (OMAP®), an integrated circuit, a product from the Apple A series or Samsung S3C2 series, a system on a chip such as a microcontroller or microprocessor, one or more memory blocks such as ROM, RAM, EEPROM or flash memory, a timing source such as an oscillator or a phase locked loop, a counter timer, a real time timer, or a power-on-reset-generator, a voltage regulator, a power management circuit or a DMA controller. The individual units may further be connected to or integrated into an Internet of Things or Industry 4.0 type network by means of a bus such as an AMBA bus.

[0067] The evaluation device and / or data processing device may be connected by or have an analog interface or port, such as one or more of a serial or parallel interface or port, a further external interface or port, such as one or more of USB, Centronics Port, FireWire, HDMI, Ethernet, Bluetooth, RFID, Wi-Fi, USART, or SPI, or one or more standardized interfaces or ports to further devices, such as a 2D camera device using an ADC or DAC, or an RGB interface such as CameraLink. The evaluation device and / or data processing device may be further connected by one or more of a processor-to-processor interface or port, an FPGA-FPGA interface, or a serial or parallel interface port. The evaluation device and data processing device may further be connected to one or more of an optical disk drive, a CD-RW drive, a DVD+RW drive, a flash drive, a memory card, a disk drive, a hard disk drive, a solid-state disk, or a solid-state hard disk.

[0068] The evaluation device and / or the data processing device may be connected by or may have one or more further external connectors, such as one or more of the following: phone connectors, RCA connectors, VGA connectors, hermaphroditic connectors, USB connectors, HDMI® connectors, 8P8C connectors, BCN connectors, IEC60320 C14 connectors, optical fiber connectors, D-subminiature connectors, RF connectors, coaxial connectors, SCART connectors, XLR connectors, and / or may incorporate at least one suitable socket for one or more of these connectors.

[0069] The evaluation device may be configured to evaluate the first image. The evaluation of the first image may include identifying reflective features in the first image. The evaluation device may be configured to perform at least one image analysis and / or image processing to identify the reflective features. The image analysis and / or image processing may use at least one feature detection algorithm. The image analysis and / or image processing may include: filtering; selecting at least one region of interest; forming a difference image between an image generated by the sensor signal and at least one offset; inverting the sensor signal by inverting the image generated by the sensor signal; forming a difference image between images generated by the sensor signal at different times; background correction; decomposition into color channels; decomposition into hue; saturation; luminance channel; frequency decomposition; singular value decomposition; applying a blob detector; applying a corner detector; applying a determinant of a Hessian filter; applying a principal curvature-based region detector; applying a maximum stable extremum region detector; applying a generalized Hough transform; applying a ridge detector; applying an affine-invariant feature detector; applying an affine-adaptive interest point operator; applying a Harris affine region detector; applying a Hessian affine region detector; applying a scale-invariant feature transform. The method may include one or more of: applying a scale-space extremum detector; applying a local feature detector; applying a fast robust feature algorithm; applying a histogram of gradient position and orientation algorithm; applying a histogram of oriented gradient descriptors; applying a Deriche edge detector; applying a differential edge detector; applying a space-time interest point detector; applying a Moravec corner detector; applying a Canny edge detector; applying a Laplace operator of a Gaussian filter; applying a differential Gaussian filter; applying a Sobel operator; applying a Laplace operator; applying a Schall operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying a Radon transform; applying a Hough transform; applying a wavelet transform; thresholding; or generating a binary image. The region of interest may be determined manually by a user or automatically, such as by recognizing features in an image generated by an optical sensor.

[0070] For example, the illumination source may be configured to generate and / or project a point cloud such that multiple illumination areas are generated on an optical sensor, e.g., a CMOS detector. Furthermore, disturbances, e.g., due to speckle and / or extraneous light and / or multiple reflections, may be present at the optical sensor. The evaluation device may be adapted to determine at least one region of interest, e.g., one or more pixels illuminated by the light beam, which are used to determine the longitudinal coordinate of the object. For example, the evaluation device may be adapted to perform filtering methods, e.g., blob analysis and / or edge filters and / or object recognition methods.

[0071] The evaluation device may be configured to perform at least one image correction, which may include at least one background subtraction, and may be adapted to remove background light from the beam profile, for example by imaging without further illumination.

[0072] Each of the reflective features includes at least one beam profile. As used herein, the term "beam profile" of a reflective feature may generally refer to at least one intensity distribution of the reflective feature, such as a light spot on an optical sensor, as a function of pixels. The beam profile may be selected from the group consisting of a trapezoidal beam profile; a triangular beam profile; a linear combination of a conical beam profile and a Gaussian beam profile. The evaluation device is configured to determine beam profile information for each of the reflective features by analyzing the beam profiles.

[0073] The evaluation device determines, by analysis of the beam profile, at least one vertical coordinate z for each of the reflection features. DPRThe evaluation device may be configured to determine the beam profile. As used herein, the term "beam profile analysis" may generally refer to the evaluation of the beam profile and may include at least one mathematical operation, and / or at least one comparison, and / or at least one symmetrization, and / or at least one filtering, and / or at least one normalization. For example, the analysis of the beam profile may include at least one of a histogram analysis step, calculation of difference measurements, application of a neural network, and application of a machine learning algorithm. The evaluation device may be configured to symmetrize and / or normalize and / or filter the beam profile to remove noise or asymmetries from the recording, especially recordings at larger angles, edges, etc. The evaluation device may filter the beam profile by removing high spatial frequencies, such as by spatial frequency analysis and / or median filtering. Aggregation may be performed by the center of the intensity of the light spot and by averaging all intensities at the same distance to the center. The evaluation device may be configured to normalize the beam profile to the maximum intensity, especially to take into account intensity differences due to the recorded distance. The evaluation device may be configured to remove contributions from background light from the beam profile, for example by imaging without illumination.

[0074] The reflection feature may cover or extend over at least one pixel of the image. For example, the reflection feature may cover or extend over multiple pixels. The evaluation device may be configured to determine and / or select all pixels that are connected to and / or belong to the reflection feature, e.g., the light spot. The evaluation device may determine the center of intensity by:

number

number

[0075] The evaluation device determines the vertical coordinate z for each of the reflective features by using a depth from photon ratio technique. DPR For Depth from Photon Ratio (DPR) techniques, see WO2018 / 091649A1, WO2018 / 091638A1 and WO2018 / 091640A1, the entire contents of which are incorporated by reference.

[0076] The evaluation device can be configured to determine a beam profile for each of the reflective features. As used herein, the term "determining a beam profile" refers to identifying at least one reflective feature provided by an optical sensor and / or selecting at least one reflective feature provided by an optical sensor and evaluating the intensity distribution of at least one of the reflective features. As an example, a region of the matrix can be used and evaluated to determine an intensity distribution, such as a three-dimensional intensity distribution or a two-dimensional intensity distribution, such as along an axis or line through the matrix. As an example, the center of illumination by the light beam can be determined, such as by determining at least one pixel with the highest illumination, and a cross-sectional axis can be selected through the center of illumination. The intensity distribution can be the intensity distribution as a function of coordinate along this cross-sectional axis through the center of illumination. Other evaluation algorithms are also possible.

[0077] The analysis of one beam profile of the reflection feature may include determining at least one first area and at least one second area of ​​the beam profile. The first area of ​​the beam profile may be area A1, and the second area of ​​the beam profile may be area A2. The evaluation device may be configured to integrate the first area and the second area. The evaluation device may be configured to derive a combined signal, in particular a quotient Q, by one or more of: dividing the integrated first area and the integrated second area; dividing a multiple of the integrated first area and the integrated second area; or dividing a linear combination of the integrated first area and the integrated second area. The evaluation device may be configured to determine at least two areas of the beam profile and / or divide the beam profile into at least two segments having different areas of the beam profile, although overlapping of the areas is possible as long as the areas do not coincide. For example, the evaluation device may be configured to determine multiple areas, such as two, three, four, five, or up to ten areas. The evaluation device may be configured to divide the light spot into at least two areas of the beam profile and / or divide the beam profile into at least two segments comprising different areas of the beam profile. The evaluation device may be configured to determine, for at least two of the areas, integrals of the beam profile over the respective areas. The evaluation device may be configured to compare at least two of the determined integrals. Specifically, the evaluation device may be configured to determine at least one first area and at least one second area of ​​the beam profile. As used herein, the term "area of ​​the beam profile" generally refers to any area of ​​the beam profile at the optical sensor location used to determine the quotient Q. The first area of ​​the beam profile and the second area of ​​the beam profile may be adjacent or overlapping, or both. The first area of ​​the beam profile and the second area of ​​the beam profile may not be coincident in area.For example, the evaluation device may be configured to divide the sensor area of ​​the CMOS sensor into at least two sub-areas, and the evaluation device may be configured to divide the sensor area of ​​the CMOS sensor into at least one left portion and at least one right portion, and / or at least one upper portion and at least one lower portion, and / or at least one inner portion and at least one outer portion. Additionally or alternatively, the display device may include at least two photosensors, and the light-sensitive areas of the first and second photosensors may be arranged such that the first photosensor is adapted to determine a first area of ​​a beam profile of the reflective feature, and the second photosensor is adapted to determine a second area of ​​the beam profile of the reflective feature. The evaluation device may be adapted to integrate the first and second areas. The evaluation device may be configured to use at least one predetermined relationship between the quotient Q and the vertical coordinate to determine the vertical coordinate. The predetermined relationship may be one or more of an empirical relationship, a semi-empirical relationship, and an analytically derived relationship. The evaluation device may comprise at least one data store for storing the predetermined relationships, such as for example a look-up list or a look-up table.

[0078] The first area of ​​the beam profile may substantially include edge information of the beam profile, the second area of ​​the beam profile may substantially include central information of the beam profile, and / or the first area of ​​the beam profile may substantially include information about the left portion of the beam profile, and the second area of ​​the beam profile may substantially include information about the right portion of the beam profile. The beam profile may have a center, i.e., the maximum value of the beam profile and / or the center point of the plateau of the beam profile and / or the geometric center of the light spot, and a trailing edge extending from the center. The second area may include an inner region of the cross-section, and the first area may include an outer region of the cross-section. As used herein, the term "substantially central information" generally refers to a low percentage of edge information, i.e., a low percentage of the intensity distribution corresponding to the edge, compared to the percentage of central information, i.e., a percentage of the intensity distribution corresponding to the center. Preferably, the central information has an edge information percentage of less than 10%, more preferably less than 5%, and most preferably, the central information does not include edge content. As used herein, the term "substantially edge information" generally refers to a low percentage of central information compared to the percentage of edge information. The edge information may include information from the entire beam profile, particularly the central and edge regions. The edge information has a percentage of central information of less than 10%, preferably less than 5%, and more preferably the edge information does not include central information. If the beam profile is close to or around the center and substantially includes central information, at least one area of ​​the beam profile may be determined and / or selected as the second area of ​​the beam profile. If the beam profile includes at least a portion of the falling edge of the cross section, at least one area of ​​the beam profile may be determined and / or selected as the first area of ​​the beam profile. For example, the entire area of ​​the cross section may be determined as the first area.

[0079] Other selections of the first area A1 and the second area A2 may also be possible. For example, the first area may include a substantially outer region of the beam profile, and the second area may include a substantially inner region of the beam profile. For example, in the case of a two-dimensional beam profile, the beam profile may be divided into a left portion and a right portion, where the first area may substantially include the area of ​​the left portion of the beam profile, and the second area may substantially include the area of ​​the right portion of the beam profile.

[0080] The edge information may include information about the number of photons in a first area of ​​the beam profile, and the center information may include information about the number of photons in a second area of ​​the beam profile. The evaluation device may be configured to determine a surface integral of the beam profile. The evaluation device may be configured to determine the edge information by integration and / or summation of the first area. The evaluation device may be configured to determine the center information by integration and / or summation of the second area. For example, the beam profile may be a trapezoidal beam profile, and the evaluation device may be configured to determine the integral of the trapezoid. Furthermore, if a trapezoidal beam profile is assumed, the determination of the edge and center signals may be replaced by equivalent evaluations utilizing characteristics of the trapezoidal beam profile, such as determining the slope and position of the edges and the height of the central plateau, and deriving the edge and center signals through geometric considerations.

[0081] In one embodiment, A1 may correspond to the entire or complete area of ​​the feature on the optical sensor. A2 may be the central area of ​​the feature on the optical sensor. The central area may be a constant value. The central area may be smaller than the total area of ​​the feature. For example, for a circular feature, the central area may have a radius between 0.1 and 0.9 of the total radius of the feature, preferably between 0.4 and 0.6 of the total radius.

[0082] In one embodiment, the illumination pattern may include at least one line pattern. A1 may correspond to an area having the full line width of the line pattern on the optical sensor, particularly on the light-sensitive area of ​​the optical sensor. The line pattern on the optical sensor may be enlarged and / or displaced compared to the line pattern of the illumination pattern, so that the line width on the optical sensor is amplified. In particular, in the case of a matrix of optical sensors, the line width of the line pattern on the optical sensor may vary from one column to another. A2 may be the central area of ​​the line pattern on the optical sensor. The line width of the central area may be a constant value, particularly corresponding to the line width of the illumination pattern. The line width of the central area may be smaller than the full line width. For example, the central area may have a line width of 0.1 to 0.9 of the full line width, preferably 0.4 to 0.6 of the full line width. The line pattern may be segmented on the optical sensor. Each column of the matrix of optical sensors may include center information of intensity in the central area of ​​the line pattern and edge information of intensity from areas extending further outward from the central area to the edge region of the line pattern.

[0083] In one embodiment, the illumination pattern may include at least one dot pattern. A1 may correspond to an area having a full radius of the dots of the dot pattern on the optical sensor. A2 may be a central area of ​​the dots of the dot pattern on the optical sensor. The central area may be a constant value. The central area may have a radius corresponding to the full radius. For example, the central area may have a radius between 0.1 and 0.9 of the full radius, preferably between 0.4 and 0.6 of the full radius.

[0084] The illumination pattern may include both at least one dot pattern and at least one line pattern. In addition to or as an alternative to line and dot patterns, other embodiments are possible.

[0085] The evaluation device may be configured to derive the quotient Q by one or more of dividing the first area and the second area, dividing the first area and a multiple of the second area, or dividing a linear combination of the first area and the second area.

number

[0086] Additionally or alternatively, the evaluation device may be adapted to determine one or both of center and edge information from at least one slice or cut of the light spot. This may be achieved, for example, by replacing the surface integral of the quotient Q with a line integral along the slice or cut. To improve accuracy, several slices or cuts through the light spot may be used and averaged. In the case of an elliptical spot profile, averaging over several slices or cuts may provide improved distance information.

[0087] For example, in the case of an optical sensor having a matrix of pixels, the evaluation device - determining the pixel with the highest sensor signal and forming at least one center signal; - evaluating the matrix of sensor signals and forming at least one sum signal; - determining a quotient Q by combining the center signal and the sum signal; - determining at least one longitudinal coordinate z of the object by evaluating the quotient Q, thereby evaluating the beam profile.

[0088] As used herein, the term "sensor signal" generally refers to a signal generated by a light sensor and / or at least one pixel of the light sensor in response to illumination. Specifically, the sensor signal may be or include at least one electrical signal, such as at least one analog electrical signal and / or at least one digital electrical signal. More specifically, the sensor signal may be or include at least one voltage signal and / or at least one current signal. Even more specifically, the sensor signal may include at least one photocurrent. Furthermore, the raw sensor signal may be used, or a display device, light sensor, or other element may be adapted to process or preprocess the sensor signal, such as by filtering, thereby generating a secondary sensor signal that may also be used as the sensor signal. The term "center signal" generally refers to at least one sensor signal that contains substantially central information of the beam profile. As used herein, the term "highest sensor signal" refers to one or both of a local maximum or maximum value in a region of interest. For example, the central signal may be the signal of a pixel having the highest sensor signal among multiple sensor signals generated by pixels of the entire matrix or a region of interest within the matrix, which region of interest may be predetermined or determinable within the image generated by the pixels of the matrix. The central signal may originate from a single pixel or from a group of optical sensors; in the latter case, for example, the sensor signals of the group of pixels may be added, integrated, or averaged to determine the central signal. The group of pixels from which the central signal originates may be, for example, a group of neighboring pixels, such as pixels that are less than a predetermined distance away from the actual pixel with the highest sensor signal, or a group of pixels that generate sensor signals within a predetermined range from the highest sensor signal. The group of pixels from which the central signal originates may be selected to be as large as possible to allow for the largest dynamic range. The evaluation device may be adapted to determine the central signal by integrating multiple sensor signals, for example, multiple pixels around the pixel with the highest sensor signal.For example, the beam profile may be a trapezoidal beam profile and the evaluation device may be adapted to determine the integral of the trapezoid, in particular the integral of the plateau of the trapezoid.

[0089] As mentioned above, the center signal may generally be a single sensor signal, such as a sensor signal from a pixel at the center of the light spot, or may be a combination of multiple sensor signals, such as a combination of sensor signals arising from pixels at the center of the light spot, or may be a secondary sensor signal derived by processing sensor signals derived from one or more of the foregoing possibilities. The determination of the center signal may be performed electronically, as comparison of sensor signals is fairly easily performed by conventional electronics, or may be performed fully or partially by software. Specifically, the center signal may be selected from the group consisting of: the highest sensor signal; the average of a group of sensor signals that are within a predetermined tolerance range of the highest sensor signal; the average of sensor signals from a group of pixels including the pixel with the highest sensor signal and a predetermined group of adjacent pixels; the sum of sensor signals from a group of pixels including the pixel with the highest sensor signal and a predetermined group of adjacent pixels; the sum of a group of sensor signals that are within a predetermined tolerance range of the highest sensor signal; the average of a group of sensor signals that exceed a predetermined threshold; the sum of a group of sensor signals that exceed a predetermined threshold; the integral of sensor signals from a group of optical sensors including the optical sensor with the highest sensor signal and a predetermined group of adjacent pixels; the integral of a group of sensor signals that are within a predetermined tolerance range of the highest sensor signal; the integral of a group of sensor signals that exceed a predetermined threshold.

[0090] Similarly, the term "sum signal" generally refers to a signal that contains substantially edge information of the beam profile. For example, the sum signal can be derived by adding, integrating, or averaging sensor signals of the entire matrix or a region of interest within the matrix, where the region of interest is predetermined or determinable within the image generated by the optical sensors of the matrix. When summing, integrating, or averaging the sensor signals, the actual optical sensors at which the sensor signals are generated may be excluded from the summation, integration, or averaging, or may be included in the summation, integration, or averaging. The evaluation device may be adapted to determine the sum signal by integrating signals of the entire matrix or a region of interest within the matrix. For example, the beam profile may be a trapezoidal beam profile, and the evaluation device may be adapted to determine the integral of the entire trapezoid. Furthermore, if a trapezoidal beam profile is assumed, the determination of the edge signal and the center signal may be replaced by equivalent evaluations utilizing the characteristics of the trapezoidal beam profile, such as determining the slope and position of the edges and the height of the central plateau, and deriving the edge signal and the center signal through geometric considerations.

[0091] Similarly, the center signal and the edge signal can also be determined by using a segment of the beam profile, such as a circular segment of the beam profile. For example, the beam profile can be divided into two segments by a dividing line or chord that does not pass through the center of the beam profile. Thus, one segment will substantially contain edge information, and the other segment will substantially contain center information. For example, the edge signal can be further subtracted from the center signal to further reduce the amount of edge information in the center signal.

[0092] The quotient Q may be a signal generated by combining the center signal and the sum signal. Specifically, the determination may include one or more of: forming a quotient of the center signal and the sum signal, or its inverse; forming a quotient of a multiple of the center signal and a multiple of the sum signal, or its inverse; forming a quotient of a linear combination of the center signals and a linear combination of the sum signal, or its inverse. Additionally or alternatively, the quotient Q may include any signal or combination of signals that includes at least one item of information regarding the comparison between the center signal and the sum signal.

[0093] As used herein, the term "longitudinal coordinate of an object" refers to the distance between the optical sensor and the object. The evaluation device may be configured to use at least one predetermined relationship between the quotient Q and the longitudinal coordinate to determine the longitudinal coordinate. The predetermined relationship may be one or more of an empirical relationship, a semi-empirical relationship, and an analytically derived relationship. The evaluation device may include at least one data storage device for storing the predetermined relationship, such as a look-up list or a look-up table.

[0094] The evaluation unit may be configured to execute at least one depth from photon ratio algorithm that calculates distances for all zeroth and higher order reflectance features.

[0095] Evaluating the first image may include sorting the identified reflective features by brightness. As used herein, the term "sorting" may refer to assigning reflective features by brightness for further evaluation, particularly assigning a sequence of reflective features with decreasing brightness, starting with the reflective feature with the greatest brightness. Sorting by decreasing brightness may refer to sorting according to and / or with respect to decreasing brightness. As used herein, the term "brightness" may refer to the size of the reflective features in the first image and / or the intensity of the reflective features in the first image. Brightness may refer to a defined passband, such as the visible or infrared spectral range, or may be wavelength independent. If the brightest reflective features are prioritized in the DPR calculation, the vertical coordinate z DPR This can increase the robustness of the determination of , mainly because the zero-order reflection feature of the diffraction grating is always brighter than the higher-order spurious features.

[0096] The evaluation device measures the vertical coordinate z DPR The depth from photon ratio technique can be used to unambiguously match reflection features with corresponding illumination features. The vertical coordinates determined by the depth from photon ratio technique can be used to solve the so-called correspondence problem. In this way, the distance information for each reflection feature can be used to find correspondences for a known laser projector grid. As used herein, the term "matching" refers to identifying and / or determining and / or evaluating corresponding illumination and reflection features. As used herein, the term "corresponding illumination and reflection features" can refer to the fact that each illumination feature of an illumination pattern generates a reflection feature in a scene, where the generated reflection feature is assigned to the illumination feature that generated it.

[0097] As used herein, the term "unequivocally matching" may refer to only one reflectance feature being assigned to one illumination feature and / or no other reflectance features being assigned to the same matching illumination feature.

[0098] Illumination features corresponding to reflection features can be determined using epipolar geometry. For a description of epipolar geometry, see, for example, Chapter 2 of "Dreidimensionales Computersehen" by X. Jiang and H. Bunke, Springer, Berlin Heidelberg, 1997. Epipolar geometry may assume that the illumination image, i.e., an image of an undistorted illumination pattern, and the first image are images determined at different spatial positions and / or spatial orientations with a fixed distance. The distance is a relative distance and is also referred to as a baseline. The illumination image is also referred to as a reference image. The evaluation device may be adapted to determine an epipolar line in the reference image. The relative positions of the reference image and the first image may be known. For example, the relative positions of the reference image and the first image may be stored in at least one storage unit of the evaluation device. The evaluation device may be adapted to determine a straight line extending from a selected reflection feature in the first image to the real-world feature from which it originates. Thus, the straight line may include a possible object feature corresponding to the selected reflection feature. The line and the baseline define an epipolar plane. Because the reference image is determined at a different relative position from the first image, possible corresponding object features can be imaged on a line called an epipolar line in the reference image. The epipolar line can be the intersection of the epipolar plane and the reference image. Thus, a feature in the reference image that corresponds to a selected feature in the first image lies on the epipolar line.

[0099] Depending on the distance to the object in the scene that reflected the illumination feature, the reflection feature corresponding to the illumination feature may be displaced in the first image. The reference image may include at least one displacement region in which an illumination feature corresponding to a selected reflection feature is imaged. The displacement region may include only one illumination feature. The displacement region may include multiple illumination features. The displacement region may include an epipolar line or a section of the epipolar line. The displacement region may include multiple epipolar lines or multiple sections of multiple epipolar lines. The displacement region can extend along the epipolar line, or orthogonal to the epipolar line, or both. The evaluation device may be adapted to determine the illumination feature along the epipolar line. The evaluation device may be adapted to determine a vertical coordinate z of the reflection feature and an error interval ±ε from the combined signal Q, and determine a displacement region along the epipolar line or orthogonal to the epipolar line corresponding to ±ε. The measurement uncertainty of the distance measurement using the combined signal Q may result in a non-circular displacement region in the second image, since the measurement uncertainty may vary depending on the direction. Specifically, the measurement uncertainty along the epipolar line may be larger than the measurement uncertainty in a direction perpendicular to the epipolar line. The displacement region may include an area extending perpendicular to the epipolar line or multiple epipolar lines. The evaluation device may be adapted to match the selected reflection feature with at least one illumination feature in the displacement region. The evaluation device may be adapted to match the determined vertical coordinate z DPRThe evaluation device may be adapted to match selected features of the first image with illumination features within the displacement region by using at least one evaluation algorithm that takes into account the following: (a) a linear scaling algorithm; (b) an evaluation algorithm; (c) a linear scaling algorithm; (d) the evaluation device may be adapted to determine an epipolar line closest to the displacement region and / or within the displacement region; (e) the evaluation device may be adapted to determine an epipolar line closest to the image position of the reflection feature; (f) the extent of the displacement region along the epipolar line may be greater than the extent of the displacement region perpendicular to the epipolar line; (g) the evaluation device may be adapted to determine the epipolar line before determining the corresponding illumination feature; (h) the evaluation device may determine a displacement region around the image position of each reflection feature; (i) the evaluation device may be adapted to assign an epipolar line to each displacement region of each image position of the reflection feature, such as by assigning an epipolar line closest to the displacement region and / or an epipolar line within the displacement region and / or an epipolar line closest to the displacement region along a direction perpendicular to the epipolar line. The evaluation device may be adapted to determine the illumination feature corresponding to the reflection feature by determining the illumination feature closest to the assigned displacement region and / or the illumination feature within the assigned displacement region and / or the illumination feature closest to the assigned displacement region along the assigned epipolar line and / or the illumination feature within the assigned displacement region along the assigned epipolar line.

[0100] Additionally or alternatively, the evaluation device may perform the following steps: - determining a displacement area of ​​the image location of each reflection feature; - assigning epipolar lines to the displacement region of each reflection feature, such as by assigning the epipolar line closest to the displacement region and / or the epipolar line within the displacement region and / or the epipolar line closest to the displacement region along a direction orthogonal to the epipolar line; - assigning and / or determining at least one illumination feature to each reflection feature, such as by assigning the illumination feature closest to the assigned displacement area and / or the illumination feature within the assigned displacement area and / or the illumination feature closest to the assigned displacement area along the assigned epipolar line and / or the illumination feature within the assigned displacement area along the assigned epipolar line; may be configured to perform

[0101] Additionally or alternatively, the evaluation device may be adapted to decide between a plurality of epipolar lines and / or illumination features to be assigned to a reflection feature, such as by comparing distances of the reflection features and / or epipolar lines in the illumination image and / or by comparing error-weighted distances, such as ε-weighted distances, of the illumination features and / or epipolar lines in the illumination image, and assigning the epipolar line and / or illumination feature with the shorter distance and / or ε-weighted distance to the illumination feature and / or reflection feature.

[0102] As explained above, the diffraction grating generates multiple reflection features, e.g., one real feature and multiple false features for each illumination feature. Matching is performed starting with the brightest reflection feature and decreasing the brightness of the reflection features. Other reflection features cannot be assigned to matched illumination features. Due to display artifacts, the generated false features are generally darker than the real features. By sorting the reflection features by brightness, brighter reflection features are prioritized in correspondence matching. If the illumination feature correspondence has already been used, the false feature cannot be assigned to the used, i.e., matched, illumination feature.

[0103] The evaluation device may be configured to classify reflection features that match the illumination features as real features and to classify reflection features that do not match the illumination features as false features. As used herein, the term "classifying" may refer to assigning a reflection feature to at least one category. As used herein, the term "real feature" may refer to a zero-order reflection feature of a diffraction grating. As used herein, the term "false feature" may refer to a higher-order reflection feature of a diffraction grating, i.e., order ≧1. The zero-order of a diffraction grating is always brighter than higher-order false features.

[0104] The evaluation device rejects false features and calculates the vertical coordinate z DPR The display device may be configured to generate a depth map for a real feature by using a 3D map of a scene, such as a facial scene. As used herein, the term "depth" may refer to the distance between an object and a light sensor, and may be given by a vertical coordinate. As used herein, the term "depth map" may refer to the spatial distribution of depth. The display device may be used to generate a 3D map from a scene, such as a facial scene.

[0105] Structured light methods typically use a camera and a projector with a fine grid of points, typically several thousand points. A well-known projector pattern is used to find the correspondences between point patches on the scene. Once the point correspondences are known, distance information can be obtained by triangulation. If the camera is located behind the display, the image will be spatially distorted due to diffraction. Therefore, finding the point pattern on the distorted image is a challenging task. Compared to structured light methods, this invention proposes to use a depth-from-photon-ratio technique to evaluate the beam profile, which is not directly affected by the display's diffraction grating. Distortion does not affect the beam profile.

[0106] The depth map may be further refined by using additional depth measurement techniques such as triangulation and / or depth from defocus and / or structured light. The evaluation device may use triangulation and / or depth from defocus and / or structured light techniques to determine at least one second vertical coordinate z for each of the reflective features. triang The method may be configured to determine:

[0107] The evaluation device may be adapted to determine the displacement of the illumination feature and the reflection feature. The evaluation device may be adapted to determine the displacement of the matched illumination feature and the selected reflection feature. The evaluation device, e.g., at least one data processing unit of the evaluation device, may be configured to determine the displacement of the illumination feature and the reflection feature, particularly by comparing the respective image positions of the illumination image and the first image. As used herein, the term "displacement" refers to the difference between the position in the illumination image and the position in the first image. The evaluation device may be adapted to determine second vertical information of the matched feature using a predetermined relationship between the second vertical coordinate and the displacement. The evaluation device may be adapted to determine the predetermined relationship by using a triangulation method. If the position of the selected reflection feature and the position of the matched illumination feature in the first image and / or the relative displacement of the selected reflection feature and the matched illumination feature are known, the vertical coordinate of the corresponding object feature can be determined by triangulation. Thus, the evaluation device may be adapted to select reflection features, for example, sequentially and / or row by row, and to determine a corresponding distance value using triangulation for each potentially possible position of the illumination feature. The displacements and corresponding distance values ​​may be stored in at least one storage device of the evaluation device. The evaluation device may, for example, include at least one data processing device, such as at least one processor, at least one DSP, at least one FPGA, and / or at least one ASIC. Furthermore, at least one data storage device may be provided to store at least one predetermined or determinable relationship between the second vertical coordinate z and the displacement, e.g., by providing one or more look-up tables for storing predetermined relationships. The evaluation device may be adapted to store parameters for intrinsic and / or extrinsic calibration of the camera and / or display device. The evaluation device may be adapted to generate parameters for intrinsic and / or extrinsic calibration of the camera and / or display device, e.g., by performing a Tsai camera calibration.The evaluation device may be adapted to calculate and / or estimate parameters such as the focal length of the transfer device, the distortion coefficient of the radiation lens, the coordinate of the center of distortion of the radiation lens, a scale factor to account for uncertainties due to imperfections in the scanning and digitization hardware timing, the rotation angle of the transformation between world coordinates and camera coordinates, the translation component of the transformation between world coordinates and camera coordinates, the aperture angle, the image sensor type, the principal point, the skew factor, the camera center, the camera orientation, the baseline, rotation or translation parameters of the camera and / or illumination source, the aperture, the focal length, etc.

[0108] The evaluation device is triang and the vertical coordinate z DPR The bond vertical coordinate may be determined by the second vertical coordinate z triang and the vertical coordinate z DPR The combined vertical coordinate can be used to generate a depth map.

[0109] The display device may comprise an additional illumination source. The additional illumination source may include at least one light-emitting diode (LED). The additional illumination source may be configured to generate light in the visible spectral range. The light sensor may be configured to determine at least one second image comprising at least one two-dimensional image of the scene. The additional illumination source may be configured to provide additional illumination for imaging the second image. For example, the display device configuration may be extended by additional flood illumination LEDs. The additional illumination source may illuminate the scene, such as a face, with an LED, in particular without an illumination pattern, and the light sensor may be configured to capture the two-dimensional image. The two-dimensional image may be used for face detection and verification algorithms. If the impulse response of the display is known, the distorted image captured by the light sensor can be restored. The evaluation device may be configured to determine at least one corrected image I0 by deconvolving the second image I with a lattice function g, where I=I0 *g). The lattice function is also called the impulse response. An undistorted image can be restored by a deconvolution approach, such as Van-Cittert or Wiener deconvolution. A display device can be configured to determine the lattice function g. For example, the display device can be configured to illuminate a black scene with an illumination pattern containing a small, single bright point. The captured image can be the lattice function. This procedure can be performed only once, such as during calibration. To determine a corrected image even when imaging through a display, the display device can be configured to capture an image and use a deconvolution approach on the captured impulse response g. The resulting image can be a reconstructed image with fewer display artifacts and can be used in several applications, such as face recognition.

[0110] The evaluation device may be configured to determine at least one material property m of the object by evaluating at least one beam profile of the reflective feature, preferably the beam profiles of a plurality of reflective features. For details of determining at least one material property m by evaluating the beam profile, reference is made to WO2020 / 187719, the contents of which are incorporated by reference.

[0111] As used herein, the term "material property" refers to at least one property of a material configured for characterization and / or identification and / or classification of the material. For example, the material property may be a property selected from the group consisting of roughness, penetration depth of light into the material, a property characterizing the material as a biological or non-biological material, reflectance, specular reflectance, diffuse reflectance, surface properties, a measure of translucency, scattering, specifically backscattering behavior, etc. The at least one material property may also be a property selected from the group consisting of scattering coefficient, translucency, transparency, deviation from Lambertian surface reflectance, speckle, etc. As used herein, the term "identifying at least one material property" refers to one or more of determining a material property and assigning a material property to an object. The evaluation device may include at least one database including a list and / or table, such as a look-up list and / or look-up table, of predefined and / or predetermined material properties. The list and / or table of material properties can be determined and / or generated by performing at least one test measurement using the display device according to the present invention, for example, by performing material testing using samples with known material properties. The lists and / or tables of material properties can be determined and / or generated at the manufacturer site and / or by a user of the display device. The material properties may be further assigned to a material classification such as, for example, one or more of: material name, material group such as biological or non-biological material, translucent or non-translucent material, metal or non-metal, skin or non-skin, fur or non-fur, carpet or non-carpet, reflective or non-reflective, specular or non-specular, foam or non-foam, hair or non-hair, roughness group, etc. The evaluation device may include at least one database including lists and / or tables containing material properties and associated material names and / or material groups.

[0112] For example, without wishing to be bound by this theory, human skin may have a reflected beam profile, also referred to as a backscattering profile, that includes a portion generated by the back reflection of the surface, referred to as the surface reflection, and a portion generated by the highly diffuse reflection from light penetrating the skin, referred to as the diffuse portion of the backscattering. For the reflectance profile of human skin, see "Lasertechnikinder Medizin: Grundlagen, Systeme, Anwendungen," "Wirkung von Laserstrahlung auf Gewebe," 1991, pp. 10171-266, Juergen Eichler, Theo Seiler, Springer Verlag, ISBN 0939-0979. The surface reflectance of skin may increase as the wavelength increases toward the near infrared. Furthermore, the penetration depth may increase as the wavelength increases from visible light to the near infrared. The diffuse portion of the backscattering may increase as the penetration depth of light increases. These characteristics can be used to distinguish skin from other materials by analyzing the backscattering profile.

[0113] Specifically, the evaluation device may be configured to compare the beam profile of the reflection feature, also referred to as the reflected beam profile, with at least one predetermined and / or pre-recorded and / or pre-defined beam profile. The predetermined and / or pre-recorded and / or pre-defined beam profile may be stored in a table or look-up table, for example, empirically determined, and may be stored in at least one data storage device of the display device, for example. For example, the predetermined and / or pre-recorded and / or pre-defined beam profile may be determined upon initial startup of a mobile device including the display device. For example, the predetermined and / or pre-recorded and / or pre-defined beam profile may be stored in at least one data storage device of the mobile device, for example, by software, specifically an app downloaded from an app store, etc. The reflection feature may be indicated as being generated by biological tissue if the reflected beam profile and the predetermined and / or pre-recorded and / or pre-defined beam profile are identical. The comparison may include overlaying the reflected beam profile and the predetermined or pre-defined beam profile so that their centers of intensity match. The comparison may include determining a deviation, e.g., a sum of squares of inter-point distances, between the reflected beam profile and a predetermined and / or pre-recorded and / or pre-defined beam profile. The evaluation device may be configured to compare the determined deviation with at least one threshold value, and if the determined deviation is lower than and / or equal to the threshold value, the surface is indicated as biological tissue and / or the detection of biological tissue is confirmed. The threshold value may be stored in a table or look-up table, may be determined, e.g., empirically, and may, for example, be stored in at least one data storage device of the display device.

[0114] Additionally or alternatively, to identify whether a reflection feature is generated by biological tissue, the evaluation device may be configured to apply at least one image filter to the image of the area. As further used herein, the term "image" refers to a two-dimensional function f(x,y), where brightness and / or color values ​​are given for any x,y position in the image. The positions may be discretized corresponding to the recording pixels. The brightness and / or color may be discretized corresponding to the bit depth of the optical sensor. As used herein, the term "image filter" refers to at least one mathematical operation applied to the beam profile and / or at least one specific region of the beam profile. Specifically, the image filter Φ maps the image f or a region of interest in the image to a real number Φ(f(x,y))=φ, where φ denotes a feature, particularly a material feature. Images may be affected by noise, and the same applies to features. Therefore, features may be random variables. Features may follow a normal distribution. If the features do not follow a normal distribution, they may be transformed to follow a normal distribution by, for example, Box-Cox transformation.

[0115] The evaluation device determines at least one material characteristic φ by applying at least one material-dependent image filter Φ2 to the image. 2m As used herein, the term "material dependent" image filter refers to an image having a material dependent output. The output of a material dependent image filter is referred to herein as a "material feature φ 2m " or "material-dependent feature φ 2m The material signature may be or may include at least one piece of information regarding at least one material property of the surface of the area that produced the reflection signature.

[0116] The material dependent image filter may be an intensity filter; a spot shape filter; a squared norm gradient; a standard deviation; a smoothness filter such as a Gaussian filter or a median filter; a gray level generation based contrast filter; a gray level generation based energy filter; a gray level generation based uniformity filter; a gray level generation based dissimilarity filter; a rho energy filter; a threshold domain filter; or a linear combination thereof; or an intensity filter, a spot shape filter, a squared norm gradient, a standard deviation, a smoothness filter, a gray level generation based energy filter, a gray level generation based uniformity filter, a gray level generation based dissimilarity filter, a rho energy filter, or a threshold domain filter, or |ρ Ф2other,Фm These linear combinations (Φ) with | ≥ 0.40 m is a further material-dependent image filter Φ that is correlated to one or more of a brightness filter, a spot shape filter, a squared norm gradient, a standard deviation, a smoothness filter, a gray level generation-based energy filter, a gray level generation-based uniformity filter, a gray level generation-based dissimilarity filter, a low energy filter, or a threshold area filter, or one of their linear combinations. 2other , and at least one filter selected from the group consisting of: a further material-dependent image filter Φ 2other is the material-dependent image filter Ф m and one or more of |ρ Ф2other,Фm |≧0.60, preferably |ρ Ф2other,Фm may be correlated by |≧0.80.

[0117] The material-dependent image filter may be at least one arbitrary filter Φ that passes hypothesis validation. As used herein, the term "passes hypothesis validation" refers to the fact that the null hypothesis H0 is rejected and an alternative hypothesis H1 is accepted. The hypothesis validation may include validating the material dependency of the image filter by applying the image filter to a predefined data set. The data set may include multiple beam profile images. As used herein, the term "beam profile image" refers to NB refers to the sum of Gaussian radial basis functions,

number

number

number

number

[0118] Next, for each image f k, , the feature value φ corresponding to the filter Φ k can be calculated, and Φ(f k (x,y),z k )=φ k , in the formula z k is an image f from a predefined dataset k This is the distance value corresponding to the generated feature value φ k For hypothesis testing, a null hypothesis may be used, in which the filter does not distinguish between material classes. The null hypothesis is H0: μ1=μ2==μ j , where μ m is the feature value φ k The index m represents the material group. Hypothesis testing can be performed using an alternative hypothesis that the filter distinguishes between at least two material classes. The alternative hypothesis is

number

number

number

number

number

number

number

number

number

[0119] In the following, we will describe the image filter assuming that the reflection image contains at least one reflection feature, in particular a spot image. The spot image f is expressed by the function

number

[0120] For example, the material dependent image filter may be a luminance filter, which may return a luminance measurement of a spot as a material characteristic.

number

number

number

[0121] For example, a material dependent image filter may be a filter whose output depends on the spot shape. The material dependent image filter may return a value as a material characteristic that correlates with the translucency of the material. The translucency of the material affects the shape of the spot. The material characteristic may be:

number

number

[0122] For example, the material dependent image filter may be a squared norm gradient, which may return a value that correlates to a measure of the soft-hard transition and / or roughness of the spot as a material characteristic.

number

number

[0123] In the formula, μ is

number

[0124] For example, the material-dependent image filter may be a smoothness filter, such as a Gaussian filter or a median filter. In one embodiment of the smoothness filter, the image filter may refer to the observation that volumetric scattering exhibits less speckle contrast compared to diffuse scattering materials. The image filter may quantify the smoothness of spots corresponding to the speckle contrast as a material feature. The material feature may be:

number

number

number

number

[0125] The material characteristics of this filter are:

number

[0126] In the formula, Var represents the variance function.

[0127] For example, the image filter may be a gray level generation based contrast filter. The material filter may be a gray level generation matrix

number

[0128] The material characteristics of the gray level generation based contrast filter are:

number

[0129] For example, the image filter may be a gray level generation based energy filter, which is based on the gray level generation matrix defined above.

[0130] The material characteristics of the gray level generation based energy filter are:

number

[0131] For example, the image filter may be a gray level generation based uniformity filter, where the material filter is based on the gray level generation matrix defined above. The material characteristics of the gray level generation based uniformity filter are:

number

[0132] For example, the image filter may be a gray level generation-based dissimilarity filter, which is based on the gray level generation matrix defined above. The material features of the gray level generation-based dissimilarity filter are:

number

[0133] For example, the image filter may be a low energy filter, where the material filter is a low vector L5=[1,4,6,4,1] and E5=[-1,-2,0,-2,-1] and material L5(E5) T and E5(L5) T Based on image f k These are the matrices:

number

[0134] where the material characteristics of the low energy filter are:

number

[0135] For example, the material-dependent image filter may be a threshold domain filter. This material feature may relate two areas in the image plane. The first area, Ω1, may be the area where the function f is greater than α times the maximum value of f. The second area, Ω2, may be the area where the function f is less than α times the maximum value of f but greater than a threshold of ε times the maximum value of f. Preferably, α may be 0.5 and ε may be 0.05. Due to speckle or noise, the areas may not simply correspond to the inner and outer circles around the spot center. For example, Ω1 may include speckle or disconnected areas in the outer circle. The material feature may be:

number

[0136] The evaluation device calculates the material characteristic φ to determine the material properties of the surface that generated the reflection characteristic. 2mand a material property of the surface that produced the reflection feature. The predetermined relationship may be one or more of an empirical relationship, a semi-empirical relationship, and an analytically derived relationship. The evaluation device may comprise at least one data storage device for storing the predetermined relationship, such as a look-up list or a look-up table.

[0137] The evaluation device is configured to identify a reflection feature as being generated by irradiating biological tissue if its corresponding material characteristic meets at least one predetermined or predefined criterion. A reflection feature can be identified as being generated by biological tissue if the material characteristic indicates "biological tissue." A reflection feature may be identified as being generated by biological tissue if the material characteristic is lower than or equal to at least one threshold or range, and if the determined deviation is lower than or equal to the threshold, the reflection feature is identified as being generated by biological tissue and / or the detection of biological tissue is confirmed. The at least one threshold and / or range may be stored in a table or look-up table, for example, empirically determined, and may, for example, be stored in at least one data storage device of the display device. The evaluation device is configured to identify a reflection feature as being background otherwise. Accordingly, the evaluation device may be configured to assign depth information and a material characteristic (e.g., yes or no for skin) to each projection spot.

[0138] The material properties are given by the information about the vertical coordinate z in terms of φ 2m After determining the vertical coordinate z, φ can be taken into account for the evaluation of 2m can be determined by evaluating

[0139] In a further aspect, the present invention discloses a method for measuring through a light-transmitting display, in which a display device according to the present invention is used, the method comprising the following steps: a) illuminating at least one scene by using at least one illumination light beam generated by at least one illumination source, said illumination source being positioned in front of the display in a direction of propagation of the illumination beam; b) measuring at least one reflected light beam generated by the scene in response to illumination by the illumination beam by using at least one light sensor, the light sensor having at least one light-sensitive area, the light sensor being positioned in front of the display in a propagation direction of the illumination beam; c) controlling said display by using at least one control unit, wherein said display is turned off in the area of ​​the illumination source during illumination and / or in the area of ​​the light sensor during measurement; Includes:

[0140] The method steps may be performed in a predetermined order or in a different order. Furthermore, one or more additional method steps not listed may be present. Furthermore, one, more than one, or even all of the method steps may be performed repeatedly. For details, options, and definitions, reference may be made to the display device described above. Thus, in particular, as described above, the method may include use of a display device according to the present invention, such as according to one or more embodiments given above or given in more detail below.

[0141] The at least one control unit and / or the at least one evaluation device may be configured to execute at least one computer program, such as at least one computer program configured to perform or support one or more or even all of the method steps of the method according to the invention. By way of example, one or more algorithms may be implemented which are capable of determining the position of an object.

[0142] In a further aspect of the present invention, the use of a display device according to the present invention, such as according to one or more embodiments given above or in more detail below, is proposed for use purposes selected from the group consisting of: position determination in traffic technology; entertainment applications; security applications; surveillance applications; safety applications; human-machine interface applications; tracking applications; photography applications; imaging or camera applications; mapping applications for generating a map of at least one space; homing or tracking beacon detectors for vehicles; outdoor applications; mobile applications; communication applications; machine vision applications; robotics applications; quality control applications; manufacturing applications; automotive applications.

[0143] For example, the display device can be used in vehicle applications such as driver monitoring, personalized vehicles, and the like.

[0144] For further uses of the display device and device of the present invention, reference is made to WO2018 / 091649A1, WO2018 / 091638A1 and WO2018 / 091640A1, the contents of which are incorporated by reference.

[0145] Overall, in the context of the present invention, the following embodiments are considered preferred: Embodiment 1: - at least one illumination source configured to project at least one illumination beam onto at least one scene; - at least one light sensor having at least one light-sensitive area, said light sensor configured to measure at least one reflected light beam generated by said scene in response to illumination by an illumination beam; - at least one light-transmitting display configured to display information, wherein the illumination source and the light sensor are arranged in front of the display in a direction of propagation of an illumination light beam; - at least one control unit configured to switch off the display in the area of ​​the illumination source during illumination and / or in the area of ​​the light sensor during measurements; and A display device comprising:

[0146] Embodiment 2: The display device of the preceding embodiment, wherein the light-transmissive display is a full-size display having a display material that extends across the full size of the display.

[0147] Embodiment 3: A display device according to any one of the preceding embodiments, wherein the display is configured to show a black area in the area of ​​the illumination source and / or in the area of ​​the light sensor when the control unit turns off the display in the area of ​​the illumination source during illumination and / or in the area of ​​the light sensor during measurement.

[0148] Embodiment 4: A display device according to any one of the preceding embodiments, wherein the control unit is configured to turn off the display in the area of ​​the illumination source so that the display in the area of ​​the illumination source acts as an adjustable notch, and / or to turn off the display in the area of ​​the optical sensor so that the display in the area of ​​the optical sensor acts as an adjustable notch, the adjustable notch being configured to be active during illumination and / or measurement and inactive otherwise.

[0149] Embodiment 5: A display device as described in any one of the preceding embodiments, wherein the display device is configured to perform facial recognition using the optical sensor, the control unit is configured to emit an indication indicating that facial recognition is active while facial recognition is being performed, and the translucent display is configured to display the indication while facial recognition is being performed.

[0150] Embodiment 6: The display device of any one of the preceding embodiments, wherein the optical sensor includes at least one CMOS sensor.

[0151] Embodiment 7: The display device of any one of the preceding embodiments, wherein the illumination source includes at least one infrared light source.

[0152] Embodiment 8: A display device according to any one of the preceding embodiments, wherein the illumination source comprises at least one laser projector, the laser projector comprising at least one laser light source and at least one diffractive optical element (DOE).

[0153] Embodiment 9: A display device according to any one of the preceding embodiments, wherein the illumination source is configured to generate at least one illumination pattern, the illumination pattern comprising a periodic dot pattern.

[0154] Embodiment 10: The display device of any one of the preceding embodiments, wherein the illumination source comprises at least one flood illumination light emitting diode.

[0155] Embodiment 11: A display device according to any one of the preceding embodiments, wherein the display, the illumination source and the light sensor are synchronized.

[0156] Embodiment 12: A display device according to any one of the preceding embodiments, wherein the display is or comprises at least one organic light-emitting diode (OLED) display, and wherein the OLED display is inactive in the area of ​​the illumination source when the control unit turns off the display in the area of ​​the illumination source, and / or the OLED display is inactive in the area of ​​the light sensor when the control unit turns off the display in the area of ​​the light sensor.

[0157] Embodiment 13: The illumination source is configured to project at least one illumination pattern onto the at least one scene, the illumination pattern including a plurality of illumination features; the optical sensor is configured to determine, in response to illumination by the illumination features, at least one first image including a plurality of reflection features produced by the scene; the display device further comprises at least one evaluation device, the evaluation device being configured to evaluate the first image, the evaluation of the first image comprising identifying reflection features of the first image and sorting the identified reflection features with respect to brightness, each of the reflection features comprising at least one beam profile; and the evaluation device being configured to determine, by analysis of the beam profile, at least one vertical coordinate z for each of the reflection features. DPR and the evaluation device is configured to determine the vertical coordinate z DPR and the evaluation device is configured to unambiguously match a reflection feature with a corresponding illumination feature using a vertical coordinate z DPR 10. A display device according to any one of the preceding embodiments, configured to generate a depth map of the real features by using

[0158] Embodiment 14: The evaluation device uses triangulation and / or depth from defocus and / or structured light techniques to determine for each of the reflective features at least one second vertical coordinate z triang 4. The display device of any preceding embodiment, configured to determine:

[0159] Embodiment 15: The evaluation device is configured to triang and the vertical coordinate z DPR and the bond vertical coordinate is determined by the second vertical coordinate z triang and the vertical coordinate zDPR , and the combined vertical coordinate is used to generate a depth map.

[0160] Embodiment 16: A display device described in any one of the preceding three embodiments, wherein the evaluation device is configured to determine beam profile information for each of the reflective features by using a depth from photon ratio technique.

[0161] Embodiment 17: A display device described in any one of the preceding four embodiments, wherein the evaluation device is configured to determine at least one material property m of the object by evaluating the beam profile of at least one of the reflective features.

[0162] Embodiment 18: The display device comprises a further illumination source, the further illumination source comprising at least one light emitting diode (LED), the further illumination source configured to generate light in the visible spectral range, the light sensor configured to determine at least one second image comprising at least one two-dimensional image of the scene, the further illumination source configured to provide additional illumination for imaging of the second image, and the evaluation device configured to determine at least one corrected image I0 by deconvolving the second image I with a grating function g, where I=I0 * 10. The display device of any one of the preceding embodiments, wherein

[0163] Embodiment 19: A display device according to any one of the preceding embodiments, wherein the display device is a mobile device selected from the group consisting of a television device, a mobile phone, a smartphone, a game console, a tablet computer, a personal computer, a laptop, a tablet, a virtual reality device, or another type of portable computer.

[0164] Embodiment 20: A method for measuring through a light-transmitting display, in which at least one display device according to any one of the preceding embodiments is used, said method comprising the steps of: a) illuminating at least one scene by using at least one illumination light beam generated by at least one illumination source, said illumination source being positioned in front of the display in a direction of propagation of the illumination beam; b) measuring at least one reflected light beam generated by the scene in response to illumination by an illumination beam by using at least one light sensor, the light sensor having at least one light-sensitive area, the light sensor being positioned in front of the display in a propagation direction of the illumination beam; c) controlling said display by using at least one control unit, wherein said display is turned off in the area of ​​the illumination source during illumination and / or in the area of ​​the light sensor during measurement; 10. A method for measuring through a translucent display, comprising:

[0165] Embodiment 21: Use of a display device described in any one of the preceding embodiments relating to a display device, wherein the purpose of use is selected from the group consisting of: position measurement in traffic technology; entertainment applications; security applications; surveillance applications; safety applications; human-machine interface applications; tracking applications; photography applications; imaging or camera applications; mapping applications for generating maps of at least one space; homing or tracking beacon detectors for vehicles; outdoor applications; mobile applications; communication applications; machine vision applications; robotics applications; quality control applications; manufacturing applications; automotive applications. [Brief explanation of the drawings]

[0166] Further optional details and features of the invention are evident from the following description of preferred exemplary embodiments in conjunction with the dependent claims. In this context, certain features can be implemented in isolation or in combination with other features. The invention is not limited to the exemplary embodiments, which are illustrated diagrammatically in the figures. The same reference numerals in the individual figures refer to identical elements or elements with the same function, or elements which correspond to each other in terms of function.

[0167] Specifically, in the drawing: [Figure 1] 1 illustrates an embodiment of a display device according to the present invention. [Figure 2] 2A-2C show an embodiment for synchronizing a display, illumination source, and light sensor. DETAILED DESCRIPTION OF THE INVENTION

[0168] Detailed Description of the Embodiments: 1 highly schematically illustrates one embodiment of a display device 1 of the present invention. For example, the display device may be a mobile device selected from the group consisting of a television device, a mobile phone, a smartphone, a game console, a tablet computer, a personal computer, a laptop, a tablet, a virtual reality device, or another type of portable computer.

[0169] The display device 1 comprises at least one light-transmissive display 2 configured to display information. The display device 1 comprises at least one light sensor 4 having at least one light-sensitive area. The display device 1 comprises at least one illumination source 5 configured to project at least one illumination beam onto at least one scene. The scene may be any object or region of space. The scene may include at least one object and its surrounding environment.

[0170] The illumination source 5 is configured to project an illumination beam, in particular at least one illumination pattern comprising a plurality of illumination features, onto the scene. The illumination source 5 may be adapted to directly or indirectly illuminate the scene, the illumination beam being reflected or scattered by surfaces of the scene and thereby directed at least in part towards the light sensor. The illumination source 5 may be adapted to illuminate the scene reflecting the light beam, for example by directing a light beam towards the scene.

[0171] The illumination source 5 may include at least one light source. The illumination source 5 may also include multiple light sources. The illumination source 5 may include an artificial illumination source, in particular at least one laser source, and / or at least one incandescent lamp, and / or at least one semiconductor light source, such as at least one light-emitting diode, in particular an organic and / or inorganic light-emitting diode. The illumination source 5 may comprise at least one infrared light source. The illumination source 5 may be configured to generate at least one illumination light beam, in particular at least one illumination pattern, in the infrared range. The use of light in the near-infrared range allows light that is not or only weakly detectable by the human eye but can be detected by silicon sensors, in particular standard silicon sensors.

[0172] The illumination source 5 may comprise at least one laser projector. The laser projector may be a vertical cavity surface emitting laser (VCSEL) projector combined with refractive optics. However, other embodiments are possible. The laser projector may comprise at least one laser source and at least one diffractive optical element (DOE). The illumination source 5 may be or include at least one multiple beam light source. For example, the illumination source 5 may include at least one laser source and one or more diffractive optical elements (DOE). In particular, the illumination source 5 may comprise at least one laser and / or laser source. Various types of lasers may be used, such as semiconductor lasers, double heterostructure lasers, external cavity lasers, separate confinement heterostructure lasers, quantum cascade lasers, distributed Bragg reflector lasers, polariton lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, indium arsenide lasers, transistor lasers, diode-pumped lasers, distributed feedback lasers, quantum well lasers, interband cascade lasers, gallium arsenide lasers, semiconductor ring lasers, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alternatively, non-laser light sources, such as LEDs and / or light bulbs, may be used. The illumination source 5 may include one or more diffractive optical elements (DOEs) adapted to generate an illumination pattern. For example, the illumination source 5 may be adapted to generate and / or project a point cloud, and may, for example, include one or more of: at least one digital light processing projector; at least one LCoS projector; at least one spatial light modulator; at least one diffractive optical element; at least one array of light-emitting diodes; or at least one array of laser light sources. Considering their generally defined beam profile and other characteristics of ease of handling, the use of at least one laser source as the illumination source 5 is particularly preferred. The illumination source 5 may be integrated into the housing of the display device.

[0173] The illumination source 5 is configured to generate at least one illumination pattern. The illumination pattern may include a plurality of illumination features. The illumination pattern may be selected from the group consisting of at least one dot pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; and at least one pattern including a periodic or aperiodic feature arrangement. The illumination pattern may include a regular and / or regular and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or even a convex tiling pattern. The illumination pattern may exhibit at least one illumination feature selected from the group consisting of at least one dot; at least one line; at least two lines, such as parallel or intersecting lines; at least one dot and one line; at least one periodic or aperiodic feature arrangement; and at least one feature of any shape. The illumination pattern may include at least one pattern selected from the group consisting of: at least one dot pattern, particularly a pseudorandom dot pattern; a random dot pattern or a quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern including at least one known feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern; at least one pattern including a convex uniform tiling; at least one line pattern including at least one line; or at least one line pattern including at least two lines, such as parallel or intersecting lines. For example, the illumination source 5 may be adapted to generate and / or project a point cloud. The illumination source 5 may include at least one light projector adapted to generate a point cloud so that the illumination pattern can include multiple point patterns. The illumination source 5 may include at least one mask adapted to generate the illumination pattern from at least one light beam generated by the illumination source 5.

[0174] The light sensor 4 is configured to measure at least one reflected light beam generated by the scene in response to illumination by the illumination beam. The display device 1 may comprise a single camera with the light sensor 4. The display device 1 may comprise multiple cameras, each with one or more light sensors 4. The display device 1 may include multiple light sensors 4, each with a light-sensitive area.

[0175] The display device 1 may be configured to perform at least one distance measurement, such as based on a time-of-flight (ToF) technique, and / or based on a depth from defocus technique, and / or based on a depth from photon ratio technique, also called beam profile analysis. Regarding the depth from photon ratio (DPR) technique, reference is made to WO2018 / 091649A1, WO2018 / 091638A1 and WO2018 / 091640A1, the entire contents of which are incorporated by reference. The optical sensor 4 may be or comprise at least one distance sensor.

[0176] The photosensor 4 may be or include at least one photodetector, preferably an inorganic photodetector, more preferably an inorganic semiconductor photodetector, and most preferably a silicon photodetector. Specifically, the photosensor 4 may be sensitive in the infrared spectral range. All pixels of the matrix, or at least a group of photosensors of the matrix, may be identical. Groups of identical pixels of the matrix may be provided for different spectral ranges, or all pixels may be identical in terms of spectral sensitivity. Furthermore, the pixels may be identical in terms of size and / or their electronic or optoelectronic properties. Specifically, the photosensor 4 may be or include at least one inorganic photodiode sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers. Specifically, the photosensor 4 may be sensitive in the part of the near-infrared range, particularly in the range of 700 nm to 1100 nm, where silicon photodiodes are applicable. The infrared light sensor that can be used in the optical sensor may be a commercially available infrared light sensor, such as the infrared light sensor sold under the brand name Hertzstueck® by trinamiX GmbH, D-67056 Ludwigshafen am Rhein, Germany. Thus, by way of example, the optical sensor 4 may include at least one optical sensor of an intrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of a Ge photodiode, an InGaAs photodiode, an extended InGaAs photodiode, an InAs photodiode, an InSb photodiode, and an HgCdTe photodiode. Additionally or alternatively, the optical sensor 4 may include at least one optical sensor of an extrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of a Ge:Au photodiode, a Ge:Hg photodiode, a Ge:Cu photodiode, a Ge:Zn photodiode, a Si:Ga photodiode, and a Si:As photodiode.Additionally or alternatively, the optical sensor 4 may comprise at least one photoconductive sensor, such as a PbS or PbSe sensor, a bolometer, preferably a bolometer selected from the group consisting of a VO bolometer and an amorphous Si bolometer.

[0177] For example, the light sensor 4 may be or include at least one element selected from the group consisting of a photodiode, a photocell, a photoconductor, a phototransistor, or any combination thereof. For example, the light sensor 4 may be or include at least one element selected from the group consisting of a CCD sensor element, a CMOS sensor element, a photodiode, a photocell, a photoconductor, a phototransistor, or any combination thereof. Any other type of photosensitive element may also be used. The photosensitive element generally can be made entirely or partially of inorganic materials and / or can be made entirely or partially of organic materials. Most commonly, one or more photodiodes, such as commercially available photodiodes, e.g., inorganic semiconductor photodiodes, may be used.

[0178] The display device 1 comprises at least one translucent display 2 configured to display information. An illumination source 5 and a light sensor 4 are arranged in front of the translucent display 2 in the direction of propagation of the illuminating light beam. The translucent display 2 may be or may comprise at least one screen. The screen may have any shape, preferably rectangular. The information displayed by the display 2 may be any information, such as at least one image, at least one diagram, at least one histogram, at least one graphic, text, numbers, at least one symbol, an operating menu, etc.

[0179] The light-transmitting display 2 may be a full-size display having a display material extending across the full size of the display 2. The light-transmitting display 2 may be free of recesses or cutouts. The light-transmitting display 2 may have an entire active display area. The light-transmitting display 2 may be designed so that the entire display area is activatable. The light-transmitting display 2 may have a continuous distribution of display material. The light-transmitting display 2 may be designed so that there are no recesses or cutouts. For example, the display device 1 may include a front side having a display area, such as a rectangular display area, on which the light-transmitting display 2 is disposed. The display area may be completely covered by the light-transmitting display, particularly the display material, specifically so that there are no recesses or notches. This may allow for a larger display size, particularly the area of ​​the display device 1 configured to display information. For example, the entire and / or entire front size of the display device 1 may be covered by the display material, although a frame surrounding the display 2 may be possible.

[0180] The light-transmissive display 2 may be or include at least one organic light-emitting diode (OLED) display. The OLED may be configured to emit visible light.

[0181] The display device comprises at least one control unit 8. The control unit 8 is configured to turn off the display 2 in the area of ​​the illumination source 5 during illumination and / or in the area of ​​the light sensor 4 during measurement. The control unit 8 may be configured to control at least one further component of the display device 1, such as the illumination source 5 and / or the light sensor 2 and / or the display 2, in particular by using at least one processor and / or at least one application-specific integrated circuit. Thus, by way of example, the control unit 8 may comprise at least one data processing device having stored therein software code comprising a number of computer commands. The control unit 8 may provide one or more hardware elements for performing one or more specified operations and / or one or more processors having software running thereon for performing one or more specified operations. Thus, by way of example, the control unit 8 may comprise one or more programmable devices, such as one or more computers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), or field-programmable gate arrays (FPGAs), configured to perform the above-mentioned controls. Additionally or alternatively, however, the control unit 8 may also be embodied fully or partly by hardware.

[0182] The control unit 8 is configured to turn off the display 2 in the area of ​​the irradiation source 5 during irradiation and / or in the area of ​​the light sensor 4 during measurement. Turning off the display 2 in an area may in particular involve adjusting to prevent and / or interrupt and / or stop the power supply to a particular area of ​​the display 2. As outlined above, the display 2 may comprise at least one OLED display. When the control unit 8 turns off the display 2 in the area of ​​the irradiation source 5, the OLED display may be inactive in the area of ​​the irradiation source 5. When the control unit 8 turns off the display 2 in the area of ​​the light sensor 4, the OLED display may be inactive in the area of ​​the light sensor 4. The control unit 8 may be configured to turn off an area of ​​the display 2 while measurement is active.

[0183] The illumination source 5 may have an emission area from which an illumination beam, in particular an illumination pattern, is emitted towards the scene. The emission area may be defined by an aperture angle of the illumination source 5. The illumination source 5 and the light sensor 4 may be arranged in a defined area. The illumination source 5 and the light sensor 4 may be arranged at a fixed position relative to each other. For example, the illumination source 5 and the light sensor 4 may be arranged adjacent to each other, in particular at a constant distance. The illumination source 5 and the light sensor 4 may be arranged such that the area of ​​the translucent display 2 covered by the emission area and the light-sensitive area is minimized.

[0184] The display 2 may be configured to show a black area 6 in the area of ​​the illumination source 5 and / or in the area of ​​the light sensor 4 when the control unit 8 turns off the display 2 in the area of ​​the illumination source 4 during illumination and / or in the area of ​​the light sensor 4 during measurement. The black area 6 may be an area that does not emit light and / or an area that has a reduced amount of light compared to other areas of the display 2. For example, the black area 6 may be a darkened area. Specifically, the control unit 8 is configured to turn off the display 2 in the area of ​​the illumination source 5 so that the display 2 in the area of ​​the illumination source 5 functions as an adjustable notch, and / or to turn off the display 2 in the area of ​​the light sensor 4 so that the display 2 in the area of ​​the light sensor 4 functions as an adjustable notch. The adjustable notch may be configured to be active during illumination and / or measurement and inactive otherwise. The adjustable notch may function as a virtual notch that is active when the display device 1 is not in use, such as during measurement, such as during face unlock, and inactive when the light sensor 4, particularly the front sensor, is not needed. In the case of an OLED display used, this may mean no activation of the display 2 at all. This may make it possible to ensure that the color of any pixel is not changed by IR light. Furthermore, the display device 1, in particular the control unit 8 and / or further processing devices and / or further optical elements, may be configured to perceive, for example, IR laser flicker and correct the color of the display.

[0185] The adjustable notch may include a sharp edge. However, in other embodiments, the adjustable notch may be realized with a luminance gradient to avoid any sharp fringes. The display device 1 may comprise a luminance reduction element configured to introduce a luminance gradient at the edge of the display 2 where the light sensor 4 is normally located to avoid sharp fringes. This may make it possible to provide a reduced luminance in the area of ​​the adjustable notch.

[0186] The control unit 8 can be configured to synchronize the display 2 and the illumination source 5 in a way that they do not interfere with each other, in a so-called toggle mode. The control unit 8 may be configured to emit an indication when the light sensor 4 and / or the illumination source 5 are active. The translucent display 2 may be configured to display the indication when the light sensor 4 and / or the illumination source 5 are active. For example, the display device 1 may be configured to perform facial recognition using the illumination source 5 and the light sensor 4. Methods and technologies for facial recognition are generally known to those skilled in the art. The control unit 8 may be configured to emit an indication indicating that facial recognition is active while facial recognition is being performed. The translucent display 2 may be configured to display the indication while facial recognition is being performed. For example, the indication may be at least one warning element. The indication may be one or more of an icon and / or a logo and / or a symbol and / or an animation indicating that the light sensor 4 and / or the illumination source 5, particularly facial recognition, are active. For example, the black area 6 may include an identification mark indicating that secure authentication is active. This allows a user to recognize that they are in a secure environment for, for example, payment, signing, etc. For example, the warning element may change color and / or appearance to indicate that face recognition is active. This indication may further allow the user to recognize that the light sensor, in particular the camera, is on to avoid espionage. The control unit 8 and / or the further safety zone may be configured to issue at least one command to display at least one watermark in the black area. The watermark may be a symbol that cannot be imitated by low-security apps, for example, a store.

[0187] However, positioning the illumination source 5 and light sensor 4 in the direction of propagation of light reflected by the scene behind the light-transmissive display 2 may result in the display's diffraction grating generating multiple laser points on the scene and also in the image captured by the light sensor 4. These multiple spots on the image may therefore not contain useful distance information. The display device 1 may comprise at least one evaluation device 10 configured to find and evaluate the zero-order reflection features of the diffraction grating, i.e., real features, and can ignore higher-order reflection features, i.e., spurious features.

[0188] The illumination source 5 may be configured to project at least one illumination pattern including a plurality of illumination features onto at least one scene. The optical sensor 4 may be configured to determine at least one first image including a plurality of reflection features produced by the scene in response to illumination by the illumination features. The display device 1 may further comprise at least one evaluation device 10 configured to evaluate the first image, wherein evaluation of the first image includes identifying reflection features of the first image and sorting the identified reflection features with respect to brightness. Each of the reflection features may include at least one beam profile. The evaluation device 10 may determine, by analysis of the beam profile, at least one vertical coordinate z for each of the reflection features. DPR The evaluation device 10 can be configured to determine the vertical coordinate z DPR The evaluation device 10 may be configured to unambiguously match a reflection feature with a corresponding illumination feature using the vertical coordinate z. The matching may be performed starting with the brightest reflection feature and decreasing the brightness of the reflection feature. The evaluation device 10 may be configured to classify reflection features that match illumination features as real features and reflection features that do not match illumination features as false features. The evaluation device 10 may reject false features and classify reflection features that do not match illumination features as false features. DPR can be configured to generate a depth map of the real features by using

[0189] The evaluation device 10 may be configured to perform at least one image analysis and / or image processing to identify reflective features. The image analysis and / or image processing may use at least one feature detection algorithm. The image analysis and / or image processing may include: filtering; selecting at least one region of interest; forming a difference image between an image generated by the sensor signal and at least one offset; inverting the sensor signal by inverting the image generated by the sensor signal; forming a difference image between images generated by the sensor signal at different times; background correction; decomposition into color channels; decomposition into hues; saturation; luminance channel; frequency decomposition; singular value decomposition; applying a blob detector; applying a corner detector; applying a determinant of a Hessian filter; applying a principal curvature-based region detector; applying a maximum stable extremum region detector; applying a generalized Hough transform; applying a ridge detector; applying an affine-invariant feature detector; applying an affine-adaptive interest point operator; applying a Harris affine region detector; applying a Hessian affine region detector; applying a scale-invariant feature transform. The method may include one or more of: applying a scale-space extremum detector; applying a local feature detector; applying a fast robust feature algorithm; applying a histogram of gradient position and orientation algorithm; applying a histogram of oriented gradient descriptors; applying a Deriche edge detector; applying a differential edge detector; applying a space-time interest point detector; applying a Moravec corner detector; applying a Canny edge detector; applying a Laplace operator of a Gaussian filter; applying a differential Gaussian filter; applying a Sobel operator; applying a Laplace operator; applying a Schall operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying a Radon transform; applying a Hough transform; applying a wavelet transform; thresholding; or generating a binary image. The region of interest may be determined manually by a user or automatically, such as by recognizing features in an image generated by an optical sensor.

[0190] For example, the illumination source 5 may be configured to generate and / or project a point cloud such that a plurality of illumination areas are generated on the optical sensor 4, e.g., a CMOS detector. Furthermore, disturbances, e.g., due to speckle and / or extraneous light and / or multiple reflections, may be present at the optical sensor. The evaluation device 10 may be adapted to determine at least one region of interest, e.g., one or more pixels illuminated by the light beam, which are used to determine the longitudinal coordinate of the object. For example, the evaluation device 10 may be adapted to perform filtering methods, e.g., blob analysis and / or edge filters and / or object recognition methods.

[0191] The evaluation device 10 may be configured to perform at least one image correction, which may include at least one background subtraction, and may be adapted to remove background light from the beam profile, for example by imaging without further illumination.

[0192] Each of the reflective features includes at least one beam profile, which may be selected from the group consisting of a trapezoidal beam profile, a triangular beam profile, a conical beam profile, and a linear combination of a Gaussian beam profile. The evaluation device 10 is configured to determine beam profile information for each of the reflective features by analyzing the beam profiles.

[0193] The evaluation device 10 determines, by analysis of the beam profile, at least one vertical coordinate z for each of the reflection features. DPRThe evaluation device 10 is configured to determine the beam profile. For example, the analysis of the beam profile may include at least one of a histogram analysis step, calculation of difference measurements, application of a neural network, and application of a machine learning algorithm. The evaluation device 10 may be configured to symmetrize and / or normalize and / or filter the beam profile to remove noise or asymmetries from the recording, especially recordings at larger angles, edges, etc. The evaluation device 10 may filter the beam profile by removing high spatial frequencies, such as by spatial frequency analysis and / or median filtering. Aggregation may be performed by the center of the intensity of the light spot and by averaging all intensities at the same distance to the center. The evaluation device 10 may be configured to normalize the beam profile to a maximum intensity, especially to take into account intensity differences depending on the recorded distance. The evaluation device 10 may be configured to remove the influence of background light from the beam profile, for example by imaging without illumination.

[0194] The evaluation device 10 calculates the vertical coordinate z for each of the reflective features by using the depth from photon ratio technique. DPR For Depth from Photon Ratio (DPR) techniques, see WO2018 / 091649A1, WO2018 / 091638A1 and WO2018 / 091640A1, the entire contents of which are incorporated by reference.

[0195] The evaluation device 10 can be configured to determine a beam profile for each of the reflective features. Determining the beam profile may include identifying at least one reflective feature provided by the optical sensor 4 and / or selecting at least one reflective feature provided by the optical sensor 4 and evaluating the intensity distribution of at least one of the reflective features. As an example, a region of the image may be used and evaluated to determine an intensity distribution, such as a three-dimensional intensity distribution or a two-dimensional intensity distribution, such as along an axis or line through the image. As an example, the center of illumination by the light beam may be determined, such as by determining at least one pixel with the highest illumination, and a cross-sectional axis may be selected through the center of illumination. The intensity distribution may be the intensity distribution as a function of coordinate along this cross-sectional axis through the center of illumination. Other evaluation algorithms are also possible.

[0196] The analysis of one beam profile of the reflection features may include determining at least one first area and at least one second area of ​​the beam profile. The first area of ​​the beam profile may be area A1, and the second area of ​​the beam profile may be area A2. The evaluation device 10 may be configured to integrate the first area and the second area. The evaluation device 10 may be configured to derive a combined signal, in particular a quotient Q, by one or more of: dividing the integrated first area and the integrated second area; dividing a multiple of the integrated first area and the integrated second area; or dividing a linear combination of the integrated first area and the integrated second area. The evaluation device 10 may be configured to determine at least two areas of the beam profile and / or divide the beam profile into at least two segments having different areas of the beam profile, although overlapping of the areas is possible as long as the areas do not coincide. For example, the evaluation device 10 may be configured to determine multiple areas, such as two, three, four, five, or up to ten areas. The evaluation device 10 may be configured to divide the light spot into at least two areas of the beam profile and / or divide the beam profile into at least two segments comprising different areas of the beam profile. The evaluation device 10 may be configured to determine, for at least two of the areas, an integral of the beam profile over each area. The evaluation device 10 may be configured to compare at least two of the determined integrals. Specifically, the evaluation device 10 may be configured to determine at least one first area of ​​the beam profile and at least one second area. The first area of ​​the beam profile and the second area of ​​the beam profile may be adjacent areas or overlapping areas, or both. The first area of ​​the beam profile and the second area of ​​the beam profile may not be coincident in area.For example, the evaluation device 10 may be configured to divide the sensor area of ​​the CMOS sensor into at least two sub-areas, and the evaluation device may be configured to divide the sensor area of ​​the CMOS sensor into at least one left part and at least one right part, and / or at least one upper part and at least one lower part, and / or at least one inner part and at least one outer part.

[0197] Additionally or alternatively, the display device 1 may comprise at least two light sensors 4, the light sensitive areas of the first and second light sensors being arranged such that the first light sensor is adapted to determine a first area of ​​the beam profile of the reflective feature and the second light sensor is adapted to determine a second area of ​​the beam profile of the reflective feature. The evaluation device 10 may be adapted to integrate the first area and the second area.

[0198] In one embodiment, A1 may correspond to the entire or complete area of ​​the feature on the optical sensor. A2 may be the central area of ​​the feature on the optical sensor. The central area may be a constant value. The central area may be smaller than the total area of ​​the feature. For example, for a circular feature, the central area may have a radius between 0.1 and 0.9 of the total radius of the feature, preferably between 0.4 and 0.6 of the total radius.

[0199] The evaluation device 10 may be configured to derive the quotient Q by one or more of dividing the first area by the second area, dividing the first area by a multiple of the second area, or dividing a linear combination of the first area by the second area.

number

[0200] The evaluation device 10 may be configured to use at least one predetermined relationship between the quotient Q and the vertical coordinate to determine the vertical coordinate. The predetermined relationship may be one or more of an empirical relationship, a semi-empirical relationship, and an analytically derived relationship. The evaluation device may include at least one data storage device for storing the predetermined relationship, such as a look-up list or a look-up table.

[0201] The evaluation device 10 may be configured to execute at least one depth from photon ratio algorithm that calculates distances for all zeroth and higher order reflectance features.

[0202] Evaluation of the first image may include sorting the identified reflective features with respect to brightness. The sorting may include assigning the reflective features with respect to brightness for further evaluation, particularly assigning a sequence of reflective features with decreasing brightness, starting with the reflective feature with the greatest brightness. If the brightest reflective feature is prioritized in the DPR calculation, then the vertical coordinate z DPR This can increase the robustness of the determination of , mainly because the zero-order reflection feature of the diffraction grating is always brighter than the higher-order spurious features.

[0203] The evaluation device 10 calculates the vertical coordinate z DPR The technique is configured to unambiguously match reflection features with corresponding illumination features using a depth from photon ratio technique. The vertical coordinates determined by the depth from photon ratio technique can be used to solve the so-called correspondence problem. In that way, the distance information for each reflection feature can be used to find the correspondence of the known laser projector grid.

[0204] Illumination features corresponding to reflection features can be determined using epipolar geometry. For a description of epipolar geometry, see, for example, Chapter 2 of "Dreidimensionales Computersehen" by X. Jiang and H. Bunke, Springer, Berlin Heidelberg, 1997. Epipolar geometry may assume that the illumination image, i.e., an image of an undistorted illumination pattern, and the first image are images determined at different spatial positions and / or spatial orientations with a fixed distance. The distance is a relative distance and is also referred to as a baseline. The illumination image is also referred to as a reference image. The evaluation device 10 may be adapted to determine an epipolar line in the reference image. The relative positions of the reference image and the first image may be known. For example, the relative positions of the reference image and the first image may be stored in at least one storage unit of the evaluation device. The evaluation device 10 may be adapted to determine a straight line extending from a selected reflection feature in the first image to the real-world feature from which it originates. Thus, the straight line may include a possible object feature corresponding to the selected reflection feature. The line and the baseline define an epipolar plane. Because the reference image is determined at a different relative position from the first image, possible corresponding object features can be imaged on a line called an epipolar line in the reference image. The epipolar line can be the intersection of the epipolar plane and the reference image. Thus, a feature in the reference image that corresponds to a selected feature in the first image lies on the epipolar line.

[0205] Depending on the distance to the object in the scene that reflected the illumination feature, the reflection feature corresponding to the illumination feature may be displaced in the first image. The reference image may include at least one displacement region in which an illumination feature corresponding to a selected reflection feature is imaged. The displacement region may include only one illumination feature. The displacement region may include multiple illumination features. The displacement region may include an epipolar line or a section of the epipolar line. The displacement region may include multiple epipolar lines or multiple sections of multiple epipolar lines. The displacement region may extend along the epipolar line, or orthogonal to the epipolar line, or both. The evaluation device 10 may be adapted to determine the illumination feature along the epipolar line. The evaluation device 10 may be adapted to determine the vertical coordinate z of the reflection feature and an error interval ±ε from the combined signal Q, and determine a displacement region along the epipolar line or orthogonal to the epipolar line corresponding to ±ε. The measurement uncertainty of the distance measurement using the combined signal Q may result in a non-circular displacement region in the second image, since the measurement uncertainty may vary depending on the direction. Specifically, the measurement uncertainty along the epipolar line may be larger than the measurement uncertainty in the direction perpendicular to the epipolar line. The displacement region may include an area extending perpendicular to the epipolar line or multiple epipolar lines. The evaluation device 10 may be adapted to match the selected reflection feature with at least one illumination feature in the displacement region. The evaluation device 10 may be adapted to match the determined vertical coordinate z DPRThe evaluation device 10 may be adapted to match selected features of the first image with illumination features within the displacement region by using at least one evaluation algorithm that takes into account the following: (a) the distance between the selected feature of the first image and the illumination feature within the displacement region; (b) the distance between the selected feature of the first image and the illumination feature within the displacement region; (c) the distance between the selected feature of the first image and the illumination feature within the displacement region; (d) the distance between the selected feature of the first image and the illumination feature within the displacement region; (e) the distance between the selected feature of the first image and the illumination feature within the displacement region; (f) the distance between the selected feature of the first image and the illumination feature within the displacement region; (g) the distance between the selected feature of the first image and the illumination feature within the displacement region; (h) the distance between the selected feature of the first image and the illumination feature within the displacement region; (i) the distance between the selected feature of the first image and the illumination feature within the displacement region; (j) the distance between the selected feature of the first image and the illumination feature within the displacement region; The evaluation device 10 may be adapted to determine the illumination feature corresponding to the reflection feature by determining the illumination feature closest to the assigned displacement region and / or the illumination feature within the assigned displacement region and / or the illumination feature closest to the assigned displacement region along the assigned epipolar line and / or the illumination feature within the assigned displacement region along the assigned epipolar line.

[0206] Additionally or alternatively, the evaluation device 10 performs the following steps: - determining a displacement area of ​​the image location of each reflection feature; - assigning epipolar lines to the displacement region of each reflection feature, such as by assigning the epipolar line closest to the displacement region and / or the epipolar line within the displacement region and / or the epipolar line closest to the displacement region along a direction orthogonal to the epipolar line; - assigning and / or determining at least one illumination feature to each reflection feature, such as by assigning the illumination feature closest to the assigned displacement area and / or the illumination feature within the assigned displacement area and / or the illumination feature closest to the assigned displacement area along the assigned epipolar line and / or the illumination feature within the assigned displacement area along the assigned epipolar line; may be configured to perform

[0207] Additionally or alternatively, the evaluation device 10 may be adapted to decide between a plurality of epipolar lines and / or illumination features to be assigned to a reflection feature, such as by comparing distances of the reflection features and / or epipolar lines in the illumination image and / or by comparing error-weighted distances, such as ε-weighted distances, of the illumination features and / or epipolar lines in the illumination image, and assigning the epipolar line and / or illumination feature with the shorter distance and / or ε-weighted distance to the illumination feature and / or reflection feature.

[0208] As explained above, the diffraction grating generates multiple reflection features, e.g., one real feature and multiple false features for each illumination feature. Matching is performed starting with the brightest reflection feature and decreasing the brightness of the reflection features. Other reflection features cannot be assigned to matched illumination features. Due to display artifacts, the generated false features are generally darker than the real features. By sorting the reflection features by brightness, brighter reflection features are prioritized in correspondence matching. If the illumination feature correspondence has already been used, the false feature cannot be assigned to the used, i.e., matched, illumination feature.

[0209] 2A-2C show an embodiment in which the display 2, illumination source 5 and light sensor 4 are synchronized.

[0210] 2A, the display device 1 may comprise at least one projector 12, e.g., a laser projector module, configured to generate at least one illumination pattern, an additional flood illumination 14 for illuminating a scene, and an illumination source 5 including a light sensor 4, e.g., an IR camera module with a shutter. The display device 1 may be configured such that these components are positioned in front of the display 2 in the propagation direction of the illumination light beam.

[0211] The light-transmitting display 2 may be at least one OLED display. The OLED display may have a transmittance of approximately 25% or greater. However, even lower transmittance OLED display embodiments may be possible. FIG. 2C illustrates an OLED display embodiment. Reference numeral 16 indicates potential locations for the IR camera module, projector 12, and flood illumination 14. The display 2 may have a resolution VxH (where V is the vertical extension and H is the height). The OLED display may comprise a plurality of pixels arranged in a matrix array VxH. The OLED may update and / or refresh its content row by row from top to bottom of the matrix. In FIG. 2C, the first line, e.g., line 0, is indicated by reference numeral 18, and the last line V is indicated by reference numeral 20. The update direction is indicated by reference numeral 22. The control unit 8 may be configured to synchronize the display 2, projector 12, flood illumination 14, and light sensor 4. Elements of the control unit 8 are shown in FIG. 2, for example, as part of the SoC 26 and / or as elements of the display 2, light sensor 4 and illumination sources 12, 14. The display 2, and in particular the display driver, may be configured to emit at least one signal indicating that updating and / or refreshing will wrap around from the last row 20 to the first row 18. The display driver may be part of the control unit. For example, a vertical synchronization (VSYNC) signal, also denoted display VSYNC 24, may be emitted by the display 2 when updating and / or refreshing will wrap around from the last row 20 to the first row 18.

[0212] The emission of light through the OLED display can be timed slightly before the content is updated and / or refreshed, particularly overwritten, thereby minimizing visible distortion. The light sensor 4 can be synchronized with the projector 12 and the flood illumination 14. The light sensor 4 can be active during illumination, i.e., in a mode for capturing an image and / or in a mode for detecting light. For example, synchronization of the light sensor 4 with the illumination source 5 can be achieved as shown in FIG. 2A.

[0213] As shown in FIG. 2A , the control unit may include a system-on-chip (SoC) 26. The SoC 26 may include a display interface 28. The SoC 26 may include at least one application programming interface (API) 30 connected to at least one application 32. The SoC 26 may further include at least one image signal processor (ISP) 34. The light sensor 4 may be connected to the SoC 26, particularly the ISP 34 and / or the API 30, via a connection 35. The connection 35 may be configured for one or more of power control, providing a clock signal (CLK), and transferring image signals. Additionally or alternatively, the connection 35 may be embodied as an Inter-Integrated Circuit (I2C). Additionally or alternatively, the connection may be embodied as an image data interface, such as a MIP.

[0214] The application 32 can request illumination 40 from one or more of the illumination sources 12, 14. The SoC 26, via the API 30, can provide power to the light sensor 4 via connection 35. The light sensor 4 can issue a VSYNC signal, also referred to as camera VSYNC 36, to the SoC 26 and a strobe signal 38 to the illumination sources 12, 14. The SoC 26, via the API 30, can issue trigger signals 41, 42 to the illumination sources 12, 14 in response to the camera VSYNC 36 to activate the illumination sources, respectively. When the respective trigger signals 41, 42 and strobe signal 38 are received by the respective illumination sources 12, 14, particularly via an AND logic gate, the respective drivers 43 of the illumination sources 12, 14 drive illumination. The light sensor 4 signal is forwarded via connection 35 to the SoC 26, e.g., the API 30 and the ISP 34, and can be provided to the application 32 for further evaluation 44, e.g., along with metadata, etc.

[0215] As further shown in FIG. 2A , the light sensor 4 and the display 2 may be synchronized. The display 2 may be connected to the display interface 28 via at least one display serial interface (DSI) 46, in particular a MIPI display serial interface (MIPI DSI®). The display interface 28 may transfer at least one SW signal 48 to the light sensor 4. The display 2 may be operated in two operating modes: a “video mode” or a “command mode.” In the video mode, a VSYNC signal may be emitted from the display 2. In the command mode, the VSYNC signal may be generated and emitted by the SoC, in particular by software. Thus, the VSYNC signal of the display 2 may be emitted by the display itself and / or by the SoC 26.

[0216] The display device 1 can be configured to pass the display VSYNC 24 to the light sensor 4 as a trigger signal to synchronize the display VSYNC 24 with the exposure edge of the camera frame. Depending on the trigger requirement of the light sensor 4, the display VSYNC 24 can be adapted and specifically adjusted to fully meet the requirement before being passed to the light sensor 4. For example, the frequency of the display VSYNC can be adjusted to half.

[0217] 2B shows the evolution of display VSYNC 24, strobe signal 38, camera VSYNC, and trigger signals 41 and 42 as a function of time, specifically as frames per second (FPS). The camera exposure is shown to occur just before display VSYNC 24, i.e., just before the refresh of the first line where the transparent area at location 16 is located. [Explanation of symbols]

[0218] 1 Display device 2. Translucent display 4. Optical Sensor 5 Irradiation source 6 Black Area 8. Control Unit 10 Evaluation equipment 12 Projector 14 Flood Illumination 16 positions 18 Line 0 20 Line V 24 Display VSYNC 26 SoC 28 Display Interface 30 API 32 Applications 34 ISP 35 connections 36 Camera VSYNC 38 Strobe Signal 40 requests 41 Trigger Signal 42 Trigger Signal 44 provided 46 DSI 48 SW signal

Claims

1. at least one illumination source (5) configured to project at least one illumination beam onto at least one scene; at least one light sensor (4) having at least one light-sensitive area, said light sensor (4) being configured to measure at least one reflected light beam generated by said scene in response to illumination by an illumination beam; at least one light-transmitting display (2) configured to display information, said illumination source (5) and said light sensor (4) being arranged in front of said display (2) in the direction of propagation of an illumination light beam; at least one control unit (8) configured to switch off the display (2) in the area of ​​the irradiation source (5) during irradiation and / or in the area of ​​the light sensor (4) during measurement; A display device (1) comprising:

2. 2. The display device (1) of claim 1, wherein the light-transmissive display (2) is a full-size display having a display material extending over the full size of the display (2).

3. 3. The display device (1) according to claim 1 or 2, wherein the display (2) is configured to show a black area (6) in the area of ​​the illumination source (5) and / or in the area of ​​the light sensor (4) when the control unit (8) turns off the display (2) in the area of ​​the illumination source (5) during illumination and / or in the area of ​​the light sensor (4) during measurement.

4. 4. The display device (1) according to claim 1, wherein the control unit (8) is configured to turn off the display (2) in the area of ​​the irradiation source (5) so that the display (2) in the area of ​​the irradiation source (5) acts as an adjustable notch and / or to turn off the display (2) in the area of ​​the light sensor (4) so ​​that the display (2) in the area of ​​the light sensor (4) acts as an adjustable notch, the adjustable notch being configured to be active during irradiation and / or measurement and inactive otherwise.

5. The display device (1) of any one of claims 1 to 4, wherein the display device (1) is configured to perform facial recognition using the optical sensor (4), the control unit (8) is configured to emit an indication indicating that facial recognition is active while facial recognition is being performed, and the translucent display (2) is configured to display the indication while facial recognition is being performed.

6. Display device (1) according to any one of claims 1 to 5, wherein the light sensor (4) comprises at least one CMOS sensor.

7. Display device (1) according to any one of claims 1 to 6, wherein said illumination source (5) comprises at least one infrared light source.

8. Display device (1) according to any one of the preceding claims, wherein the illumination source (5) comprises at least one laser projector configured to generate at least one illumination pattern.

9. Display device (1) according to any one of claims 1 to 8, wherein said illumination source (5) comprises at least one flood illumination light emitting diode.

10. The display device (1) according to any one of the preceding claims, wherein the display (2), the illumination source (5) and the light sensor (4) are synchronized.

11. 11. The display device (1) according to any one of claims 1 to 10, wherein the display (2) is or comprises at least one organic light emitting diode (OLED) display, and wherein the OLED display is inactive in the area of ​​the illumination source (5) when the control unit (8) turns off the display (2) in the area of ​​the illumination source (5), and / or the OLED display is inactive in the area of ​​the light sensor (4) when the control unit (8) turns off the display (2) in the area of ​​the light sensor (4).

12. The illumination source (5) is configured to project at least one illumination pattern onto the at least one scene, the illumination pattern comprising a plurality of illumination features, the optical sensor is configured to determine at least one first image comprising a plurality of reflection features produced by the scene in response to illumination by the illumination features, the display device (1) further comprises at least one evaluation device (10), the evaluation device (10) is configured to evaluate the first image, the evaluation of the first image comprising identifying reflection features of the first image and sorting the identified reflection features with respect to brightness, each of the reflection features comprising at least one beam profile, the evaluation device (10) being configured to determine, by analysis of the beam profile, at least one vertical coordinate z DPR and the evaluation device (10) is configured to determine the vertical coordinate z DPR and the evaluation device (10) is configured to unambiguously match a reflection feature with a corresponding illumination feature using a vertical coordinate z DPR The display device (1) according to any one of claims 1 to 11, configured to generate a depth map of said real features by using

13. The evaluation device (10) uses triangulation and / or depth from defocus and / or structured light techniques to determine at least one second vertical coordinate z for each of the reflective features. triang 13. A display device (1) according to claim 12, configured to determine:

14. The evaluation device (10) determines the second vertical coordinate z triang and the vertical coordinate z DPR and the bond vertical coordinate is determined by the second vertical coordinate z triang and the vertical coordinate z DPR 14. The display device (1) according to claim 13, wherein said combined vertical coordinate is used to generate a depth map.

15. 15. The display device (1) of any one of claims 12 to 14, wherein the evaluation device (10) is configured to determine beam profile information for each of the reflective features by using a depth from photon ratio technique.

16. The display device (1) according to any one of claims 12 to 15, wherein the evaluation device (10) is configured to determine at least one material property m of an object by evaluating the beam profile of at least one of the reflection features.

17. The display device (1) according to any one of claims 1 to 16, wherein the display device (1) is a mobile device selected from the group consisting of a television device, a mobile phone, a smartphone, a game console, a tablet computer, a personal computer, a laptop, a tablet, a virtual reality device, or another type of portable computer.

18. A method for measuring through a light-transmitting display (2), wherein at least one display device (1) according to any one of claims 1 to 17 is used, and the method comprises the following steps: a) illuminating at least one scene by using at least one illumination light beam generated by at least one illumination source (5), said illumination source (5) being arranged in front of the display (2) in the direction of propagation of the illumination beam; b) measuring at least one reflected light beam generated by the scene in response to illumination by the illumination beam by using at least one light sensor (4), the light sensor (4) having at least one light-sensitive area, the light sensor (4) being positioned in front of the display (2) in the direction of propagation of the illumination beam; c) controlling said display (2) by using at least one control unit (8), wherein said display (2) is switched off in the area of ​​the illumination source (5) during illumination and / or in the area of ​​the light sensor (4) during measurement; 1. A method for measuring through a translucent display (2), comprising:

19. 20. Use of a display device (1) according to any one of claims 1 to 18 for a display device, the use being selected from the group consisting of: position determination in traffic technology; entertainment applications; security applications; surveillance applications; safety applications; human machine interface applications; tracking applications; photography applications; imaging or camera applications; mapping applications for generating at least one map of a space; homing or tracking beacon detectors for vehicles; outdoor applications; mobile applications; communication applications; machine vision applications; robotics applications; quality control applications; manufacturing applications; automotive applications.