OLED rear authentication

Abstract

The present invention provides - at least one pattern illumination source (114) configured to emit at least one infrared light pattern having a plurality of infrared light spots, the number of said infrared light spots being equal to or less than 4000 spots; at least one flood illumination source (116) configured to emit infrared flood light; and at least one image generating unit (118) configured to generate at least one pattern image while the pattern illumination source (114) emits an infrared light pattern, and configured to generate at least one flood image while the flood illumination source (116) emits infrared flood light; The present invention relates to an optoelectronic device (112) comprising: The invention further relates to a method of using the optoelectronic device (112), a method for authenticating a user of the device (110), a computer program, a computer-readable storage medium and a non-transitory computer-readable medium. The present invention provides an approach that allows for relocation of the camera position and improves the operation of active authentication techniques using a device (110) behind a display (120).

Description

[Technical Field]

[0001] The present invention relates to an optoelectronic device, a method for using an optoelectronic device, a device for authenticating a user, and a method for authenticating a user. The present invention further relates to a computer program, a computer-readable storage medium, and a non-transitory computer-readable medium. The device, method, and use according to the present invention are particularly employed in various fields, for example, in everyday life, security technology, games, transportation technology, production technology, art, photography, such as digital photography or videography for document or technical purposes, safety technology, information technology, agriculture, crop protection, maintenance, cosmetics, medical technology, or science. However, other applications are also possible. [Background technology]

[0002] Authentication systems available for mobile devices, such as smartphones and tablets, include a camera. Such mobile devices typically have a front-facing display, such as an organic light-emitting diode (OLED) area. In order to incorporate a camera, a cutout in the display is required at the camera's location. Such a cutout reduces the available display area, a so-called notch, and therefore reduces the display area available to the user. Because the cutout is typically an unpleasant feature for users, cameras are typically restricted to the outermost positions possible to avoid dark areas, such as the center of the display.

[0003] As such, currently available mobile devices have the disadvantages of reducing the display area due to the camera and not being able to adapt the camera position to improve processing efficiency. There is a demand for changing the camera position while improving the user experience.

[0004] Furthermore, to further improve the security of biometric authentication methods such as facial recognition, passive image processing techniques used for this purpose are increasingly being complemented with active techniques. This allows for improved detection of counterfeit authentication attempts. Active methods can use laser-based projection technology to enrich the scene with additional information. Such techniques generally require that the optical paths of the camera (Rx) and laser projector (Tx) are unaffected. For example, established active methods such as structured light, active stereo, and time-of-flight typically use light passing through maximally transparent protective glass to reconstruct the three-dimensional structure of the illuminated scene from the captured images. For example, a CMOS camera with sensitivity in the near-infrared spectral range and a laser projector emitting light in the near-infrared spectral range can be used to ensure the invisibility of the method to the user. As outlined above, when using a smartphone, for example, to perform secure biometric unlocking of the smartphone, such a setup requires an additional cutout in the display to be used with a selfie camera that may operate in the visible spectral range.

[0005] Laser projectors can be configured to operate behind transflective OLED displays, which reduces the number and size of cutouts in the display. Transflective OLED displays typically have an OLED-pixel structure, defined by an optically non-transparent cathode. Transparent conductive tracks and drive electronics design allow for (semi-)transparent regions between OLED pixels. While passive operation, such as in selfie cameras, ensures adequate transparency, light from a laser projector suffers from diffraction losses. Therefore, active systems may achieve effective transmittances of only 3% to 20%. These diffraction losses occur not only in the light emission (Tx) phase but also in the light detection (Rx) phase. Furthermore, diffraction losses can be visible in recorded images as artifacts, such as reduced contrast. Therefore, known methods and techniques suffer from the following drawbacks: The light available for light detection is significantly reduced due to the double pass through the display. Diffraction artifacts can further degrade the remaining image quality. The available laser power required to compensate for such losses is limited by requirements for eye safety (maximum dose after emission from the display) and display stability (maximum thermal dose). Summary of the Invention [Problem to be solved by the invention]

[0006] It is therefore an object of the present invention to provide a device and a method that addresses the above-mentioned technical problems of known devices and methods, in particular to provide a device and a method that not only improves the operation of active techniques of authentication using a device behind a display, but also allows for relocation of the camera position. [Means for solving the problem]

[0007] This problem is solved by the invention with the features of the independent claims. Advantageous developments of the invention, which can be realized individually or in combination, are set out in the dependent claims and / or in the following description and detailed embodiments.

[0008] In a first aspect of the present invention, an optoelectronic device is disclosed, comprising the following components: at least one pattern illumination source configured to emit at least one infrared light pattern having a plurality of infrared light spots, wherein the number of the infrared light spots is 4000 spots or less; at least one flood illumination source configured to emit infrared flood light; and and at least one image generating unit configured to generate at least one pattern image while the pattern illumination source emits an infrared light pattern and to generate at least one flood image while the flood illumination source emits infrared flood light.

[0009] As used herein, the term "optoelectronic device" is a broad term and is to be given its ordinary and customary meaning to those skilled in the art, and is not intended to be limited to any special or customized meaning. Specifically, the term may refer to, but is not limited to, devices or systems that operate with light and electric current.

[0010] The term "light" as used herein is a broad term and should be given its ordinary and customary meaning to those skilled in the art, without being limited to any special or customized meaning. This term can specifically refer to electromagnetic radiation in one or more of the following spectral ranges: infrared, visible, and ultraviolet. Here, the term "ultraviolet spectral range" generally refers to electromagnetic radiation having wavelengths between 1 nm and 380 nm, preferably between 100 nm and 380 nm. Furthermore, in accordance with the ISO-21348 standard, part of the version effective as of the date of this specification, the term "visible spectral range" generally refers to the spectral range between 380 nm and 760 nm. The term "infrared spectral range" (IR) generally refers to electromagnetic radiation between 760 nm and 1000 μm. The range of 760 nm to 1.5 μm is commonly designated the "near-infrared spectral range" (NIR), the range of 1.5 μm to 15 μm is designated the "mid-infrared spectral range" (MidIR), and the range of 15 μm to 1000 μm is designated the "far-infrared spectral range" (FIR). Preferably, the light used for typical purposes of the present invention is light in the infrared (IR) spectral range, more preferably light in the near-infrared (NIR) and / or mid-infrared spectral range (MidIR), in particular light having a wavelength of 1 μm-5 μm, preferably 1 μm-3 μm.

[0011] The term "illuminating," as used herein, is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and should not be limited to any particular or customized meaning. This term may specifically, but is not limited to, refer to the process of exposing at least one element to light. The term "illumination source," as used herein, is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and should not be limited to any particular or customized meaning. This term may specifically, but is not limited to, refer to any device configured to generate or provide light within the meaning of the definition above.

[0012] The term "pattern illumination source" as used herein is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and should not be limited to any special or customized meaning. This term may refer to, but is not limited to, any device configured to generate or provide at least one light pattern, particularly at least one infrared light pattern. The term "light pattern" as used herein is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and should not be limited to any special or customized meaning. This term may refer to, but is not limited to, at least one arbitrary pattern having a plurality of light spots. The light spots may be at least partially spatially extended. At least one spot or any spots may have any shape. In some cases, a circular shape for at least one spot or any spots may be preferred. The spots may be arranged taking into account the structure of a display formed by a device further comprising an optoelectronic device. Typically, this may take into account the arrangement of the OLED-pixel structure of the display. The term "infrared light pattern" as used herein is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and should not be limited to any special or customized meaning. The term may refer to, but is not limited to, a light pattern having spots in the infrared spectral range. The infrared light pattern may also be a near-infrared light pattern.

[0013] The infrared light may be coherent. The infrared light pattern may be a coherent infrared light pattern.

[0014] The patterned illumination source may be configured to emit light of a single wavelength, for example in the near infrared region, hi other embodiments, the patterned illumination source may be adapted to emit light of multiple wavelengths, for example to enable additional measurements in other wavelength channels.

[0015] The infrared light pattern can have at least one regular and / or regular and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or even a pattern with a convex slope. The terms "triangular," "rectangular," and "hexagonal" used herein are broad terms and should be given their usual and customary meanings to those skilled in the art and should not be limited to any special or customized meaning. The terms may refer to, but are not limited to, a two-dimensional distribution of spots, particularly the unit cell of the pattern. The unit cell of a triangular pattern is a triangle. A triangular pattern may have multiple groups of spots, each group having at least three spots forming a triangle. In a rectangular pattern, the spots in the rows and columns are rectangular relative to each other, such as a square unit cell, a rectangular unit cell, or a central rectangular unit cell. A hexagonal pattern has multiple groups of spots, each group having at least four spots forming a hexagonal unit cell, such as three hexagonal cells forming a hexagonal prism. In a hexagonal pattern, a 120° angle is formed between adjacent spots in the rows and columns. Different packing densities and ratios are possible for the pattern. For example, a hexagonal pattern may have a 2 / 5 packing density. For example, the infrared light pattern may be a hexagonal pattern, preferably a hexagonal infrared light pattern, preferably a 2 / 5 hexagonal infrared light pattern. Using a periodic 2 / 5 hexagonal pattern allows for the distinction between artifacts and usable signals.

[0016] The infrared light pattern may include at least one dot pattern. The infrared light pattern has a low dot density. The number of infrared light spots is 4,000 or less. The infrared light pattern may have 3,000 or less spots, preferably 2,000 or less spots. The number of spots may be less than 2,000 and / or greater than 0, preferably greater than 5, more preferably greater than 10, and most preferably greater than 100. The infrared light pattern may have a low dot density, especially compared to other structured light technologies, which typically have a dot density of 10k-30k in a 55x38° field of view. Using such a low dot density can compensate for the diffraction losses mentioned above. Reducing the number of spots projected onto the object and / or user can increase the contrast of the pattern image. Increasing the number of dots reduces the emitted irradiance per dot. Reducing the number of dots increases the emitted irradiance of the dots, which may result in increased contrast of the pattern image of the projection of the infrared light pattern. The infrared light pattern may have a periodic dot pattern with a reduced number of dots, each having a high brightness. Such a light pattern can ensure improved authentication using an illumination source and an image generating unit behind the display. Furthermore, the small number of spots ensures that eye safety and stability requirements are met. The allowable dose can be divided among the spots of the light pattern.

[0017] At least one of the infrared light spots can be associated with a beam divergence of 0.2°-0.5°, preferably 0.1°-0.3°. As used herein, the term "beam divergence" is a broad term and is given its ordinary and customary meaning to those skilled in the art, and is not limited to any special or customized meaning. The term may specifically, but is not limited to, refer to at least one measure of increase in at least one diameter and / or at least one diameter equivalent, such as a radius, with distance from the optical aperture through which the beam exits. The measurement may be an angle or an angle equivalent. In the context of the present invention, beam divergence is typically defined as 1 / e 2 It may be determined by.

[0018] The pattern illumination source may include at least one pattern projector configured to generate an infrared light pattern. The pattern illumination source, e.g., a pattern projector, may have at least one emitter, particularly multiple emitters. As used herein, the term "emitter" is a broad term and should be given its ordinary and customary meaning to those skilled in the art and should not be limited to a specific or customized meaning. The term may refer to, but is not limited to, at least one device configured to provide at least one light beam. The light beam may generate an infrared light pattern. The emitter may include at least one element selected from the group consisting of at least one laser light source, such as at least one semiconductor laser, at least one double heterostructure laser, at least one external cavity laser, at least one separate confinement heterostructure laser, at least one quantum cascade laser, at least one distributed Bragg reflector laser, at least one polariton laser, at least one hybrid silicon laser, at least one extended cavity diode laser, at least one quantum dot laser, at least one volume Bragg grating laser, at least one indium arsenide laser, at least one gallium arsenide laser, at least one transistor laser, at least one diode-pumped laser, at least one distributed feedback laser, at least one quantum well laser, at least one interband cascade laser, at least one semiconductor ring laser, at least one vertical cavity surface-emitting laser (VCSEL), or at least one non-laser light source, such as at least one LED or at least one bulb. For example, the pattern projector may have at least one VCSEL, preferably multiple VCSELs. The VCSELs may be arranged in at least one array, for example comprising a matrix of VCSELs, the VCSELs being arranged on the same substrate or on different substrates.As used herein, the term "vertical cavity surface-emitting laser" is a broad term and is given its ordinary and customary meaning to those skilled in the art, without any special or customized meaning. The term may refer to, but is not limited to, a semiconductor laser diode configured for laser beam emission perpendicular to its top surface. Examples of VCSELs are described, for example, at en.wikipedia.org / wiki / Verticalcavity_surface-emitting_laser. VCSELs are generally known to those skilled in the art, for example, in WO 2017 / 222618 A, each of which is configured to generate at least one light beam. The VCSEL or VCSELs may be configured to generate a desired number of spots, up to 4,000 spots, preferably up to 3,000 spots, and more preferably up to 2,000 spots. The generated spots may be associated with an infrared light pattern. The VCSEL may be configured to emit a light beam in the 800-1,000 nm wavelength range. For example, the VCSEL may be configured to emit a light beam at 808 nm, 850 nm, 940 nm, and / or 980 nm. Preferably, the VCSEL emits at 940 nm because terrestrial solar radiation has a local minimum in emitted irradiance at this wavelength, as described, for example, in CIE 085-1989 "Solar spectral Irradiance."

[0019] The pattern illumination source may include at least one optical element configured to increase, e.g., replicate, the number of spots, e.g., spots generated by a pattern projector. The pattern illumination source, particularly the at least one optical element, may include at least one diffractive optical element (DOE) and / or at least one metasurface element. The DOE and / or metasurface element may be configured to generate multiple light beams from a single incident light beam. For example, a VCSEL projecting up to 2000 spots and an optical element consisting of multiple metasurface elements may be used to double the number of spots. Further arrangements are possible, particularly those with different numbers of projected VCSELs and / or at least one different optical element configured to increase the number of spots. Other multiplication factors are also possible. For example, a VCSEL or multiple VCSELs may be used to replicate the generated laser spots using at least one DOE.

[0020] The pattern illumination source may include at least one transfer device. As used herein, the term "transfer device," also referred to as "transfer system," is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and should not be limited to any special or customized meaning. This term can refer, without limitation, to one or more optical elements adapted to modify a light beam, particularly a light beam used to generate at least a portion of the infrared light pattern, such as by modifying one or more of the beam parameters of the light beam, the width of the light beam, or the direction of the light beam. The transfer device may have at least one imaging optical device. Specifically, the transfer device may have one or more of the following: These include at least one lens, for example, at least one lens selected from the group consisting of at least one adjustable-focus lens, at least one aspherical lens, at least one spherical lens, and at least one Fresnel lens, at least one diffractive optical element, at least one concave mirror, at least one beam deflecting element, preferably at least one mirror, at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitting mirror, at least one multi-lens system, at least one holographic optical element, and at least one meta-optical element. Specifically, the transfer device may have at least one refractive optical lens stack. Therefore, the transfer device may have a multi-lens system with refractive properties.

[0021] The light beam generated by the pattern illumination source may propagate parallel to the optical axis. The pattern illumination source may include at least one reflecting element, preferably at least one prism, for deflecting the light beam onto the optical axis. As an example, the light beam, such as a laser light beam, and the optical axis may have an angle of less than 10°, preferably less than 5°, or even less than 2°. However, other embodiments are also possible. Furthermore, the light beam may be on or off the optical axis. As an example, the light beam may be parallel to the optical axis, or may be coincident with the optical axis, with a distance of less than 10 mm from the optical axis, preferably less than 5 mm from the optical axis, or even less than 1 mm from the optical axis.

[0022] As used herein, the term "flood illumination source" is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and should not be limited to any special or customized meaning. This term may refer, but is not limited to, to at least one device configured to provide substantially continuous spatial illumination. As used herein, the term "flood light" is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and should not be limited to any special or customized meaning. This term may refer, but is not limited to, to substantially continuous spatial illumination, particularly diffuse and / or uniform illumination. Flood light has wavelengths in the infrared range, particularly the near-infrared range. The flood illumination source may include at least one LED or at least one VCSEL, preferably multiple VCSELs. As used herein, the term "substantially continuous spatial illumination" is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and should not be limited to any special or customized meaning. This term specifically refers, but is not limited to, to uniform spatial illumination, although non-uniform areas are possible. The area illuminated from a flood lighting source, for example, an area covering a user, a portion of a user, and / or a user's face, may be continuous. The power may be spread across the entire field of illumination. In contrast, the illumination provided by a light pattern may consist of at least two continuous areas, particularly multiple continuous areas, and / or the power may be concentrated in a small area (compared to the entire field of illumination) of the field of illumination. Infrared flood lighting is suitable for illuminating a continuous area, particularly one continuous area. Infrared pattern lighting is suitable for illuminating at least two continuous areas.

[0023] The flood illumination source may illuminate a user, a portion of the user, and / or a measurement area, such as the user's face, with a substantially constant illumination intensity. As used herein, the term "constant" is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and is not limited to any special or customized meaning. The term may refer to, but is not limited to, a temporal aspect during exposure time. The flood light may vary over time and / or may be substantially constant over time. As used herein, the term "substantially constant" is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and is not limited to any special or customized meaning. The term may specifically refer to, but is not limited to, perfectly constant illumination and embodiments in which the deviation from constant illumination is ±10% or less, preferably ±5% or less, and more preferably ±2% or less.

[0024] The term "image generation unit" as used herein is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and should not be limited to any special or customized meaning. The term may refer, without limitation, to at least one unit of an optoelectronic device configured to generate at least one image. The image may be generated via a hardware and / or software interface that can be considered an image generation unit. The term "image generation" as used herein is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and should not be limited to any special or customized meaning. The term may specifically refer, without limitation, to capturing and / or generating and / or determining and / or recording at least one image using an image generation unit. Image generation may include capturing and / or recording an image. Image generation may include capturing multiple images, such as a single image and / or a series of images. When generating images via a hardware and / or software interface, the capturing and / or generating and / or determining and / or recording of the image may be triggered and / or initiated by the hardware and / or software interface. For example, image generation may involve continuously recording a series of images, such as a video or a movie. Image generation may be initiated by a user action or may be initiated automatically, for example, upon automatic detection of the presence of at least one object or user within the field of view and / or within a predetermined sector of the field of view of the image generation unit.

[0025] The image generation unit may include at least one optical sensor, particularly at least one pixelated optical sensor. The image generation unit may include at least one CMOS sensor or at least one CCD chip. For example, the image generation unit may include at least one CMOS sensor sensitive in the infrared spectral range. The term "image" as used herein is a broad term and should be given its ordinary and customary meaning to those skilled in the art, without being limited to any special or customized meaning. This term may specifically refer to data recorded using an optical sensor, such as, but not limited to, multiple electronic readouts from a CMOS or CCD chip. The image may include raw image data or may be a preprocessed image. For example, preprocessing may include applying at least one filter to the raw image data and / or applying at least one background correction and / or at least one background subtraction.

[0026] For example, the image generation unit may include one or more of at least one monochrome camera including, for example, monochrome pixels, at least one color (e.g., RGB) camera including, for example, color pixels, and at least one IR camera. The camera may be a CMOS camera. The camera may include at least one monochrome camera chip, for example, a CMOS chip. The camera may include at least one color camera chip, for example, an RGB CMOS chip. The camera may include at least one IR camera chip, for example, an IR CMOS chip. For example, the camera may include monochrome pixels, for example, black and white pixels, and color pixels. The color and monochrome pixels are combined within the camera. The camera may generally include a one-dimensional or two-dimensional array of image sensors, such as pixels.

[0027] As outlined above, the image generating unit may be at least one camera. For example, the camera may be an internal camera and / or an external camera of a device having an optoelectronic device. As described above, the internal camera and / or the external camera of the device may be accessed through a hardware and / or software interface included in the optoelectronic device used as the image generating unit. If the device is a smartphone or has a smartphone, the image generating unit may be a front camera, such as a selfie camera, and / or a back camera of the smartphone.

[0028] The image generation unit may have a field of view between 10° x 10° and 75° x 75°, preferably 55° x 65°. For example, the field of view may be between 20° x 20° and 65° x 65°, more preferably between 30° x 30° and 60° x 60°, and most preferably between 55° x 65°. The image generation unit may have a resolution of less than 2MP, preferably between 0.3MP and 1.5MP.

[0029] The image generating unit may include one or more optical elements, such as one or more lenses. As an example, the optical sensor may be a fixed-focus camera with at least one lens that is fixedly adjusted relative to the camera. Alternatively, the camera may have one or more variable lenses that are automatically or manually adjusted. The camera may include at least one optical filter, such as at least one bandpass filter. The bandpass filter may be tuned to the spectrum of the light emitter. However, other cameras are also possible.

[0030] The term "pattern image" as used herein is a broad term and is given its ordinary and customary meaning to those skilled in the art, without being limited to any special or customized meaning. This term may specifically, but not exclusively, refer to an image, e.g., of an object and / or a user, generated by an image generation unit while illuminating with an infrared light pattern. The pattern image may include an image showing a user, particularly at least a portion of the user's face, particularly an image on a respective region of interest constituted by the image, while the user is illuminated with an infrared light pattern. The pattern image may be generated by imaging and / or recording light reflected by an object and / or user illuminated by the infrared light pattern. The pattern image showing the user may include at least a portion of the illuminated infrared light pattern on at least a portion of the user. For example, illumination by a pattern illumination source and imaging using an optical sensor may be synchronized, e.g., using at least one control unit of the optoelectronic device.

[0031] The term "flood image," as used herein, is a broad term and should be given its ordinary and customary meaning to those skilled in the art and should not be limited to any special or customized meaning. The term may specifically, but not exclusively, refer to an image generated by an image generation unit while an illumination source emits, for example, infrared flood light onto an object and / or a user. A flood image may include an image showing a user, particularly the user's face, while the user is illuminated by the flood light. A flood image may be generated by imaging and / or recording light reflected by an object and / or user illuminated by the flood light. A flood image showing a user may have at least a portion of the flood light on at least a portion of the user. For example, illumination by a flood illumination source and imaging using a light sensor may be synchronized, for example, using at least one control unit of an optoelectronic device.

[0032] The image generation unit may be configured to capture and / or record the pattern image and the flood image simultaneously or at different times. The image generation unit may be configured to capture and / or record the pattern image and the flood image in at least partially overlapping measurement areas or areas corresponding to the measurement areas.

[0033] The optoelectronic device may be included in a device. In particular, the optoelectronic device is part of the device. The device may include at least one display, and the infrared light pattern passes through the display as emitted from the pattern illumination source and / or the infrared flood light passes through the display as emitted from the flood illumination source. The display is at least partially transparent in at least one continuous area covering the pattern illumination source and / or the flood illumination source and / or the image generating unit.

[0034] The term "display" as used herein is a broad term and is given its ordinary and accustomed meaning to those skilled in the art, without being limited to any special or customized meaning. The term may refer to, but is not limited to, any form of device configured to display an item of information. The item of information may be any information, such as at least one image, at least one diagram, at least one histogram, at least one graphic, text, numbers, at least one symbol, an operation menu, etc. A display may be at least one screen or may have at least one screen. A display may have any form, for example, a rectangular form. A display may be a front display of a device.

[0035] The display may be or may have at least one organic light-emitting diode (OLED) display. As used herein, the term "organic light-emitting diode" is a broad term and is to be given its ordinary and customary meaning to those skilled in the art, and is not limited to any special or customized meaning. The term may refer, but is not limited to, a light-emitting diode (LED), which is a film of organic compounds in which a light-emitting electroluminescent layer is configured to emit light in response to an electric current. An OLED display may be configured to emit visible light.

[0036] The display, in particular the display area, may be made of glass and / or covered with glass, in particular the display may have at least one glass cover.

[0037] The display may be at least partially transparent. As used herein, the term "at least partially transparent" is a broad term and should be given its ordinary and customary meaning to those skilled in the art, without being limited to any special or customized meaning. Specifically, the term may refer, without limitation, to the property of a display that at least partially transmits light in a certain wavelength range, for example, in the infrared spectral region, particularly in the near-infrared spectral region. For example, the display may be semi-transparent in the near-infrared region. For example, the display may have a transparency of 20%-50% in the near-infrared region. The display may have a different transparency for other wavelength regions. The present invention may propose an optoelectronic device including an image generating unit and two illumination sources that can be positioned behind the display of the device. The transparent region(s) of the display may enable operation of the optoelectronic device behind the display. The display may be an at least partially transparent display, as described above. The display may have a reduced pixel density and / or a reduced pixel size and / or may include at least one transparent conducting path. The transparent area(s) of the display can have a pixel density of 360-440 PPI (pixels per inch). Other areas of the display, such as non-transparent areas, can have a pixel density greater than 400 PPI, for example, 460-500 PPI.

[0038] The display may have a display area. As used herein, the term "display area" is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and should not be limited to any special or customized meaning. This term may refer to, but is not limited to, the active area of ​​the display, particularly the activatable area. The display may have additional areas, such as recesses or cutouts. The display may be at least partially transparent in at least one continuous area, preferably at least two continuous areas. At least one of the continuous areas may at least partially cover the image generating unit and / or the pattern illumination source and / or the flood illumination source. The pattern illumination source, the flood illumination source, and the image generating unit may be positioned in front of the display in the direction of propagation of the infrared light pattern.

[0039] The display may have a first region associated with a first pixel density (pixels per inch (PPI)) value and a second region associated with a second pixel density value. Illustratively, the display may include a first region having a PPI value less than 350 and / or a second region having a PPI value greater than or equal to 400. The first pixel density value may be lower than the second pixel density value. The first pixel density value may be 450 PPI or less, preferably 300-440 PPI, more preferably 350-450 PPI. The second pixel density value may be 400-500 PPI, preferably 450-500 PPI. The first pixel density value may be associated with at least one contiguous region being at least partially transparent. The first pixel density value may be less than 350, and the second pixel density value is 400 or greater.

[0040] As mentioned above, known displays that meet these requirements have limited user appeal, making it desirable to reduce the transparent areas. Furthermore, covering an image-generating unit with such a display can result in reduced contrast because the transparent areas typically have a light transmittance of 15%–50%. Contrast can be defined as the difference between the signal and the backlight. In this case, contrast can be considered to be reduced by transmittance. However, contrast can also be defined as the ratio of signal to background. In this case, contrast is not reduced solely by transmittance but can also be reduced by diffraction during spot pattern projection. This is because higher-order light contributes additional intensity in addition to the zero-order intensity. Furthermore, the structure of display wiring can cause additional diffraction, further reducing the emitted irradiance. Furthermore, a spot pattern projected through a display can produce higher-order spots, reducing the intensity of the main spot corresponding to the pattern before it passes through the display area. These two effects reduce the emitted irradiance to approximately 3–5% of the initial emitted irradiance, resulting in the presence of undesirable additional spots within the pattern. In particular, this effect can occur when exiting the device (first pass through the display) and when entering the device (second pass through the display). The present invention allows for sufficient emitted illuminance and therefore contrast in the pattern image. Contrast can be increased by reducing the number of spots projected onto the user. This increases the emitted illuminance of the spots and increases the contrast of the projected image of the spot pattern. It is also possible to reduce the transparent area of ​​the display. The use of such optoelectronic devices allows, for example, the relocation of the camera from its outermost position on the display. This relocation allows for further optimization of the position of the image generating unit, thereby optimizing wiring. This can reduce the amount of wiring used or improve battery operation.

[0041] The relative distance between the flood lighting source and the pattern lighting source may be less than 3.0 mm. The relative distance between the flood lighting source and the pattern lighting source may be less than 3.0 mm. The distance between the flood lighting source and the pattern lighting source may define the distance between the most distant points of the two light sources. The relative distance between the flood lighting source and the pattern lighting source may be less than 2.5 mm, preferably less than 2.0 mm. The lower limit of the relative distance may be 50 μm, preferably 60 μm, more preferably 70 μm, even more preferably 80 μm, and most preferably 100 μm.

[0042] The pattern illumination light source and the flood illumination light source can be integrated into one module. For example, the pattern illumination light source and the flood illumination light source can be arranged on the same substrate with a minimum relative distance, which is defined by the physical extension of the flood illumination light source and the pattern illumination light source. By arranging the pattern illumination light source and the flood illumination light source with a relative distance of less than 3.0 mm, the space occupied by both illumination light sources can be reduced. In particular, the illumination light sources can be integrated into a single module. Such a space reduction can reduce the transparent area(s) in the display required for the operation of the illumination light source(s) behind the display.

[0043] In a further aspect of the present invention, a method of using an optoelectronic device according to the present invention to authenticate a user of a device including the optoelectronic device is disclosed.

[0044] In a further aspect of the present invention, a device for authenticating a user of a device to perform at least one operation on the device that requires authentication is disclosed.

[0045] The device has the following components: at least one flood illumination source configured to emit infrared flood light; at least one pattern illumination source configured to emit at least one infrared light pattern having a plurality of infrared light spots, wherein the number of the infrared light spots is 4000 spots or less; at least one image generating unit configured to generate at least one pattern image while the pattern illumination source emits an infrared light pattern, and configured to generate at least one flood image while the flood illumination source emits infrared flood light; at least one display illuminated by a pattern illumination source through which an infrared light pattern passes and / or illuminated by a flood illumination source through which infrared flood light passes, the display of the device being at least partially transparent in at least one continuous area overlying the pattern illumination source, the flood illumination source, and / or the image generating unit; and At least one authentication unit configured to perform at least one authentication process of a user using the flood image and the pattern image.

[0046] In particular, the device comprises at least one optoelectronic device according to the invention, and therefore for details, options and definitions reference can be made to the devices and optoelectronic devices as described above or in more detail below.

[0047] The device may be selected from the group consisting of a television device, a games console, a personal computer, a mobile device, in particular a mobile phone, and / or a smartphone, and / or a tablet computer, and / or a laptop, and / or a tablet, and / or a virtual reality device, and / or a wearable such as a smart watch, or another type of portable computer.

[0048] The term "authenticate" as used herein is a broad term and is given its ordinary and customary meaning to those skilled in the art, and is not limited to any special or customized meaning. The term may specifically refer to, but is not limited to, verifying a user's identity. Specifically, authentication may include distinguishing a user from other people or objects, particularly distinguishing between authorized and unauthorized access. Authentication may include confirming the identity of each user and / or assigning an identity to a user. Authentication may include generating and / or providing identification information to other devices or units, such as at least one authorization unit for authorization to provide access to a device. Identity information may be proven by authentication. For example, identity information may be and / or may have at least one identity token. If authentication is successful, the facial image recorded by the image generation unit is verified to be the user's facial image and / or the user's identity is verified. Authentication may be performed using at least one authentication process. The authentication process may have multiple steps, such as at least one face detection on the flood image and at least one identification step in which an identity is assigned to the detected face, and / or at least one identification step in which at least one identity check and / or verification of the user's identity is performed.

[0049] The term "authentication unit" as used herein is a broad term and is to be given its ordinary and customary meaning to those skilled in the art, without being limited to any special or customized meaning. This term may refer, but is not limited to, to at least one unit configured to perform at least one authentication process for a user. The authentication unit may be or have at least one processor. A processor may be any logic circuit configured to perform basic computer or system operations and / or, in general, a device configured to perform calculations or logical operations. In particular, a processor may be configured to process basic instructions that drive a computer or system. As an example, the processor may have at least one arithmetic logic unit (ALU), at least one floating-point unit (FPU), such as a mathematical coprocessor or numeric coprocessor, multiple registers, specifically registers configured to supply operands to the ALU and store operation results, and memory, such as an L1 cache memory and an L2 cache memory. In particular, the processor may be a multi-core processor. In particular, the processor may be or have a central processing unit (CPU). Additionally or alternatively, the processor may be or include a microprocessor, and thus, in particular, the elements of the processor may be included in one single integrated circuit (IC) chip. Additionally or alternatively, the processor may be or include one or more chips, such as one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs) and / or one or more tensor processing units (TPUs) and / or dedicated chips optimized for machine learning. The processor may be specifically configured, such as by software programming, to perform one or more evaluation operations.At least one or any component of the computer program configured to perform the authentication process may be executed by the processing device. Alternatively or additionally, the authentication unit may be or have a connection interface. The connection interface may be configured to transfer data from the device to a remote device or vice versa. At least one or any component of the computer program configured to perform the authentication process may be executed by the remote device.

[0050] For example, the authentication unit may perform at least one face detection using flood images. The face detection may be performed locally on the device. However, face identification, i.e., assigning an identity to a detected face, may be performed remotely, e.g., in the cloud, especially if identification and not just verification is required. User templates may be stored on a remote device, e.g., in the cloud, and do not need to be stored locally. This may be advantageous in terms of storage space and security.

[0051] The authentication unit can be configured to identify the user based on the flood image. In particular, the authentication unit can then transfer data to a remote device. Alternatively or additionally, the authentication unit may perform the user identification based on the flood image, in particular by executing an appropriate computer program having the respective functionality. The term "identify" as used herein is a broad term and should be given its ordinary and customary meaning to those skilled in the art and should not be limited to a special or customized meaning. The term may specifically refer to, but is not limited to, assigning an identity to a detected face and / or confirming at least one identity and / or verifying the identity of a user.

[0052] The authentication process may include multiple steps. For example, the authentication process may include performing at least one face detection. The face detection step may include analyzing the flood image. Further, for example, the authentication process may include identifying. Identifying may include assigning an identity to the detected face and / or performing at least one identity check and / or verifying the identity of the user. Identifying may include performing face verification to determine whether the captured face is the user's face. Identifying the user may include, for example, matching the flood image with a template that outlines parts of the user, particularly parts of the user's face. Identifying the user may include determining whether the captured face is the user's face, particularly whether the captured face corresponds to at least one image of the user's face stored, for example, in at least one memory of the device.

[0053] Analysis of the flood image may include one or more of the following: filtering, selecting at least one region of interest, forming a difference image between the flood image and at least one offset, inverting the flood image, background correction, decomposition into color channels, decomposition into hue, saturation, and brightness channels, frequency decomposition, singular value decomposition, applying a Canny edge detector, applying a Laplacian of Gaussian filter, applying a difference Gaussian filter, applying a Sobel operator, applying a Laplace operator, applying a Scherr operator, applying a Prewitt operator, applying a Roberts operator, applying a Kirsch operator, applying a high-pass filter, applying a low-pass filter, applying a Fourier transform, applying a Radon transform, applying a Hough transform, applying a wavelet transform, thresholding, and creating a binary image. The region of interest may be determined manually by a user or automatically, such as by recognizing a user in the image. In particular, analysis of the flood image may involve the use of at least one image recognition technique, particularly facial recognition. The image recognition technique may include at least one process for identifying a user in the image. The image recognition comprises using at least one technique selected from the following: color-based image recognition using features such as template matching, segmentation and / or blob analysis using features such as size and shape, machine learning and / or deep learning using at least one convolutional neural network.

[0054] The analysis of the flood image may include determining a plurality of facial features. The analyzing may include comparing, particularly matching, the determined facial features with template features. The template features may be features extracted from at least one template. The template may be or include at least one image generated during a registration process, such as when initializing the device. The template may be an image of an authenticated user. The template features and / or facial features may be composed of vectors. Matching the features includes determining the distance between the vectors. Identifying the user may include comparing the vector distance with at least one predefined limit value. If the distance is at least within a tolerance and is equal to or less than the predefined limit value, the user is successfully identified. Otherwise, the user is declined and / or rejected.

[0055] For example, image recognition may involve using a trained model including at least one model, particularly at least one face recognition model. Analysis of flood images may be performed using a face recognition system such as FaceNet, as described, for example, by Florian Schroff, Dmitry Kalenichenko, and James Philbin, "FaceNet: A Unified Embedding for Face Recognition and Clustering," arXiv:1503.03832. The trained model may include at least one convolutional neural network. For example, a convolutional neural network may be designed as described in M.D. Zeiler and R. Fergus, "Visualizing and Understanding Convolutional Networks," CoRR, abs / 1311.2901, 2013, or C. Szegedy et al., "Going Deeper with Convolutions," CoRR, abs / 1409.4842, 2014. For more information about convolutional neural networks for face recognition systems, see Florian Schroff, Dmitry Kalenichenko, and James Philbin, "FaceNet: A Unified Embedding for Face Recognition and Clustering," arXiv:1503.03832. Labeled image data from an image database can be used as training data.Specifically, the datasets may be used from one or more of the YouTube® Faces Database or the Google® Facial Expression Comparison dataset, as described in G.B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” Technical Repory 07-49, University of Massachusetts, Amhersy, October 2007, and L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrained videos with matched background similarity,” in IEEE Conf. on CVPR, 2011. Training of the convolutional neural network may be performed as described in Florian Schroff, Dmitry Kalenichenko, James Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering,” arXiv:1503.03832.

[0056] The authentication unit may be further configured to determine material data based on the pattern image. Therefore, in particular, the authentication unit may transfer the data to a remote device. Alternatively or additionally, the authentication unit may perform material determination based on the pattern image, in particular by executing a suitable computer program having the respective functionality. In particular, by considering the material as a parameter for verifying the authentication process, the authentication process may be robust against being circumvented by using a recorded image of the user.

[0057] The authentication unit is configured to extract material data from the pattern image by beam profile analysis of the light spots. Regarding beam profile analysis, reference is made to WO 2018 / 091649 A1, WO 2018 / 091638 A1, and WO 2018 / 091640 A1, the entire contents of which are incorporated by reference. Beam profile analysis enables reliable classification of a scene based on a small number of light spots. Each of the light spots in the pattern image can have a beam profile. As used herein, the term "beam profile" may generally refer to at least one intensity distribution of the light spots on the optical sensor as a function of pixels. The beam profile may be selected from the group consisting of a trapezoidal beam profile, a triangular beam profile, a conical beam profile, and a linear combination of a Gaussian beam profile.

[0058] The authentication unit may be configured to outsource at least one step of the authentication process, such as user identification, and / or at least one step of the verification of the authentication process, such as reviewing material data, to a remote device, specifically a server and / or a cloud server. The device and the remote device may be part of a computer network, specifically the Internet. This may allow the device to be used as a field device used by a user to generate data required in the authentication process and / or its verification. The device may transmit the generated data and / or data related to intermediate steps of the authentication process and / or its verification to the remote device. In such a scenario, the authentication unit may be and / or have a connection interface configured to transmit information to the remote device. Data generated by the remote device for use in the authentication process and / or its verification may be further transmitted to the device. This data may be received by a connection interface configured by the device. The connection interface may be specifically configured for transmitting or exchanging information. In particular, the connection interface may provide a data transfer connection. As an example, the connection interface may be or have at least one port including one or more of a network or Internet port, a USB port, and a disk drive.

[0059] It is emphasized that data from the device may be sent to a particular remote device depending on at least one condition, such as the date, the day, the load of the particular remote device, etc. The particular remote device is not selected by the field device. Rather, the further device may select which particular remote device to send the data to. The authentication process and / or generation of the verification data may involve the use of multiple different entities of the remote device. At least one entity may generate intermediate data and send the intermediate data to at least one further entity.

[0060] The authentication unit is configured to use a facial recognition authentication process that operates on the flood image, the pattern image and / or the extracted material data. The authentication unit may be configured to extract the material data from the pattern image.

[0061] In one embodiment, extracting material data from a pattern image may include generating a material type and / or data derived from the material type. Preferably, extracting the material data may be based on the pattern image. The material data may be extracted by using at least one model. Extracting the material data may include providing the pattern image to the model and / or receiving the material data from the model. Providing the image to the model may include receiving the pattern image at an input layer of the model or through a loss function of the model, followed by the receiving of the pattern image. The model may be a data-driven model. The data-driven model may have an encoder-decoder structure such as a convolutional neural network and / or an autoencoder. Other examples for generating representations may include FFT, wavelet, deep learning such as CNN, energy models, normalized flows, GANs, visual transformers, or transformers used in natural language processing, autoregressive image modeling, normalized flows, deep autoencoders, and deep energy-based models. Supervised or unsupervised schemes may be applied to generate representations, and embeddings such as cosine or Euclidean metrics in ML languages ​​are also possible. The data-driven model can be parametrized according to a training dataset including at least one image and material data, preferably at least one pattern image and material data. In another embodiment, extracting the material data can include providing an image to the model and / or receiving material data from the model. In another embodiment, the data-driven model can be trained according to a training dataset including at least one image and material data. In another embodiment, the data-driven model can be parametrized according to a training dataset including at least one image and material data. The data-driven model can receive an image and be parametrized according to the training dataset to provide material data based on the received image.The data-driven model may be trained according to a training dataset to receive an image and provide material data as an output based on the received image. The training dataset may include at least one image and material data, preferably material data associated with the at least one image. The image may include a representation of the image. The representation may be a low-dimensional representation of the image. The representation may include at least a portion of data or information related to the image. The representation of the image may include a feature vector. In an embodiment, determining a representation, particularly a low-dimensional representation, may be based on principal component analysis (PCA) mapping or radiative basis function (RBF) mapping. Determining a representation may also be referred to as generating a representation. Generating a representation based on PCA mapping may include clustering based on features of the pattern image and / or the subimage. Additionally or alternatively, generating a representation may be based on a neural network structure suitable for reducing dimensionality. The neural network structure suitable for reducing dimensionality may be composed of an encoder and / or a decoder. In one example, the neural network structure may be an autoencoder. In one example, the neural network structure may include a convolutional neural network (CNN). The CNN may include at least one convolutional layer and / or at least one pooling layer. The CNN can reduce the dimension of the sub-image and / or the image, for example, by applying convolution based on the convolutional layer and / or by pooling. Applying convolution may be suitable for selecting features related to material information of the pattern image.

[0062] In one embodiment, the model is adapted to determine an output based on an input. In particular, the model may be adapted to determine material data based on an image as an input. The model may be a deterministic model, a data-driven model, or a hybrid model. The deterministic model preferably reflects physical phenomena in a mathematical form, including, for example, a first-principles model. The deterministic model may have a set of equations that describe the interaction between the material and the patterned electromagnetic emissions, thereby resulting in a state measurement, a vital sign measurement, or the like. The data-driven model may be a classification model. The hybrid model may be a classification model that includes at least one machine learning architecture with deterministic or statistical adaptation and model parameters. The statistical or deterministic adaptation can be introduced to improve the quality of the results, as it provides a systematic relationship between empirical theory and theory. In one embodiment, the data-driven model may be a classification model. The classification model has at least one machine learning architecture and model parameters. For example, the machine learning architecture may be or include one or more of linear regression, logistic regression, random forest, piecewise linear, nonlinear classifier, support vector machine, naive Bayes classification, nearest neighbor classification, neural network, convolutional neural network, generative adversarial network, support vector machine, or gradient boosting algorithm, etc. In the case of a neural network, the model may be, but is not limited to, a multiscale neural network or a recurrent neural network (RNN), such as a gated recurrent unit (GRU) recurrent neural network or a long short-term memory (LSTM) recurrent neural network. A data-driven model may be parameterized according to a training dataset. A data-driven model may be trained based on a training dataset. Training a model may include parametrizing the model. The term training may also be written as learning.The term may refer, without limitation, to the process of building a classification model, and in particular the process of determining and / or updating the parameters of the classification model. Updating the parameters of a classification model may also be referred to as retraining. Retraining may be included when training is referred to herein. In an embodiment, the training dataset may include at least one of images and material information.

[0063] In embodiments, extracting material data from an image using a data-driven model may include providing the image to the data-driven model. Additionally or alternatively, extracting material data from an image using the data-driven model may include generating an embedding associated with the image based on the data-driven model. An embedding may refer to a low-dimensional representation associated with the image, such as a feature vector. The feature vector may be suitable for suppressing background while maintaining a material signature indicative of the material data. In this context, background may refer to information independent of the material signature and / or the material data. Furthermore, background may refer to information related to biometric features, such as facial features. The material data may be determined using a data-driven model based on an embedding associated with the image. Additionally or alternatively, extracting material data from an image by providing the image to a data-driven model may include converting the image into material data, in particular, into material feature vectors indicative of the material data. Thus, the material data may further include material feature vectors, and / or the material feature vectors may be used to determine the material data.

[0064] In an embodiment, the authentication process may be verified based on the extracted material data.

[0065] In one embodiment, verifying based on the extracted material data may include determining whether the extracted material data matches desired material data. Determining whether the extracted material data matches desired material data may be referred to as verifying. Allowing or denying a user and / or object from performing at least one operation on a device requiring authentication based on the material data may include verifying the authentication or authentication process. Verification may be performed based on the material data and / or the image. Determining whether the extracted material data corresponds to the desired material data may include determining a similarity between the extracted material data and the desired material data. Determining the similarity between the extracted material data and the desired material data may include comparing the extracted material data with the desired material data. The desired material data may refer to predetermined material data. As an example, the desired material data may be skin. It may be determined whether the material data corresponds to the desired material data. In one example, the material data is a non-skin material or silicone. Determining whether the material data corresponds to the desired material data may include comparing the material data with the desired material data. A comparison of the material data with the desired material data can result in allowing and / or denying the user and / or object from performing at least one operation requiring authentication. In one example, skin as the desired material data is compared with non-skin material or silicon as material data, and the result can be a denial because silicon or non-skin material can be different from skin.

[0066] In one embodiment, the authentication process or verification thereof may include generating at least one feature vector from the material data and matching the material feature vector with an associated reference template vector for the material.

[0067] The authentication unit may be configured to authenticate the user if it can identify the user and / or if the material data matches the desired material data. The device may include at least one authentication unit configured to allow the user to perform at least one operation on the device, such as unlocking the device, if the authentication of the user is successful, or to deny the user from performing at least one operation on the device if the authentication is unsuccessful. This allows the user to know the result of the authentication.

[0068] As previously mentioned, the relative distance between the flood illumination source and the pattern illumination source may be less than 3.0 mm.

[0069] In a further aspect, the present invention discloses a method for authenticating a user of a device to perform at least one operation on the device that requires authentication.

[0070] The device includes a display, and the method includes the following steps: illuminating the user with at least one infrared light pattern from at least one pattern illumination source of the device, wherein the number of infrared light spots is 4000 spots or less; illuminating the user with infrared flood light from at least one flood illumination source of the device; generating, by an image generating unit of the device, at least one pattern image showing the user, in particular at least a portion of the user's face, while the user is illuminated with an infrared light pattern; and generating, by an image generating unit of the device, at least one image showing the user while the user is illuminated with infrared flood light, wherein a display of the device is at least partially transparent in at least one continuous area covering the pattern illumination source, the flood illumination source and / or the image generating unit; identifying the user based on the flood image using at least one authentication unit of the device; extracting material data from the at least one pattern image using an authentication unit; and Enabling the user to perform at least one operation on the device that requires authentication based on the material data and the identification.

[0071] The method steps may be performed in a given order or in a different order. Furthermore, one or more additional method steps not listed may be present. Furthermore, one, more than one, or all of the method steps may be performed repeatedly. For details, options, and definitions, reference may be made to the optoelectronic devices and devices described above. Thus, in particular, the method may comprise using a device according to the present invention, such as according to one or more embodiments given above or given in more detail below.

[0072] Identifying a user may involve matching flood images to templates.

[0073] The method may include using a facial recognition authentication process operating on the flood image, the pattern image, and / or the extracted material data. The pattern image and / or the image showing the user while illuminated with infrared flood light may show at least a portion of the user's face.

[0074] All described method steps can be performed using a device. Thus, a single processing device may be configured to exclusively execute at least one computer program used in at least one embodiment of the method according to the present invention, in particular at least one line of computer program code configured to execute at least one algorithm. Here, the computer program executed on the single processing device may include all instructions that cause the computer to execute the method. Alternatively, or additionally, at least one method step may be executed using at least one remote device, in particular selected from at least one server or cloud server, in cases where the device and the remote device may be part of a computer network. In this case, the computer program may include at least one remote component executed by the at least one remote processing device to perform at least one method step. The remote component may have functionality for performing user identification and / or material data extraction. Furthermore, the computer program may include at least one interface configured to transfer data to and / or receive data from at least one remote component of the computer program.

[0075] The method may include permitting or denying a user from performing at least one operation on the device. In one embodiment, permitting or denying a user from performing at least one operation requiring authentication on the device based on material data may include permitting the user to perform at least one operation requiring authentication on the device if the material data matches desired material data and / or authentication may be successful. The desired material data may be predetermined material data. Authentication may be successful if the user can be identified and / or if the material data matches desired material data. Furthermore, permitting or denying an object from performing at least one operation on the device requiring authentication based on material data may include refusing to perform at least one operation on the device requiring authentication if the material data does not match the desired material data and / or authentication may fail. Authentication may fail if the pattern image does not match the image template and / or if the material data does not match the desired material data.

[0076] The at least one operation on the device requiring authentication may be access to the device, e.g., unlocking the device, and / or access to an application, preferably associated with the device, and / or access to a portion of an application, preferably associated with the device. In embodiments, allowing a user to access a resource may include allowing the user to perform at least one operation on the device and / or system. The resource may be a device, a system, a device function, a system function, and / or an entity. Additionally and / or alternatively, allowing a user to access a resource may include allowing the user to access an entity. The entity may be a physical entity and / or a virtual entity. The virtual entity may be, for example, a database. The physical entity may be an area with restricted access. The area with restricted access may be one of a security area, a room, an apartment, a vehicle, some of the aforementioned examples, etc. The device and / or system may be locked. The device and / or system may be unlocked only by an authorized user.

[0077] The term "user" as used herein is a broad term and is given its ordinary and accustomed meaning to those skilled in the art and is not limited to any special or customized meaning. The term may refer particularly, but not limited to, to a person intended to use and / or who uses a device.

[0078] The method may be computer-implemented. As used herein, the term "computer-implemented" is a broad term and should be given its ordinary and customary meaning to those skilled in the art, and should not be limited to any special or customized meaning. The term may specifically refer to, but is not limited to, a method including at least one computer and / or at least one computer network. The computer and / or computer network may include at least one processor configured to perform at least one of the method steps of the method according to the present invention. Specifically, each method step is performed by the computer and / or computer network. The method may also be performed fully automatically, specifically without user interaction.

[0079] Further disclosed and proposed herein is a computer program comprising computer-executable instructions for performing the method according to the present invention in one or more of the embodiments encompassed herein when the program is executed on a computer or computer network. Specifically, the computer program is stored on a computer-readable data carrier and / or a computer-readable storage medium. The computer program may be executed on at least one processor configured by an optoelectronic device and / or apparatus. The computer program may generate input data by accessing and / or controlling at least one unit of the optoelectronic device and / or apparatus, such as a pattern illumination source and / or a flood illumination source and / or an image generation unit. The computer program may generate result data based on the input data, particularly by using an authentication unit.

[0080] In this specification, the terms "computer-readable data carrier" and "computer-readable storage medium" may refer in particular to non-transitory data storage means such as a hardware storage medium having computer-executable instructions stored thereon. The stored computer-executable instructions may be associated with a computer program. The computer-readable data carrier or storage medium may in particular be or have a storage medium such as a random access memory (RAM) and / or a read-only memory (ROM).

[0081] Thus, in particular, one, more than one, or even all of the method steps a. to f. set out above can be carried out using a computer or a computer network, preferably using a computer program.

[0082] Further disclosed and proposed herein is a computer program product having program code means for carrying out the method according to the present invention in one or more of the embodiments encompassed herein when the program is run on a computer or a computer network. In particular, the program code means may be stored on a computer-readable data carrier and / or on a computer-readable storage medium.

[0083] Further disclosed and proposed herein is a data carrier having a data structure stored thereon, which data carrier is capable of performing a method according to one or more embodiments disclosed herein after being loaded into a computer or computer network, for example into a working memory or main memory of the computer or computer network.

[0084] Further disclosed and proposed herein is a computer program product having program code means stored on a machine-readable carrier for performing a method according to one or more embodiments disclosed herein when the program is run on a computer or a computer network. In this specification, a computer program product refers to a program as a tradeable product. The product can generally exist in any form, such as a paper format, on a computer-readable data carrier and / or on a computer-readable storage medium. In particular, the computer program product may be distributed via a data network.

[0085] Further disclosed and suggested herein is a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to one or more of the embodiments disclosed herein.

[0086] Finally, disclosed and suggested herein is a modulated data signal containing instructions readable by a computer system or computer network for carrying out a method according to one or more of the embodiments disclosed herein.

[0087] With reference to computer-implemented aspects of the present invention, one or more, or even all, of the method steps of the methods according to one or more of the embodiments disclosed herein can be performed using a computer or a computer network. Thus, generally, any of the method steps involving providing and / or manipulating data are performed using a computer or a computer network. Generally, these method steps can include any of the method steps, except for those that typically require manual intervention, such as certain aspects of providing a sample and / or performing the actual measurement.

[0088] Specifically, the following features are further disclosed herein: a computer or computer network comprising at least one processor, the processor being adapted to execute a method according to one of the embodiments described herein, - a computer-loadable data structure adapted to perform a method according to one of the embodiments described herein while the data structure is being executed on a computer; a computer program, which, when said program is run on a computer, is adapted to carry out a method according to one of the embodiments described herein, a computer program comprising program means for carrying out the method according to one of the embodiments described herein while the computer program is running on a computer or on a computer network, a computer program comprising program means according to the preceding embodiment, the program means being stored on a computer-readable storage medium; - a storage medium having a data structure stored thereon, the storage medium being adapted to perform a method according to one of the embodiments described herein after the data structure has been loaded into a main and / or working storage device of a computer or a computer network; and - a computer program product having program code means, which may be stored or is stored on a storage medium for performing a method according to one of the embodiments described herein when the program code means is executed on a computer or on a computer network.

[0089] As used herein, the terms "have," "comprise," "include," or any grammatical variations thereof, are used in a non-exclusive manner. Thus, these terms can refer both to a situation in which, besides the features introduced by these terms, no further features are present in the entity described in this context, and to a situation in which one or more further features are present. As an example, the expressions "A has B," "A comprises B," and "A includes B" can refer both to a situation in which, besides B, no other elements are present in A (i.e., a situation in which A consists only of B), and to a situation in which, besides B, one or more further elements are present in entity A, such as element C, elements C and D, or even other elements.

[0090] Additionally, it should be noted that the terms "at least one," "one or more," or similar phrases indicating that a feature or element may be present one or more times are typically used only once when introducing each feature or element. In most cases, the phrases "at least one" or "one or more" will not be repeated when referring to each feature or element, regardless of the fact that each feature or element may be present one or more times.

[0091] Furthermore, in this specification, terms such as "preferably," "more preferably," "particularly," "more particularly," "particularly," and "more particularly" are used in combination with any feature without limiting substitutability. Features introduced by these terms are therefore optional features and are not intended to limit the scope of the claims in any way. The present invention can be practiced using alternative features, as will be recognized by those skilled in the art. Similarly, features introduced by "in one embodiment of the present invention" or similar expressions are intended to be optional features, without any limitation regarding alternative embodiments of the invention, without any limitation regarding the scope of the invention, and without any re-restriction regarding the possibility of combining the feature so introduced with other optional or non-optional features of the invention.

[0092] Overall, in the context of the present invention, the following embodiments are considered preferred:

[0093] In one embodiment, an optoelectronic device is disclosed, the optoelectronic device including the following components: at least one pattern illumination source configured to emit at least one infrared light pattern including a plurality of infrared light spots, wherein the number of the infrared light spots is 4000 spots or less; at least one flood illumination source configured to emit infrared flood light; and and at least one image generating unit configured to generate at least one pattern image while the pattern illumination source emits an infrared light pattern and to generate at least one flood image while the flood illumination source emits infrared flood light.

[0094] In one embodiment, the optoelectronic device may be configured in a device, which may include at least one display, wherein the infrared light pattern may pass through the display while being emitted from the pattern illumination source, and / or the infrared flood light may pass through the display while being emitted from the flood illumination source, and wherein the display is at least partially transparent in at least one continuous area covering the pattern illumination source, the flood illumination source, and / or the image generating unit.

[0095] In one embodiment, the display comprises: a first region having a pixel per inch (PPI) of less than 350; and A second region with a PPI value of 400 or greater Includes:

[0096] In one embodiment, the infrared light pattern includes no more than 3000 spots, preferably no more than 2000 spots.

[0097] In one embodiment, the spots are circular.

[0098] In one embodiment, the infrared light is coherent, where the infrared light pattern is a coherent infrared light pattern.

[0099] In one embodiment, the infrared light pattern is a hexagonal pattern, preferably a hexagonal infrared light pattern, preferably a 2 / 5 hexagonal infrared light pattern.

[0100] In one embodiment, at least one of the infrared light spots may be associated with a beam divergence of 0.2°-0.5°, preferably 0.1°-0.3°.

[0101] In one embodiment, the infrared light pattern may be a near-infrared light pattern.

[0102] In one embodiment, the image generation unit may have a field of view between 10°x10°-75°x75°, preferably 55°x65°, preferably the field of view is between 20°x20° and 65°x65°, more preferably between 30°x30° and 60°x60°, most preferably 55°x65.

[0103] In one embodiment, the image generation unit has a resolution of less than 2MP, preferably between 0.3MP and 1.5MP.

[0104] In one embodiment, the image generation unit includes at least one CMOS sensor or at least one CCD chip.

[0105] In one embodiment, the pattern illumination source includes at least one pattern projector configured to generate an infrared light pattern.

[0106] In one embodiment, a pattern illumination source, such as a pattern projector, includes at least one vertical cavity surface emitting laser (VCSEL), preferably multiple VCSELs.

[0107] In one embodiment, the pattern illumination source may include at least one optical element configured to increase the number of spots, the optical element including at least one diffractive optical element (DOE) and / or at least one metasurface element.

[0108] In one embodiment, the relative distance between the flood illumination source and the pattern illumination source is less than 3.0 mm.

[0109] In one embodiment, the relative distance between the flood illumination source and the pattern illumination source is less than 2.5 mm, preferably less than 2.0 mm.

[0110] In one embodiment, a method of using an optoelectronic device according to any one of the previous embodiments for authenticating a user of a device comprising the device is disclosed.

[0111] In one embodiment, a device for authenticating a user of a device to perform at least one operation on the device that requires authentication, the device comprising the following components: at least one flood illumination source configured to emit infrared flood light; at least one pattern illumination source configured to emit at least one infrared light pattern having a plurality of infrared light spots, wherein the number of the infrared light spots is 4000 spots or less; at least one image generating unit configured to generate at least one pattern image while the pattern illumination source emits an infrared light pattern, and configured to generate at least one flood image while the flood illumination source emits infrared flood light; at least one display through which an infrared light pattern passes as illuminated by a pattern illumination source and / or through which an infrared floodlight passes as illuminated by a flood illumination source, the display of the device being at least partially transparent in at least one continuous area covering the pattern illumination source, the flood illumination source, and / or the image generating unit; and At least one authentication unit configured to perform at least one authentication process of a user using the flood image and the pattern image.

[0112] In one embodiment, the display is or includes at least one organic light emitting diode (OLED) display.

[0113] In one embodiment, the display includes a display area.

[0114] In one embodiment, the display may be made of glass and / or may be covered with glass.

[0115] In one embodiment, the display of the device is at least partially transparent in at least two contiguous regions.

[0116] In one embodiment, the display may have a first region associated with a first pixel density value and a second region associated with a second pixel density value, the first pixel density value being lower than the second pixel density value, preferably the first pixel density value being 450 PPI or less.

[0117] In one embodiment, a first pixel density value is associated with at least one contiguous region that is at least partially transparent.

[0118] In one embodiment, the first pixel density value is less than 350 and the second pixel density value is 400 or greater.

[0119] In one embodiment, the number of infrared light spots may be less than 2000 and / or greater than 0, but preferably greater than 5, more preferably greater than 10, and most preferably greater than 100.

[0120] In one embodiment, the device is selected from the group consisting of a television device, a games console, a personal computer, a mobile device, in particular a mobile phone, and / or a smartphone, and / or a tablet computer, and / or a laptop, and / or a tablet, and / or a virtual reality device, and / or a wearable device such as a smart watch, or other type of portable computer.

[0121] In one embodiment, the authentication unit is configured to use a facial recognition authentication process that operates on the pattern image, the flood image, and / or the extracted material data.

[0122] In an embodiment, the device may comprise at least one optoelectronic device according to any one of the preceding embodiments referring to an optoelectronic device.

[0123] In one embodiment, the relative distance between the flood illumination source and the pattern illumination source is less than 3.0 mm.

[0124] In one embodiment, a method is disclosed for authenticating a user of a device to perform at least one operation on the device that requires authentication, the device including a display, and the method includes the following steps: illuminating the user with at least one infrared light pattern from at least one pattern illumination source of the device; illuminating the user with infrared flood light from at least one flood illumination source of the device; generating at least one pattern image with an image generation unit of the device representing the user while the user is illuminated with an infrared light pattern; and generating at least one image with the image generation unit of the device representing the user while the user is illuminated with infrared flood light, wherein a display of the device is at least partially transparent in at least one continuous area covering the pattern illumination source, the flood illumination source, and / or the image generation unit; identifying the user based on the flood image using at least one authentication unit of the device; extracting material data from the at least one pattern image using an authentication unit; and Enabling the user to perform at least one operation on the device that requires authentication based on the material data and the identification.

[0125] In one embodiment, the method includes using a facial recognition authentication process that operates on the pattern image, the flood image and / or the extracted material data.

[0126] In one embodiment, the pattern image and / or the image showing the user while the user is illuminated by an infrared floodlight shows at least a portion of the user's face.

[0127] In one embodiment, identifying a user involves matching a flood image to a template.

[0128] In one embodiment, the distance between the flood illumination source and the pattern illumination source is less than 3.0 mm.

[0129] In one embodiment, the method is computer-implemented.

[0130] In one embodiment, a computer program is disclosed that includes instructions that, when executed by a device according to any one of the preceding embodiments referencing the device, perform a method according to any one of the preceding embodiments referencing the device.

[0131] In one embodiment, a computer-readable storage medium is disclosed comprising instructions that, when executed by a device according to any one of the preceding embodiments referencing the device, cause the device to perform a method according to any one of the preceding embodiments referencing the method.

[0132] In one embodiment, a non-transitory computer-readable medium is disclosed that includes instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any one of the preceding embodiments referencing the method. [Brief explanation of the drawings]

[0133] Further optional details and features of the present invention will become apparent from the following description of preferred exemplary embodiments in conjunction with the dependent claims. In this context, certain features may be implemented alone or in combination with other features. The present invention is not limited to the exemplary embodiments. The exemplary embodiments are illustrated diagrammatically in the figures. The same reference numerals in the individual figures refer to identical elements, elements with the same function, or elements that correspond to each other in terms of function. [Figure 1] 1 shows an embodiment of a device according to the invention. [Figure 2] 1 illustrates an embodiment of the method according to the present invention. [Figure 3] 3 shows a further embodiment of the device according to the invention; DETAILED DESCRIPTION OF THE INVENTION

[0134] 1 shows in a highly schematic manner an embodiment of a device 110 of the present invention, for example, the device 110 is selected from the group consisting of a television device, a game console, a personal computer, a mobile device, in particular a mobile phone, and / or a smartphone, and / or a tablet computer, and / or a laptop, and / or a tablet, and / or a virtual reality device, and / or a wearable device such as a smart watch, or other type of portable computer.

[0135] In this embodiment, device 110 comprises an optoelectronic device 112 according to the present invention. Optoelectronic device 112 comprises at least one pattern illumination source 114 configured to emit at least one infrared light pattern having a plurality of infrared light spots, the number of which is 4000 spots or less.

[0136] The pattern illumination source 114 is configured to generate or provide at least one light pattern, particularly at least one infrared light pattern. The light pattern may have multiple light spots. The light spots may be at least partially spatially extended. The infrared light pattern may be a near-infrared light pattern. The infrared light may be coherent. The infrared light pattern may be a coherent infrared light pattern. The pattern illumination source 114 may be configured to emit light of a single wavelength, for example, in the near-infrared range. In other embodiments, the pattern illumination source 114 may be adapted to emit light of multiple wavelengths, for example, to enable additional measurements in other wavelength channels. The infrared light pattern may have at least one regular and / or regular and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or a pattern including an additional convex slope. For example, the infrared light pattern is a hexagonal pattern, preferably a hexagonal infrared light pattern, more preferably a 2 / 5 hexagonal infrared light pattern. The use of a periodic 2 / 5 hexagonal pattern makes it possible to distinguish between artifacts and usable signal.

[0137] The infrared light pattern may include at least one dot pattern. The infrared light pattern has a low dot density. The number of infrared light spots is 4,000 or less. The infrared light pattern may have 3,000 or less, preferably 2,000 or less, spots. The spots may be circular. Additional shapes are also possible. The infrared light pattern may have a low dot density, especially compared to other structured light technologies, which typically have a dot density of 10k-30k in a 55x38° field of view. Using such a low dot density can compensate for the diffraction losses mentioned above. Reducing the number of spots projected onto the object and / or user can increase the contrast of the pattern image. Increasing the number of spots reduces the emitted irradiance per point. Reducing the number of spots increases the emitted irradiance of the spots, which can result in increased contrast of the pattern image of the projection of the infrared light pattern. The infrared light pattern may have a periodic dot pattern with a reduced number of spots, each with a high emitted irradiance. Such a light pattern can ensure improved authentication using the pattern illumination source 114, at least one flood illumination source 116 as described above, at least one image generating unit 118, and behind the display 120. Furthermore, the small number of spots ensures compliance with eye safety and stability requirements. The allowable dose may be divided among the spots of the light pattern.

[0138] At least one of the infrared light spots is associated with a beam divergence of between 0.2° and 0.5°, preferably between 0.1° and 0.3°.

[0139] The pattern illumination source 114 may include at least one pattern projector configured to generate an infrared light pattern. The pattern illumination source 114, e.g., a pattern projector, may include at least one emitter, particularly multiple emitters. The emitter may include at least one element selected from the group consisting of at least one semiconductor laser, at least one double heterostructure laser, at least one external cavity laser, at least one separate confinement heterostructure laser, at least one quantum cascade laser, at least one distributed Bragg reflector laser, at least one polariton laser, at least one hybrid silicon laser, at least one extended cavity diode laser, at least one quantum dot laser, at least one volume Bragg grating laser, at least one indium arsenide laser, at least one gallium arsenide laser, at least one transistor laser, at least one diode-pumped laser, at least one distributed feedback laser, at least one quantum well laser, at least one interband cascade laser, at least one semiconductor ring laser, at least one vertical cavity surface-emitting laser (VCSEL), and at least one non-laser light source such as at least one LED or at least one light bulb. For example, the pattern illumination source 114, e.g., a pattern projector, includes at least one VCSEL, preferably multiple VCSELs. The plurality of VCSELs may be arranged in an array, e.g., a matrix of VCSELs. The VCSELs may be arranged on a common substrate or on different substrates. Examples of VCSELs are described, for example, in en.wikipedia.org / wiki / Verticalcavity_surface-emitting_laser. VCSELs are generally known to those skilled in the art, for example, from WO 2017 / 222618 A. Each of the VCSELs is configured to generate at least one light beam. The VCSEL or plurality of VCSELs may be configured to generate a desired number of spots, e.g., 4,000 spots or less, preferably 3,000 spots or less, and more preferably 2,000 spots or less.The VCSEL can be configured to emit a light beam in the wavelength range of 800-1000 nm. For example, the VCSEL can be configured to emit a light beam at 808 nm, 850 nm, 940 nm, or 980 nm. Preferably, the VCSEL emits at 940 nm because terrestrial solar radiation has a local minimum in emitted irradiance at this wavelength, as described, for example, in CIE 085-1989, "Solar spectral Irradiance."

[0140] The patterned illumination source 114 may have at least one optical element (not shown) configured to increase, e.g., replicate, the number of spots, e.g., spots generated by a pattern projector. The patterned illumination source 114 may have at least one diffractive optical element (DOE) and / or at least one metasurface element. The DOE and / or metasurface element may be configured to generate multiple light beams from a single incident light beam. For example, a VCSEL projecting up to 2000 spots and an optical element with multiple metasurface elements may be used to double the number of spots. Other replications are also possible. For example, a VCSEL or multiple VCSELs may be used to replicate the generated laser spots using at least one DOE.

[0141] The pattern illumination source 114 may include at least one transfer device (not shown). The transfer device may include at least one imaging optical device. Specifically, the transfer device may include one or more of the following: at least one lens, for example, at least one lens selected from the group consisting of at least one adjustable-focus lens, at least one aspherical lens, at least one spherical lens, and at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflecting element, preferably at least one mirror; at least one beam splitting element, preferably at least one beam splitting cube or beam splitting mirror; at least one multi-lens system; at least one holographic optical element; or at least one meta-optical element. Specifically, the transfer device includes at least one refractive optical lens stack. Thus, the transfer device may include a multi-lens system with refractive properties.

[0142] The optoelectronic device 112 includes at least one flood illumination source 116 configured to emit infrared flood light. As can be seen from FIG. 3 , the relative distance between the flood illumination source 116 and the pattern illumination source 114 is less than 3.0 mm. The relative distance between the flood illumination source 116 and the pattern illumination source 114 is less than 2.5 mm, preferably less than 2.0 mm. The pattern illumination source 114 and the flood illumination source 116 may be integrated into a single module. For example, the pattern illumination source 114 and the flood illumination source 116 may be arranged on the same substrate with a minimum relative distance. The minimum relative distance may be defined by the physical extension of the flood illumination source 116 and the pattern illumination source 114. By arranging the pattern illumination source 114 and the flood illumination source 116 with a relative distance of less than 3.0 mm, the space required for the two illumination sources 114, 116 is reduced. In particular, the illumination sources 114, 116 may be integrated into a single module. This reduced space requirement reduces the transparent area within the display required to operate the illumination sources 114, 116 behind the display 120. Reference numerals discussed in the context of Figure 2 apply equally to Figure 3.

[0143] The optoelectronic device 112 comprises at least one image generation unit 118 configured to generate at least one pattern image while the pattern illumination source 114 emits an infrared light pattern and to generate at least one flood image while the flood illumination source 116 emits infrared flood light.

[0144] The image generation unit 118 may be at least one unit of the optoelectronic device 112 configured to generate at least one image. Image generation may include capturing and / or generating and / or determining and / or recording at least one image using the image generation unit 118. Image generation may include capturing and / or recording images. Image generation may include capturing a single image and / or multiple images, such as a series of images. For example, image generation may include continuously recording a series of images, such as a video or movie. Image generation may be initiated by a user action or may be initiated automatically, for example, upon automatically detecting the presence of at least one object or user within the field of view and / or within a predetermined sector of the field of view of the image generation unit.

[0145] The image generation unit 118 may have at least one optical sensor, in particular at least one pixelated optical sensor. The image generation unit 118 may have at least one CMOS sensor or at least one CCD chip. For example, the image generation unit 118 may have at least one CMOS sensor sensitive in the infrared spectral range. The image may be image data recorded using an optical sensor, such as multiple electronic readouts from a CMOS or CCD chip. The image may be raw image data or a preprocessed image. For example, the preprocessing may include applying at least one filter to the raw image data and / or applying at least one background correction and / or at least one background subtraction.

[0146] For example, the image generation unit 118 may include a monochrome camera having, for example, monochrome pixels. For example, the image generation unit may include a color camera having, for example, color pixels. The image generation unit may include a color CMOS camera. For example, the camera may include monochrome pixels and color pixels. The color and monochrome pixels may be combined within the camera. The image generation unit 118 may include at least one color camera (e.g., RGB) and / or at least one monochrome camera, such as a monochrome CMOS. The camera may include at least one monochrome CMOS chip. A camera may generally include a one-dimensional or two-dimensional array of image sensors, such as pixels.

[0147] As outlined above, the image generation unit may be at least one color, e.g., RGB, camera. For example, the color camera may be the selfie camera of a smartphone.

[0148] The image generating unit may have a field of view between 10° x 10° and 75° x 75°, preferably 55° x 65°. The image generating unit may have a resolution of less than 2MP, preferably between 0.3MP and 1.5MP.

[0149] The image generating unit may further comprise one or more optical elements, for example one or more lenses. As an example, the optical sensor may be a fixed-focus camera with at least one lens that is fixedly adjusted relative to the camera. Alternatively, the camera may include one or more variable lenses that are automatically or manually adjusted. However, other cameras are also possible.

[0150] The pattern image may be an image generated by the image generation unit 118 while illuminating, for example, an object and / or a user with an infrared light pattern. The pattern image may include an image showing a user, in particular an image showing at least a portion of the user's face, while the user is illuminated with the infrared light pattern. The pattern image may be generated by imaging and / or recording light reflected by an object and / or a user illuminated by the infrared light pattern. For example, illumination by the pattern illumination source 114 and imaging using the optical sensor may be synchronized, for example, using at least one control unit of the optoelectronic device 112.

[0151] A flood image may be an image generated by image generation unit 118 while an illumination source emits infrared flood light, for example, onto an object and / or a user. A flood image may include an image showing a user, particularly the user's face, while the user is illuminated by the flood light. A flood image may be generated by imaging and / or recording light reflected by an object and / or user illuminated by the flood light. For example, illumination by flood illumination source 116 and imaging using a light sensor may be synchronized, for example, using at least one control unit of optoelectronic device 112.

[0152] The image generation unit 118 may be configured to capture and / or record the pattern image and the flood image simultaneously or at different times.

[0153] Optoelectronic device 112 may include device 110. Device 110 may include at least one display 120 through which an infrared light pattern passes as emitted from pattern illumination source 114 and / or infrared flood light passes as emitted from flood illumination source 116. Alternatively or additionally, display 120 may be at least partially transparent in at least one continuous region covering pattern illumination source 114, flood illumination source 116, and / or image generation unit 118.

[0154] Display 120 may be a device of any shape configured to display an item of information. The item of information may be any information, such as at least one image, at least one diagram, at least one histogram, at least one graphic, text, numbers, at least one symbol, an operation menu, etc. Display 120 may be at least one screen or may have at least one screen. Display 120 may have any shape, for example, a rectangular shape. Display 120 may be a front display of device 110.

[0155] Display 120 may be or may include at least one organic light emitting diode (OLED) display, which may be configured to emit visible light.

[0156] The display 120 may be made of glass and / or may be covered with glass, in particular, the display 120 may have at least one glass cover.

[0157] The display 120 may be at least partially transparent. For example, the display 120 may be semi-transparent in the near-infrared region. For example, the display 120 may have a transparency of 20% to 50% in the near-infrared region. The display 120 may have different transparency for other wavelength regions. The present invention proposes an optoelectronic device 112 having an image generating unit 118 and two illumination light sources 114, 116 that can be arranged behind the display 120 of the device 110. The transparent region(s) of the display 120 can operate the optoelectronic device 112 behind the display 120. The display 120 can be an at least partially transparent display, as described above. The display 120 can have a reduced pixel density and / or a reduced pixel size, and / or can have at least one transparent conductive path. The transparent region(s) of the display 120 can have a pixel density of 360 to 440 PPI (pixels per inch). Other areas of the display 120, such as non-transparent areas, may have a higher pixel density, such as 460-500 PPI.

[0158] The display 120 has a display area. The display area may be an active area, in particular an activatable area, of the display 120. The display 120 may have additional areas, such as recesses or cutouts. The display 120 may be at least partially transparent in at least one continuous area, preferably at least two continuous areas. At least one of the continuous areas may at least partially cover the image generation unit and / or the pattern illumination source 114 and / or the flood illumination source 116. The pattern illumination source 114, the flood illumination source 116, and the image generation unit 118 may be arranged in front of the display 120 in the propagation direction of the infrared light pattern.

[0159] The display 120 may have a first region associated with a first pixel density value and a second region associated with a second pixel density value. The first pixel density value may be lower than the second pixel density value. The first pixel density value may be 450 PPI or less, preferably 300-440 PPI, more preferably 350-450 PPI. The second pixel density value may be 400-500 PPI, preferably 450-500 PPI. The first pixel density value is associated with at least one continuous region being at least partially transparent. The first pixel density value is less than 350, and the second pixel density value is 400 or greater.

[0160] As mentioned above, known displays that meet these requirements lack appeal to users, and reducing the transparent area is desirable. Furthermore, covering an image-generating unit with such a display typically results in low contrast because the light transmittance of the transparent area is typically 15% to 50%. Contrast can be defined as the difference between the signal and the backlight. In this case, contrast can be considered to be reduced by transmittance. However, contrast can also be defined as the ratio of signal to background. In this case, contrast is not reduced solely by transmittance but can also be reduced by diffraction in spot pattern projection, because higher-order diffraction patterns contribute additional intensity in addition to the zero-order diffraction pattern. Due to the structure of the display wiring, additional diffraction can occur, further reducing the emitted illuminance. Furthermore, spot patterns projected through the display can produce higher-order spots, reducing the intensity of the main spot corresponding to the pattern before passing through the display area. Both effects reduce the emitted illuminance to approximately 3% to 5% of the initial emitted illuminance, resulting in the presence of undesirable additional spots within the pattern. The present invention can provide sufficient emitted illuminance and, therefore, the contrast of the pattern image. Contrast can be increased by reducing the number of spots projected to the user. This allows for an increase in the emitted illuminance of the spot, and therefore an increase in the contrast of the projected image of the spot pattern. Furthermore, it is also possible to reduce the transparent area of ​​the display. By using such an optoelectronic device 112, it is possible to relocate, for example, the camera from its outermost position on the display. Such relocation allows for further optimization of the position of the image generating unit 118, and therefore optimization of wiring. This allows for a reduction in the amount of wiring used or for improved battery operation.

[0161] The device 110 is configured to authenticate a user of the device 110 in order to perform at least one operation on the device 110 that requires authentication. The authentication may consist of verifying the identity of the user. Specifically, the authentication may consist of distinguishing the user from other people or objects, in particular distinguishing between authorized and unauthorized access. The authentication may include confirming the identity of each user and / or assigning an identity to the user. The authentication may consist of generating and / or providing identification information to other devices or units, such as at least one authorization unit for authorization to provide access to the device 110. The identification information may be proven by the authentication. For example, the identification information may be and / or may consist of at least one identification token. If the authentication is successful, the facial image recorded by the image generation unit 118 is verified to be the user's facial image and / or the user's identity is verified.

[0162] User authentication can be performed using at least one authentication unit 122. The authentication unit 122 may be configured to perform at least one authentication process for a user. The authentication unit 122 may have at least one processor. A processor may be any logic circuit configured to perform basic operations of a computer or system, and / or generally, a device configured to perform calculations or logical operations. In particular, the processor may be configured to process basic instructions that run a computer or system. As an example, the processor may have at least one arithmetic logic unit (ALU), at least one floating-point unit (FPU), such as an arithmetic coprocessor or a math coprocessor, multiple registers, specifically registers configured to supply operands to the ALU and store calculation results, and memory, such as an L1 cache memory and an L2 cache memory. In particular, the processor may be a multi-core processor. In particular, the processor may be or have a central processing unit (CPU). Additionally or alternatively, the processor may be or include a microprocessor, and thus, in particular, the elements of the processor may be included in one single integrated circuit (IC) chip. Additionally or alternatively, the processor may be or include one or more chips, such as one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs) and / or one or more tensor processing units (TPUs) and / or dedicated machine learning optimization chips. The processor may be specifically configured, such as by software programming, to perform one or more evaluation operations.

[0163] The authentication unit 122 is configured to identify a user based on the flood image. Identifying may include assigning an identity to the detected face and / or checking at least one identity and / or verifying the user's identity. The authentication process may have multiple steps. For example, the authentication process may include performing at least one face detection. The face detection step may include analyzing the flood image. Further, for example, the authentication process may include identifying. Identifying may include assigning an identity to the detected face and / or checking at least one identity and / or verifying the user's identity. Identifying may include analyzing the flood image. Analyzing the flood image may include performing face verification to determine whether the captured face is the user's face. Identifying the user may include, for example, matching the flood image with a template that outlines parts of the user, particularly parts of the user's face. Identifying the user may include determining whether the captured face is the user's face, particularly whether the captured face corresponds to at least one image of the user's face stored, for example, in at least one memory of the device.

[0164] Analysis of the flood image may include one or more of the following: filtering, selecting at least one region of interest, forming a difference image between the flood image and at least one offset, inverting the flood image, background correction, decomposition into color channels, decomposition into hue, saturation, and brightness channels, frequency decomposition, singular value decomposition, applying a Canny edge detector, applying a Laplacian of Gaussian filter, applying a difference Gaussian filter, applying a Sobel operator, applying a Laplace operator, applying a Scherr operator, applying a Prewitt operator, applying a Roberts operator, applying a Kirsch operator, applying a high-pass filter, applying a low-pass filter, applying a Fourier transform, applying a Radon transform, applying a Hough transform, applying a wavelet transform, thresholding, and creating a binary image. The region of interest may be determined manually by a user or automatically, such as by recognizing a user in the image. In particular, analysis of the flood image may include using at least one image recognition technique, particularly facial recognition techniques. The image recognition technique includes at least one process of identifying a user in the image. The image recognition may include using at least one technique selected from the following techniques: color-based image recognition using features such as template matching, image segmentation and / or blob analysis using features such as size and shape, machine learning and / or deep learning using at least one convolutional neural network.

[0165] Analyzing the flood image may include determining a plurality of facial features. Analyzing may include comparing, particularly matching, the determined facial features with template features. The template features may be features extracted from at least one template. The template may be or include at least one image generated during an enrollment process, such as when initializing the device 110. The template may be an image of an authenticated user. The template features and / or facial features may have vectors. Matching the features may include determining a distance between the vectors. Identifying the user may include comparing the vector distance with at least one predefined limit value, and if the distance is at least within a tolerance and less than or equal to the predefined limit value, the user is successfully identified. Otherwise, the user is declined and / or rejected.

[0166] For example, image recognition may involve using a trained model including at least one model, particularly at least one face recognition model. Analysis of flood images may be performed using a face recognition system such as FaceNet, as described, for example, by Florian Schroff, Dmitry Kalenichenko, and James Philbin, "FaceNet: A Unified Embedding for Face Recognition and Clustering," arXiv:1503.03832. The trained model may include at least one convolutional neural network. For example, a convolutional neural network may be designed as described in M.D. Zeiler and R. Fergus, "Visualizing and Understanding Convolutional Networks," CoRR, abs / 1311.2901, 2013, or C. Szegedy et al., "Going Deeper with Convolutions," CoRR, abs / 1409.4842, 2014. For more information about convolutional neural networks for face recognition systems, see Florian Schroff, Dmitry Kalenichenko, and James Philbin, "FaceNet: A Unified Embedding for Face Recognition and Clustering," arXiv:1503.03832. Labeled image data from an image database can be used as training data.Specifically, the datasets may be used from one or more of the YouTube® Faces Database or the Google® Facial Expression Comparison dataset, as described in G.B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” Technical Repory 07-49, University of Massachusetts, Amhersy, October 2007, and L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrained videos with matched background similarity,” in IEEE Conf. on CVPR, 2011. Training of the convolutional neural network may be performed as described in Florian Schroff, Dmitry Kalenichenko, James Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering,” arXiv:1503.03832.

[0167] The authentication unit 122 is configured to use a facial recognition authentication process that operates on the pattern image, the flood image, and / or the extracted material data. The authentication unit 122 may be configured to extract the material data from the pattern image.

[0168] The authentication unit is configured to extract material data from the pattern image by beam profile analysis of the light spots. Regarding the beam profile analysis, reference is made to WO 2018 / 091649 A1, WO 2018 / 091638 A1, and WO 2018 / 091640 A1, the entire contents of which are incorporated by reference. The beam profile analysis makes it possible to provide a reliable classification of a scene based on a small number of light spots. Each of the light spots of the pattern image can have a beam profile. The extraction of material data based on the pattern image may be performed using at least one model.

[0169] The authentication process may include verifying based on the extracted material data. Verifying based on the extracted material data may include determining whether the extracted material data matches the desired material data. Determining whether the extracted material data matches the desired material data may be referred to as verifying. Allowing or denying a user and / or object to perform at least one operation on the device 110 that requires authentication based on the material data may include verifying the authentication or authentication process. The verifying may be based on the material data and / or the image. Determining whether the extracted material data corresponds to the desired material data may include determining the similarity between the extracted material data and the desired material data. Determining the similarity between the extracted material data and the desired material data may include comparing the extracted material data with the desired material data. The desired material data may refer to predetermined material data. As an example, the desired material data may be skin. It may be determined whether the material data corresponds to the desired material data. In one example, the material data is a non-skin material or silicone. Determining whether the material data corresponds to the desired material data includes comparing the material data with the desired material data. A comparison of the material data with the desired material data can result in allowing and / or denying the user and / or object from performing at least one operation requiring authentication. In an example, skin as the desired material data is compared with non-skin material or silicone as material data, and the result can be a denial because silicone or non-skin material can be different from skin. In one embodiment, the authentication process or verification thereof can include generating at least one feature vector from the material data and matching the material feature vector with an associated reference template vector for the material.

[0170] The authentication unit 122 may be configured to authenticate the user if it can identify the user and / or if the material data matches the desired material data. The device 110 may include at least one authentication unit 124 configured to allow the user to perform at least one operation on the device 110, such as unlocking the device 110, if the user is successfully authenticated, or to deny the user from performing at least one operation on the device 110 if the authentication is not successful.

[0171] FIG. 2 illustrates an exemplary embodiment of a method for authenticating a user of device 110 to perform at least one operation on device 110 that requires authentication, the method including the following steps: (Reference numeral 126) illuminating the user with at least one infrared light pattern from at least one pattern illumination source 114 of the device 110, wherein the number of infrared light spots is 4000 spots or less; (Reference numeral 128) illuminating the user with infrared flood light from at least one flood illumination source 116 of the device 110; (Reference numeral 130) generating at least one pattern image showing the user, in particular at least a portion of the user's face, with the image generation unit 118 of the device 110 while the user is illuminated with an infrared light pattern, and generating at least one image showing the user with the image generation unit 118 of the device 110 while the user is illuminated with infrared flood light, wherein the display 120 of the device 110 is at least partially transparent in at least one continuous area covering the pattern illumination source 114, the flood illumination source 116, and / or the image generation unit 118; (Reference numeral 132) identifying a user based on the flood image using at least one authentication unit 122 of the device 110; (Reference numeral 134) extracting material data from at least one pattern image using the authentication unit 122; and (Reference numeral 136) Enabling the user to perform at least one operation on the device 110 that requires authentication based on the material data and the identification.

[0172] The method steps may be performed in a predetermined order or in a different order. Furthermore, one or more additional method steps not listed may be present. Furthermore, one, more than one, or all of the method steps may be performed repeatedly. The pattern image and / or the image showing the user while illuminated with infrared flood light may show at least a portion of the user's face.

[0173] The method may be computer-implemented. [Explanation of symbols]

[0174] 110 devices 112 Optoelectronic devices 114 Pattern Lighting Source 116 Flood Lighting Source 118 Image Generation Unit 120 Display 122 Certification Unit 124 Certification Units 126. Irradiating an infrared light pattern to a user 128. Irradiating the user with infrared flood light 130. Generating at least one pattern image and at least one flood image 132 Steps to identify users 134 Steps for extracting material data 136 Allowing Steps

Claims

1. An optoelectronic device (112) comprising: at least one pattern illumination source (114) configured to emit at least one infrared light pattern having a plurality of infrared light spots, the number of said infrared light spots being equal to or less than 4000; at least one flood illumination source (116) configured to emit infrared flood light; and at least one image generating unit (118) configured to generate at least one pattern image while said pattern illumination source (114) emits an infrared light pattern, and configured to generate at least one flood image while said flood illumination source (116) emits infrared flood light; An optoelectronic device (112) comprising:

2. 2. The optoelectronic device of claim 1, wherein the optoelectronic device is included in a device having at least one display, the device having at least one display, the infrared light pattern passing through the display as it is emitted from the pattern illumination source, and / or the infrared flood light passing through the display as it is emitted from the flood illumination source, and the display is at least partially transparent in at least one continuous area covering the pattern illumination source, the flood illumination source, and / or the image generating unit.

3. 3. An optoelectronic device (112) according to claim 1 or 2, wherein the infrared light pattern has 3000 or less spots, preferably 2000 or less spots.

4. The optoelectronic device (112) of claim 1 or 2, wherein the infrared light is coherent light.

5. The optoelectronic device (112) of claim 1 or 2, wherein the infrared light pattern is a hexagonal pattern.

6. An optoelectronic device (112) according to claim 1 or 2, wherein at least one of said infrared light spots is associated with a beam divergence angle of between 0.2° and 0.5°, preferably between 0.1° and 0.3°.

7. The optoelectronic device (112) of claim 1 or 2, wherein the image generating unit (118) has a field of view of 10°×10° to 75°×75°.

8. 3. The optoelectronic device of claim 1, wherein the patterned illumination source comprises at least one vertical cavity surface emitting laser (VCSEL), preferably a plurality of VCSELs, and / or the patterned illumination source comprises at least one optical element configured to increase the number of spots, the optical element comprising at least one diffractive optical element and / or at least one metasurface element.

9. Use of an optoelectronic device (112) according to claim 1 or 2 for authenticating a user of a device (110) including the optoelectronic device (112).

10. A device (110) for authenticating a user of the device (110) to perform at least one operation on the device (110) that requires authentication, the device comprising: at least one flood illumination source (116) configured to emit infrared flood light; at least one pattern illumination source (114) configured to emit at least one infrared light pattern comprising a plurality of infrared light spots, the number of said infrared light spots being equal to or less than 4000 spots; at least one image generating unit (118) configured to generate at least one pattern image while the pattern illumination source (114) emits an infrared light pattern, and configured to generate at least one flood image while the flood illumination source (116) emits infrared flood light; at least one display (120), through which the infrared light pattern passes as it is emitted from the pattern illumination source (114) and / or through which the infrared flood light passes as it is emitted from the flood illumination source (116), the display (120) of the device (110) being at least partially transparent in at least one continuous area covering the pattern illumination source (114), the flood illumination source (116) and / or the image generating unit (118); and at least one authentication unit configured to perform at least one authentication process of a user using said flood image and said pattern image; A device (110) comprising:

11. 11. The device of claim 2, wherein the display has a first region associated with a first pixel pitch (PPI) value and a second region associated with a second PPI value, the first PPI value being lower than the second PPI value.

12. - said first PPI value is less than 350; - said second PPI value is 400 or greater; The device (110) of claim 10.

13. 1. A method for authenticating a user of a device (110) to perform at least one operation on the device (110) that requires authentication, the device (110) including a display (120), the method comprising the steps of: a. illuminating a user with at least one infrared light pattern from a pattern illumination source (114) of said device (110); b. illuminating the user with infrared flood light from at least one flood illumination source (116) of said device (110); c) generating at least one pattern image representing the user with an image generation unit (118) of the device (110) while the user is illuminated with the infrared light pattern, and generating at least one image representing the user with the image generation unit (118) of the device (110) while the user is illuminated with the infrared flood light, wherein the display (120) of the device (110) is at least partially transparent in at least one continuous area covering the pattern illumination source (114), the flood illumination source (116), and / or the image generation unit (118); d. identifying the user based on the flood image using at least one authentication unit (122) of said device (110); e. Extracting material data from at least one pattern image using the authentication unit (122); and f. enabling a user to perform at least one operation on said device (110) that requires authentication based on said material data and identification. A method comprising:

14. A computer program comprising instructions that, when executed by a device (110) according to claim 10 relating to the device (110), cause the device (110) to perform the method according to claim 13 relating to the method.

15. A computer-readable storage medium comprising instructions that, when executed by a device (110) according to claim 10 relating to the device (110), cause the device (110) to perform the method according to claim 13 relating to the method.

16. 14. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 13.

Images