Oled under-screen authentication

Abstract

The invention relates to an optoelectronic device (112) comprising:-at least one pattern illumination source (114) configured for emitting at least one infrared light pattern comprising a plurality of infrared light spots wherein the number of infrared light spots is less than or equal to 4000 light spots,-at least one floodlight illumination source (116) configured for emitting at least one infrared light pattern comprising a plurality of infrared light spots wherein the number of infrared light spots is less than or equal to 4000 light spots, the at least one floodlight illumination source is configured to emit infrared floodlight; -at least one image generation unit (118) configured for generating at least one pattern image when the pattern illumination source (114) emits an infrared light pattern, and configured for generating at least one floodlight image when the floodlight illumination source (116) emits an infrared floodlight. The invention further relates to the use of an optoelectronic device (112), a method for authenticating a user of an apparatus (110), a computer program, a computer-readable storage medium and a non-transitory computer-readable medium. The invention provides a method of rearranging the position of a camera and improving the operation of an active technique for authentication using a device (110) behind a display (120).

Description

Technical Field

[0001] The present invention relates to an optoelectronic device, uses of the optoelectronic device, a device for authenticating a user, and a method for authenticating a user of the device. The present invention further relates to a computer program, a computer-readable storage medium, and a non-transitory computer-readable medium. The device, method, and uses according to the present invention can be used, for example, in various fields of daily life, security technology, gaming, transportation technology, production technology, photography (such as digital photography or videography for artistic, documentation, or technical purposes), safety technology, information technology, agriculture, crop protection, maintenance, cosmetics, medical technology, or science. However, other applications are also possible. Background Art

[0002] Available authentication systems in mobile devices (such as smartphones and tablets) include cameras. These devices typically have a front display, such as an organic light-emitting diode (OLED) area. To integrate the camera, a cutout is required in the display at the camera's location. This cutout reduces the available display area, known as the notch, and thus reduces the display area available to the user. Typically, because notches are an unpleasant feature for users, cameras are confined to the outermost position possible, for example to avoid dark areas in the middle of the display.

[0003] Therefore, currently available mobile devices have the following disadvantages: the display area is reduced due to the camera, and processing efficiency cannot be improved by adjusting the position of the camera. It is desired to rearrange the position of the camera while increasing the user experience.

[0004] Furthermore, to further enhance the security of biometric authentication methods such as facial recognition, passive image processing methods used for this purpose are supplemented with active methods. This can improve the detection of fraudulent authentication attempts. Active methods can use laser-based projection technology to enrich the scene with additional information. This approach typically requires unaffected optical paths for the camera (Rx) and laser projector (Tx). For example, established active methods such as structured light, active stereo, or time-of-flight use light, typically transmitted through a maximum-transparency cover glass, to reconstruct the three-dimensional structure of the illuminated scene from the captured image. For example, a CMOS camera sensitive in the near-infrared spectral range and a laser projector emitting light in the near-infrared spectral range can be used, for example, to ensure that the method is invisible to the user. As mentioned above, in the case of smartphones, for example, to perform secure biometric unlocking of the smartphone, this setup requires an additional cutout in the display, for example, to accommodate a selfie camera operating in the visible light spectrum.

[0005] Laser projectors can be configured to operate behind a semi-transparent OLED display. This can reduce the number and size of cutouts within the display. Semi-transparent OLED displays typically include an OLED pixel structure, which can be defined by an optically opaque cathode. The use of transparent conductive tracks and the design of the drive electronics can achieve (semi-)transparent areas between the OLED pixels. However, to ensure suitable transparency for passive operation, such as in selfie cameras, the light from the laser projector experiences diffraction losses. Consequently, active approaches can only achieve effective transmittances ranging from 3% to 20%. This diffraction loss occurs in both light emission (Tx) and light detection (Rx). Furthermore, diffraction losses can be visible as artifacts in the recorded image, such as reduced contrast. Consequently, known methods and devices suffer from the following disadvantages: due to the double pass through the display, the light generated at the light detection site is significantly reduced; the remaining image quality can be further degraded by diffraction artifacts; and the available laser power required to compensate for this loss is limited due to eye safety (maximum dose to the display after emission) and display stability (maximum thermal dose) requirements.

[0006] Problems to be solved by the present invention

[0007] Therefore, the object of the present invention is to provide a device and method that face the above technical challenges of known devices and methods. In particular, the object of the present invention is to provide a device and method that allows rearranging the position of the camera and improving the operation of active techniques for authentication using a device behind a display. Summary of the Invention

[0008] This problem is solved by the invention having the features of the independent patent claim. Advantageous developments of the invention which can be realized individually or in combination are presented in the dependent claims and / or in the following description and detailed examples.

[0009] In a first aspect of the present invention, a photovoltaic device is disclosed. The photovoltaic device comprises:

[0010] at least one patterned illumination source configured to emit at least one infrared light pattern comprising a plurality of infrared light spots, wherein the number of infrared light spots is lower than or equal to 4000 spots,

[0011] - at least one flood illumination source configured to emit infrared flood light,

[0012] - at least one image generation unit configured to generate at least one pattern image when the pattern illumination source emits an infrared light pattern, and configured to generate at least one flood image when the flood illumination source emits infrared flood light.

[0013] As used herein, the term "photovoltaic device" is a broad term and is to be given its ordinary and customary meaning to those skilled in the art and is not limited to a special or customized meaning. The term may specifically refer to, but is not limited to, a device or system that operates on light and electric current.

[0014] As used herein, the term "light" is a broad term and is to be given its ordinary and customary meaning to those skilled in the art and is not to be limited to a special or customary meaning. The term may specifically refer to, but is not limited to, electromagnetic radiation in one or more of the infrared spectral range, the visible spectral range, and the ultraviolet spectral range. As used herein, the term "ultraviolet spectral range" generally refers to electromagnetic radiation with a wavelength of 1 nm to 380 nm, preferably 100 nm to 380 nm. Furthermore, in accordance in part with the version of standard ISO-21348 in effect as of the date of this document, the term "visible spectral range" generally refers to the spectral range of 380 nm to 760 nm. The term "infrared spectral range" (IR) generally refers to electromagnetic radiation in the range of 760 nm to 1000 µm, of which the range of 760 nm to 1.5 µm is generally referred to as the "near infrared spectral range" (NIR), the range of 1.5 µm to 15 µm is referred to as the "mid infrared spectral range" (MidIR), and the range of 15 µm to 1000 µm is referred to as the "far infrared spectral range" (FIR). Preferably, the light used for the typical purposes of the present invention is light in the infrared (IR) spectral range, more preferably light in the near infrared (NIR) and / or mid-infrared spectral range (MidIR), especially light with a wavelength of 1 µm to 5 µm, preferably 1 µm to 3 µm.

[0015] As used herein, the term "irradiation" is a broad term and is to be given its ordinary and customary meaning to one of ordinary skill in the art and is not limited to a special or customary meaning. The term may specifically refer to, but is not limited to, the process of exposing at least one element to light. As used herein, the term "irradiation source" is a broad term and is to be given its ordinary and customary meaning to one of ordinary skill in the art and is not limited to a special or customary meaning. The term may specifically refer to, but is not limited to, any device configured to generate or provide light as defined above.

[0016] As used herein, the term "patterned illumination source" is a broad term and is to be given its ordinary and conventional meaning to those skilled in the art and is not limited to a special or custom meaning. The term specifically refers to, but is not limited to, any device configured to generate or provide at least one light pattern, particularly at least one infrared light pattern. As used herein, the term "light pattern" is a broad term and is to be given its ordinary and conventional meaning to those skilled in the art and is not limited to a special or custom meaning. The term specifically refers to, but is not limited to, at least one arbitrary pattern comprising a plurality of light spots. The light spots can be at least partially spatially extended. At least one or any light spot can have any shape. In some cases, a circular shape for at least one or any light spot can be preferred. The light spots can be arranged based on the structure of a display included in the device, which further includes an optoelectronic device. Typically, the arrangement of the OLED pixel structure of the display can be considered. As used herein, the term "infrared light pattern" is a broad term and is to be given its ordinary and conventional meaning to those skilled in the art and is not limited to a special or custom meaning. The term specifically refers to, but is not limited to, a light pattern comprising light spots within the infrared spectral range. The infrared light pattern may be a near infrared light pattern.

[0017] The infrared light may be coherent. The infrared light pattern may be a coherent infrared light pattern.

[0018] The patterned illumination source may be configured to emit light of a single wavelength, such as in the near infrared region. In other embodiments, the patterned illumination source may be adapted to emit light having multiple wavelengths, such as to allow additional measurements in other wavelength channels.

[0019] The infrared light pattern may include at least one regular and / or constant and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or a pattern including further convex tessellations. As used herein, the terms "triangular," "rectangular," and "hexagonal" are broad terms and are to be given their ordinary and customary meanings to those of ordinary skill in the art and are not limited to special or customary meanings. The terms may specifically refer to, but are not limited to, a two-dimensional distribution of light spots, particularly unit cells of the pattern. The unit cells in a triangular pattern are triangles. A triangular pattern may include multiple groups of light spots, each group including at least three light spots forming a triangle. In a rectangular pattern, the light spots in rows and columns are rectangular relative to each other, such as a square unit cell, a rectangular unit cell, or a centered rectangular unit cell. A hexagonal pattern includes multiple groups of light spots, each group including at least four light spots forming a hexagonal unit cell, for example, where three hexagonal cells form a hexagonal prism. In a hexagonal pattern, adjacent light spots in each row and column form an angle of 120°. For patterns, such as hexagonal patterns, different packing densities or fractions are conceivable, such as a 2 / 5 packing density. For example, the infrared light pattern is a hexagonal pattern, preferably a hexagonal infrared light pattern, preferably a 2 / 5 hexagonal infrared light pattern. Using a periodic 2 / 5 hexagonal pattern can allow for distinguishing artifacts from usable signals.

[0020] The infrared light pattern may include at least one dot pattern. The infrared light pattern has a low dot density. The number of infrared light spots is less than or equal to 4,000. The infrared light pattern may include equal to or less than 3,000 spots, preferably equal to or less than 2,000 spots. The number of spots may be less than 2,000 and / or greater than 0, preferably greater than 5, more preferably greater than 10, and most preferably greater than 100. The infrared light pattern may have a low dot density, particularly compared to other structured light technologies that typically have 10,000 to 30,000 dots within a 55 × 38° field of view. Using this low dot density can compensate for the aforementioned diffraction losses. By reducing the number of spots projected onto the object and / or user, contrast in the pattern image can be increased. Increasing the number of spots reduces the irradiance per dot. Reducing the number of spots may result in increased irradiance of the spots, thereby increasing the contrast of the projected pattern image of the infrared light pattern. The infrared light pattern may have a periodic dot pattern with a reduced number of spots, each of which has high brightness. This light pattern enables improved authentication using illumination sources and image generation units behind the display. Furthermore, the small number of light spots ensures compliance with eye safety and stability requirements. The permissible dose can be divided between the light spots in the light pattern.

[0021] At least one of the infrared spots may be associated with a beam divergence of 0.2° to 0.5°, preferably 0.1° to 0.3°. As used herein, the term "beam divergence" is a broad term and will be given its ordinary and conventional meaning to those of ordinary skill in the art and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, at least one measure of at least one diameter and / or at least one diameter equivalent (such as a radius) that increases with distance from the optical aperture from which the beam emerges. The measure may be an angle or an angle equivalent. In the context of the present invention, typically, the beam divergence may be determined at 1 / e 2 .

[0022] The pattern illumination source may include at least one pattern projector configured to generate an infrared light pattern. The pattern illumination source (e.g., a pattern projector) may include at least one emitter, and in particular, a plurality of emitters. As used herein, the term "emitter" is used broadly and is to be given its ordinary and customary meaning to one of ordinary skill in the art and is not limited to a special or customary meaning. The term specifically refers to, but is not limited to, at least one arbitrary device configured to provide at least one light beam. The light beam may generate the infrared light pattern. The emitter may include at least one element selected from the group consisting of at least one laser source and at least one non-laser light source, such as at least one semiconductor laser, at least one double heterostructure laser, at least one external cavity laser, at least one separated confined heterostructure laser, at least one quantum cascade laser, at least one distributed Bragg reflector laser, at least one polariton laser, at least one hybrid silicon laser, at least one extended cavity diode laser, at least one quantum dot laser, at least one volume Bragg grating laser, at least one indium arsenide laser, at least one gallium arsenide laser, at least one transistor laser, at least one diode pump laser, at least one distributed feedback laser, at least one quantum well laser, at least one interband cascade laser, at least one semiconductor ring laser, or at least one vertical cavity surface emitting laser (VCSEL); and at least one non-laser light source, such as at least one LED or at least one light bulb. For example, the pattern projector includes at least one VCSEL, preferably a plurality of VCSELs. The plurality of VCSELs may be arranged in at least one array, such as a VCSEL matrix. The VCSELs may be arranged on the same substrate or on different substrates. As used herein, the term "vertical cavity surface emitting laser" is a broad term and is to be given its ordinary and conventional meaning to a person of ordinary skill in the art and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, a semiconductor laser diode configured to emit a laser beam vertically relative to a top surface. Examples of VCSELs can be found, for example, at en.wikipedia.org / wiki / Verticalcavity_surface-emitting_laser. VCSELs are generally known to those skilled in the art, such as from WO 2017 / 222618 A. Each of the VCSELs is configured to generate at least one light beam. The VCSEL or the plurality of VCSELs can be configured to generate a desired number of light spots equal to or less than 4000 light spots, preferably equal to or less than 3000 light spots, and more preferably equal to or less than 2000 light spots. The plurality of light spots generated can be associated with an infrared light pattern. The VCSEL can be configured to emit a light beam having a wavelength in the range of 800 nm to 1000 nm.For example, the VCSEL can be configured to emit light beams at 808 nm, 850 nm, 940 nm, and / or 980 nm. Preferably, the VCSEL emits light at 940 nm because terrestrial solar radiation has a local minimum in irradiance at this wavelength, as described, for example, in CIE 085-1989 "Solar spectral irradiance."

[0023] The patterned illumination source may include at least one optical element configured to multiply (e.g., duplicate) the number of light spots (e.g., those generated by a pattern projector). The patterned illumination source, and in particular the at least one optical element, may include at least one diffractive optical element (DOE) and / or at least one metasurface element. The DOE and / or metasurface element may be configured to generate multiple light beams from a single incident light beam. For example, a VCSEL projecting up to 2,000 light spots and an optical element comprising multiple metasurface elements may be used to duplicate the number of light spots. Further arrangements, in particular comprising different numbers of projecting VCSELs and / or at least one different optical element configured to increase the number of light spots, may be possible. Other multiplication factors are also possible. For example, one or more VCSELs may be used, and the generated laser light spots may be duplicated using at least one DOE.

[0024] The patterned illumination source may include at least one delivery device. As used herein, the term "delivery device," also denoted "delivery system," is a broad term and is to be given its ordinary and customary meaning to those skilled in the art and is not limited to a special or customary meaning. The term may specifically refer to, but is not limited to, one or more optical elements adapted to modify a light beam, particularly a light beam for generating at least a portion of an infrared light pattern, such as by modifying one or more of its beam parameters, beam width, or beam direction. The delivery device may include at least one imaging optical device. The delivery device may specifically include one or more of the following: at least one lens, such as at least one lens selected from the group consisting of at least one adjustable focus lens, at least one aspheric lens, at least one spherical lens, and at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflecting element, preferably at least one reflective mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitter; at least one multi-lens system; at least one holographic optical element; or at least one meta-optical element. Specifically, the delivery device includes at least one refractive optical lens module. Thus, the delivery device may include a multi-lens system having refractive properties.

[0025] The one or more light beams generated by the patterned illumination source may propagate parallel to the optical axis. The patterned illumination source may include at least one reflective element, preferably at least one prism, for deflecting the light beams onto the optical axis. As an example, one or more light beams (such as laser beams) may be at an angle of less than 10°, preferably less than 5°, or even less than 2° to the optical axis. However, other embodiments are also possible. Furthermore, one or more light beams may be located on or off the optical axis. As an example, one or more light beams may be parallel to the optical axis and at a distance less than 10 mm from the optical axis, preferably less than 5 mm or even less than 1 mm from the optical axis, or may even coincide with the optical axis.

[0026] As used herein, the term "flood illumination source" is a broad term and is to be given its ordinary and customary meaning to those of ordinary skill in the art and is not limited to a special or customary meaning. The term specifically refers to, but is not limited to, at least one device configured to provide substantially continuous spatial illumination. As used herein, the term "flood light" is a broad term and is to be given its ordinary and customary meaning to those of ordinary skill in the art and is not limited to a special or customary meaning. The term specifically refers to, but is not limited to, substantially continuous spatial illumination, particularly diffuse and / or uniform illumination. The flood light has a wavelength in the infrared range, particularly in the near-infrared range. The flood illumination source may include at least one LED or at least one VCSEL, preferably a plurality of VCSELs. As used herein, the term "substantially continuous spatial illumination" is a broad term and is to be given its ordinary and customary meaning to those of ordinary skill in the art and is not limited to a special or customary meaning. The term specifically refers to, but is not limited to, uniform spatial illumination, although non-uniform areas are possible. The area illuminated from the flood illumination source, for example, covering a user, a portion of the user, and / or the user's face, may be continuous. Power may be distributed across the entire illumination field. In contrast, the illumination provided by a light pattern can include at least two consecutive areas, in particular multiple consecutive areas, and / or the power can be concentrated in a smaller area (compared to the entire illumination field) of the illumination field. Infrared flood illumination can be suitable for illuminating a consecutive area, in particular one consecutive area. Infrared pattern illumination can be suitable for illuminating at least two consecutive areas.

[0027] The flood illumination source may illuminate a measurement area, such as a user, a portion of a user, and / or a user's face, with a substantially constant illumination intensity. As used herein, the term "constant" is a broad term and is to be given its ordinary and conventional meaning to one of ordinary skill in the art and is not limited to a special or customized meaning. The term may specifically refer to, but is not limited to, the temporal aspect during the exposure time. The flood illumination may vary in time and / or may be substantially constant over time. As used herein, the term "substantially constant" is a broad term and is to be given its ordinary and conventional meaning to one of ordinary skill in the art and is not limited to a special or customized meaning. The term may specifically refer to, but is not limited to, completely constant illumination, as well as embodiments that allow for deviations from constant illumination of ≤ ± 10%, preferably ≤ ± 5%, and more preferably ≤ ± 2%.

[0028] As used herein, the term "image generation unit" is a broad term and is to be given its ordinary and conventional meaning to those of ordinary skill in the art and is not limited to a special or customized meaning. The term may specifically refer to, but is not limited to, at least one unit of an optoelectronic device configured to generate at least one image. The image may be generated via a hardware and / or software interface, which may be considered an image generation unit. As used herein, the term "image generation" is a broad term and is to be given its ordinary and conventional meaning to those of ordinary skill in the art and is not limited to a special or customized meaning. The term may specifically refer to, but is not limited to, capturing and / or generating and / or determining and / or recording at least one image using an image generation unit. Image generation may include performing imaging and / or recording images. Image generation may include capturing a single image and / or multiple images, such as a sequence of images. To generate an image via the hardware and / or software interface, the capturing and / or generating and / or determining and / or recording of the image may be caused and / or initiated by the hardware and / or software interface. For example, image generation may include continuously recording a sequence of images, such as a video or movie. Image generation may be initiated by a user action or may be initiated automatically, for example upon automatic detection of the presence of at least one object or user within the field of view of the image generating unit and / or within a predetermined area of ​​the field of view.

[0029] The image generating unit may comprise at least one optical sensor, in particular at least one pixelated optical sensor. The image generating unit may comprise at least one CMOS sensor or at least one CCD chip. For example, the image generating unit may comprise at least one CMOS sensor, which may be sensitive in the infrared spectral range. The term "image" as used herein is a broad term and is to be given its ordinary and conventional meaning for a person of ordinary skill in the art and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, data recorded using an optical sensor, such as a plurality of electronic readings from a CMOS or CCD chip. The image may comprise raw image data or may be a pre-processed image. For example, pre-processing may comprise applying at least one filter and / or at least one background correction and / or at least one background subtraction to the raw image data.

[0030] For example, the image generation unit may include one or more of the following: at least one monochrome camera (e.g., including monochrome pixels), at least one color (e.g., RGB) camera (e.g., including color pixels), and at least one IR camera. The camera may be a CMOS camera. The camera may include at least one monochrome camera chip, such as a CMOS chip. The camera may include at least one color camera chip, such as an RGB CMOS chip. The camera may include at least one IR camera chip, such as an IR CMOS chip. For example, the camera may include monochrome (e.g., black and white) pixels and color pixels. Color pixels and monochrome pixels may be combined within the camera. The camera may typically include a one-dimensional or two-dimensional array of image sensors (e.g., pixels).

[0031] As described above, the image generation unit may be at least one camera. For example, the camera may be an internal camera and / or an external camera of a device including an optoelectronic device. As described above, the internal camera and / or the external camera of the device may be accessed via a hardware and / or software interface included in the optoelectronic device that functions as the image generation unit. In the case where the device is or includes a smartphone, the image generation unit may be the smartphone's front-facing camera (e.g., a selfie camera) and / or a rear-facing camera.

[0032] The image generating unit may have a field of view between 10°×10° and 75°×75°, preferably 55°×65°. For example, the field of view may be between 20°×20° and 65°×65°, more preferably between 30°×30° and 60°×60°, and most preferably between 55°×65°. The image generating unit may have a resolution of less than 2 MP, preferably between 0.3 MP and 1.5 MP.

[0033] The image generation unit may include additional elements, such as one or more optical elements, for example one or more lenses. As an example, the optical sensor may be a fixed-focus camera, at least one of whose lenses is fixed relative to the camera's adjustment. Alternatively, however, the camera may also include one or more variable lenses that can be adjusted automatically or manually. The camera may also include at least one optical filter, such as at least one bandpass filter. The bandpass filter may be matched to the spectrum of the light emitter. However, other cameras are also feasible.

[0034] As used herein, the term "pattern image" is a broad term and is to be given its ordinary and conventional meaning to a person of ordinary skill in the art and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, an image generated by an image generation unit while irradiating (e.g., on an object and / or a user) with an infrared light pattern. The pattern image may include an image showing at least a portion of the user, in particular the user's face, in particular on a corresponding region of interest included in the image, while irradiating the user with the infrared light pattern. The pattern image may be generated by imaging and / or recording light reflected by an object and / or user irradiated by the infrared light pattern. The pattern image showing the user may include at least a portion of the irradiated infrared light pattern on at least a portion of the user. For example, the irradiation of the pattern illumination source and the imaging using the optical sensor may be synchronized, for example, by using at least one control unit of the optoelectronic device.

[0035] As used herein, the term "flood image" is a broad term and is to be given its ordinary and customary meaning to one of ordinary skill in the art and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, an image generated by an image generation unit when an illumination source emits infrared flood light (e.g., on an object and / or user). The flood image may include an image showing the user, particularly the user's face, while the user is illuminated by the flood light. The flood image may be generated by imaging and / or recording light reflected by the object and / or user illuminated by the flood light. The flood image showing the user may include at least a portion of the flood light on at least a portion of the user. For example, the illumination by the flood illumination source and the imaging using the optical sensor may be synchronized, for example, by using at least one control unit of the optoelectronic device.

[0036] The image generation unit may be configured to image and / or record the pattern image and the flood image simultaneously or at different times. The image generation unit may be configured to image and / or record the pattern image and the flood image at at least partially overlapping measurement areas or equivalents of these measurement areas.

[0037] The optoelectronic device may be included in the device. In particular, the optoelectronic device may be part of the device. The device may include at least one display, wherein the infrared light pattern, when emitted from the pattern illumination source, passes through the display, and / or the infrared flood light, when emitted from the flood illumination source, passes through the display. The display is at least partially transparent in at least one continuous region covering the pattern illumination source and / or the flood illumination source and / or the image generation unit.

[0038] As used herein, the term "display" is a broad term and is to be given its ordinary and customary meaning to those of ordinary skill in the art and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, a device of any shape configured to display an information item. The information item may be any information, such as at least one image, at least one chart, at least one histogram, at least one graph, text, a number, at least one symbol, an operating menu, and the like. The display may be or may include at least one screen. The display may have any shape, such as a rectangular shape. The display may be a front-facing display of the device.

[0039] The display may be or may include at least one organic light-emitting diode (OLED) display. As used herein, the term "organic light-emitting diode" is a broad term and is to be given its ordinary and customary meaning to those of ordinary skill in the art and is not limited to a special or customary meaning. The term may specifically refer to, but is not limited to, a light-emitting diode (LED), in which the emissive electroluminescent layer is an organic compound film configured to emit light in response to an electric current. OLED displays may be configured to emit visible light.

[0040] The display, in particular the display area, can be made of glass and / or covered by glass. In particular, the display can comprise at least one glass cover plate.

[0041] The display is at least partially transparent. As used herein, the term "at least partially transparent" is a broad term and is to be given its ordinary and customary meaning to those skilled in the art and is not limited to a special or customary meaning. The term may specifically, but is not limited to, the property of a display that allows light, particularly light in a specific wavelength range (e.g., light in the infrared spectral region, particularly light in the near-infrared spectral region), to at least partially pass through. For example, a display may be semi-transparent in the near-infrared region. For example, a display may have a transparency of 20% to 50% in the near-infrared region. The display may have different transparency for other wavelength ranges. The present invention may provide an optoelectronic device comprising an image generating unit and two illumination sources that may be positioned behind the device's display. Transparent regions of the display may allow the optoelectronic device to operate behind the display. As described above, the display is at least partially transparent. The display may have a reduced pixel density and / or reduced pixel size and / or may include at least one transparent conductive path. The transparent regions of the display may have a pixel density of 360 to 440 pixels per inch (PPI). Other areas of the display (eg, non-transparent areas) may have a pixel density higher than 400 PPI, such as 460 to 500 PPI.

[0042] The display may include a display area. As used herein, the term "display area" is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, an active area of ​​the display, in particular an activatable area. The display may have additional areas, such as recesses or cutouts. The display may be at least partially transparent in at least one continuous area, preferably in at least two continuous areas. At least one of the continuous areas at least partially covers the image generating unit and / or the pattern illumination source and / or the flood illumination source. The pattern illumination source, the flood illumination source and the image generating unit may be placed in front of the display in the propagation direction of the infrared light pattern.

[0043] A display may have a first area associated with a first pixel density (pixels per inch (PPI)) value and a second area associated with a second pixel density value. For example, the display may include a first area having a PPI value less than 350 and / or a second area having a PPI value equal to or greater than 400. The first pixel density value may be lower than the second pixel density value. The first pixel density value may be equal to or lower than 450 PPI, preferably from 300 to 440 PPI, more preferably 350 to 450 PPI. The second pixel density value may be 400 to 500 PPI, preferably 450 to 500 PPI. The first pixel density value may be associated with at least one continuous area that is at least partially transparent. The first pixel density value may be lower than 350 and the second pixel density value may be equal to or greater than 400.

[0044] As mentioned above, existing displays that meet these requirements are less appealing to users, and there's a desire to reduce the transparent areas. Furthermore, since the transmittance in the transparent areas is typically 15% to 50%, covering the image-generating unit with such displays can result in low contrast. Contrast can be defined as the difference between the signal and the backlight. In this case, contrast can be considered reduced by transmission. However, contrast can be defined as the ratio of signal to background. In this case, contrast may be reduced not only by transmission but also by diffraction in the projected spot pattern, as higher-order diffraction, in addition to the zeroth-order, also contributes additional intensity. Diffraction can further reduce irradiance due to the structure of the display wiring. Furthermore, the spot pattern projected through the display can produce a main spot corresponding to the pattern before passing through the display area. This main spot has reduced intensity due to the generation of higher-order spots. These two effects result in a reduction in irradiance to approximately 3%-5% of the initial irradiance, and the presence of undesirable additional spots in the pattern. This effect can occur particularly when exiting the device (first pass through the display) and entering the device (second pass through the display). The present invention allows for providing sufficient irradiance in the pattern image, thereby increasing contrast. Contrast can be increased by reducing the number of light spots projected onto the user. This can result in increased irradiance of the light spots, and therefore increased contrast in the projected image of the light spot pattern. Furthermore, transparent areas in the display can be reduced. Using this optoelectronic device can, for example, allow for repositioning the camera from its outermost position on the display. This repositioning can optimize wiring, as it allows for optimization of another location, namely the image generation unit. This can reduce wiring usage or achieve improved battery operation.

[0045] The relative distance between the flood illumination source and the pattern illumination source may be less than 3.0 mm. The relative distance between the flood illumination source and the pattern illumination source may be less than 3.0 mm. The distance between the flood illumination source and the pattern illumination source may define the distance between the points at which the flood illumination source and the pattern illumination source are furthest apart. The relative distance between the flood illumination source and the pattern illumination source may be less than 2.5 mm, preferably less than 2.0 mm. The lower limit of the relative distance may be 50 μm, preferably 60 μm, more preferably 70 μm, even more preferably 80 μm, and most preferably 100 μm.

[0046] The pattern illumination source and the flood illumination source can be combined into a single module. For example, the pattern illumination source and the flood illumination source can be arranged on the same substrate, particularly at a minimum relative distance. The minimum relative distance can be defined by the physical extension of the flood and pattern illumination sources. Arranging the pattern illumination source and the flood illumination source at a relative distance of less than 3.0 mm can reduce the space requirements of the two illumination sources. In particular, the illumination sources can even be combined into a single module. This reduced space requirement can allow for a reduction in the transparent area(s) required in the display for operation of the illumination source(s) behind the display.

[0047] In a further aspect of the invention, the use of an optoelectronic device according to the invention for authenticating a user of a device comprising the device is disclosed.

[0048] In another aspect of the present invention, a device for authenticating a user of a device to perform at least one operation on the device that requires authentication is disclosed.

[0049] The device includes:

[0050] - at least one flood illumination source configured to emit infrared flood light;

[0051] at least one patterned illumination source configured to emit at least one infrared light pattern comprising a plurality of infrared light spots, wherein the number of the infrared light spots is less than or equal to 4000 spots;

[0052] at least one image generation unit configured to generate at least one pattern image when the pattern illumination source emits an infrared light pattern, and configured to generate at least one flood image when the flood illumination source emits an infrared flood light;

[0053] at least one display, wherein the infrared light pattern passes through the display when illuminated from the pattern illumination source and / or the infrared flood light passes through the display when illuminated from the flood illumination source, wherein the display of the device is at least partially transparent in at least one continuous area covering the pattern illumination source, the flood illumination source and / or the image generating unit,

[0054] - at least one authentication unit configured to perform at least one authentication process of a user using the flood image and the pattern image.

[0055] In particular, the apparatus may comprise at least one optoelectronic device according to the invention.Thus, with regard to details, options and definitions, reference may be made to the apparatus and optoelectronic devices as discussed above or as described in further detail below.

[0056] The device may be selected from the group consisting of: a television device; a game console; a personal computer; a mobile device, in particular a mobile phone, and / or a smartphone, and / or a tablet computer, and / or a laptop computer, and / or a tablet computer, and / or a virtual reality device, and / or a wearable device, such as a smart watch; or other type of portable computer.

[0057] As used herein, the term "authentication" is a broad term and is to be given its ordinary and customary meaning to persons of ordinary skill in the art and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, verifying the identity of a user. Specifically, authentication may include distinguishing a user from other humans or objects, and in particular, distinguishing authorized access from unauthorized access. Authentication may include verifying the identity of a respective user and / or assigning an identity to the user. Authentication may include generating and / or providing identity information, for example, to another device or unit (e.g., to at least one authorization unit) for authorization to provide access to the device. Identity information may be proven through authentication. For example, the identity information may be and / or may include at least one identity token. If authentication is successful, the facial image recorded by the image generation unit may be verified as the facial image of the user, and / or the user's identity may be verified. Authentication may be performed using at least one authentication process. The authentication process may include multiple steps, such as at least one facial detection in a flood image, and at least one recognition step in which an identity is assigned to the detected face and / or at least one identity check is performed and / or the user's identity is verified.

[0058] As used herein, the term "authentication unit" is a broad term and is to be given its ordinary and customary meaning to those of ordinary skill in the art and is not limited to a special or custom meaning. The term may specifically, but is not limited to, at least one unit configured to perform at least one authentication process for a user. The authentication unit may be or include at least one processor. A processor may be any logic circuit configured to perform the basic operations of a computer or system, and / or generally refers to a device configured to perform computational or logical operations. In particular, a processor may be configured to process the basic instructions that drive a computer or system. As an example, a processor may include at least one arithmetic logic unit (ALU), at least one floating point unit (FPU), such as a math coprocessor or digital coprocessor, multiple registers, specifically registers configured to provide operands to the ALU and store operation results, and memory such as L1 and L2 cache memory. In particular, the processor may be a multi-core processor. In particular, the processor may be or include a central processing unit (CPU). Additionally or alternatively, the processor may be or include a microprocessor, and thus, in particular, the components of the processor may be contained on a single integrated circuit (IC) chip. Additionally or alternatively, the processor may be or include one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs) and / or one or more tensor processing units (TPUs) and / or one or more chips, such as dedicated machine learning optimization chips. The processor may be configured, for example, through software programming, to perform one or more evaluation operations. At least one or any component of the computer program configured to perform the authentication process may be executed by the processing device. Alternatively or additionally, the authentication unit may be or include a connection interface. The connection interface may be configured to transfer data from the device to a remote device, or vice versa. At least one or any component of the computer program configured to perform the authentication process may be executed by the remote device.

[0059] For example, the authentication unit can perform at least one facial detection using the flood image. Facial detection can be performed locally on the device. However, facial recognition (i.e., assigning an identity to the detected face) can be performed remotely, such as in the cloud, particularly when identification rather than just verification is required. The user template can be stored remotely, such as in the cloud, and need not be stored locally. This can be advantageous from a storage and security perspective.

[0060] The authentication unit may be configured to identify the user based on the floodlight image. Thus, in particular, the authentication unit may forward data to a remote device. Alternatively or additionally, the authentication unit may perform identification of the user based on the floodlight image, in particular by running an appropriate computer program with corresponding functionality. As used herein, the term "identification" is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, assigning an identity to a detected face and / or at least one identity check and / or verifying the identity of a user.

[0061] The authentication process may include multiple steps. For example, the authentication process may include performing at least one facial detection. The facial detection step may include analyzing the flood image. Additionally, for example, the authentication process may include recognition. Recognition may include assigning an identity to the detected face and / or at least one identity check and / or verifying the identity of the user. Recognition may include performing facial verification on the imaged face to confirm whether it is the face of the user. Recognizing the user may include matching the flood image (e.g., showing a portion of the user, in particular an outline of a portion of the user's face) to a template. Recognizing the user may include determining whether the imaged face is the face of the user, in particular determining whether the imaged face corresponds to at least one image of the user's face stored, for example, in at least one memory of the device.

[0062] Analysis of the flood image may include one or more of the following: filtering; selecting at least one region of interest; forming a difference image between the flood image and at least one offset; inverting the flood image; background correction; decomposition into color channels; decomposition into hue, saturation, and brightness channels; frequency decomposition; singular value decomposition; applying a Canny edge detector; applying a Laplacian of Gaussian filter; applying a Difference of Gaussian filter; applying a Sobel operator; applying a Laplacian operator; applying a Scharr operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying a Radon transform; applying a Hough transform; applying a wavelet transform; thresholding; creating a binary image. The region of interest may be manually determined by a user or may be automatically determined, such as by identifying a user within the image. In particular, analysis of the flood image may include using at least one image recognition technique, in particular a facial recognition technique. The image recognition technique includes at least one process for identifying a user within the image. Image recognition may include using at least one technique selected from the following: color-based image recognition, for example using features such as template matching; image segmentation and / or connected component analysis, for example using size or shape; machine learning and / or deep learning, for example using at least one convolutional neural network.

[0063] The analysis of the flood image may include determining a plurality of facial features. The analysis may include comparing the determined facial features with template features, in particular matching them. Template features may be features extracted from at least one template. The template may be or include at least one image generated during a registration process (e.g., when initializing the device). The template may be an image of an authorized user. The template features and / or the facial features may include vectors. Matching the features may include determining a distance between the vectors. Identifying the user may include comparing the distance of the vectors with at least one predefined limit, wherein the user is successfully identified if the distance is at least within a tolerance ≤ the predefined limit. In other cases, the user is declined and / or rejected.

[0064] For example, image recognition can include using at least one model, in particular a trained model including at least one facial recognition model. Analysis of the flood image can be performed using a facial recognition system, such as FaceNet, as described, for example, in Florian Schroff, Dmitry Kalenichenko, and James Philbin, “FaceNet: A Unified Embedding for face Recognition and Clustering,” arXiv:1503.03832. The trained model can include at least one convolutional neural network. For example, the convolutional neural network can be designed as described in MD Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” CoRR, abs / 1311.2901, 2013; or C. Szegedy et al., “Going deeper with convolutions,” CoRR, abs / 1409.4842, 2014. For more details on convolutional neural networks for facial recognition systems, see Florian Schroff, Dmitry Kalenichenko, and James Philbin, “FaceNet: A Unified Embedding for face Recognition and Clustering,” arXiv:1503.03832. Labeled image data from an image database can be used as training data.In particular, labeled faces from one or more of GB Huang, M. Ramesh, T. Berg, and E. Learned-Miller, "Labeled faces in the wild: A database for studying face recognition in unconstrained environments," Technical Report 07-49, University of Massachusetts Amherst, October 2007; the Youtube® Faces database as described in L. Wolf, T. Hassner, and I. Maoz, "Face recognition in unconstrained videos with matchedbackground similarity," IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2011; or the Google® Facial Expression Comparison Dataset can be used. Convolutional neural networks can be trained as described in Florian Schroff, Dmitry Kalenichenko, James Philbin, “FaceNet: A Unified Embedding for face Recognition and Clustering,” arXiv:1503.03832.

[0065] The authentication unit can further be configured to determine material data based on the pattern image. Thus, in particular, the authentication unit can forward the data to a remote device. Alternatively or additionally, the authentication unit can perform material determination based on the pattern image, in particular by running a suitable computer program with corresponding functionality. In particular, by considering the material as a parameter for verifying the authentication process, the authentication process can be robust against fraud using a recorded user image.

[0066] The authentication unit can be configured to extract material data from the pattern image by beam profile analysis of the light spot. Regarding beam profile analysis, reference is made to WO 2018 / 091649 A1, WO 2018 / 091638 A1 and WO 2018 / 091640A1, the entire contents of which are incorporated herein by reference. Beam profile analysis can allow reliable classification of scenes based on several light spots. Each light spot of the pattern image can include a beam profile. As used herein, the term "beam profile" can generally refer to at least one intensity distribution of a light spot on an optical sensor as a function of a pixel. The beam profile can be selected from the group consisting of: a trapezoidal beam profile; a triangular beam profile; a conical beam profile, and a linear combination of Gaussian beam profiles.

[0067] The authentication unit can be configured to outsource at least one step of the authentication process (such as user identification) and / or at least one step of the verification process (such as consideration of material data) to a remote device, specifically a server and / or a cloud server. The device and the remote device can be part of a computer network, specifically the Internet. Thus, the device can function as a field device used by a user to generate data required for the authentication process and / or its verification. The device can transmit the generated data and / or data associated with intermediate steps of the authentication process and / or its verification to the remote device. In this scenario, the authentication unit can be and / or include a connection interface configured to transmit information to the remote device. Data generated by the remote device used in the authentication process and / or its verification can be further transmitted to the device. This data can be received by a connection interface included in the device. The connection interface can be specifically configured to transmit or exchange information. In particular, the connection interface can provide a data transmission connection. As an example, the connection interface can be or include at least one port, including one or more of a network or Internet port, a USB port, and a disk drive.

[0068] It is important to note that data from a device can be transmitted to a specific remote device based on at least one condition (e.g., date, time of day, load on a specific remote device, etc.). A field device may not be able to select a specific remote device. Conversely, another device may select a specific remote device to which data may be transmitted. The authentication process and / or generation of verification data may involve several different entities using the remote device. At least one entity may generate intermediate data and transmit the intermediate data to at least one other entity.

[0069] The authentication unit may be configured to use a facial recognition authentication process operating on the flood image, the pattern image and / or the extracted material data.The authentication unit may be configured to extract the material data from the pattern image.

[0070] In an embodiment, extracting material data from a pattern image may include generating a material type and / or data derived from the material type. Preferably, extracting material data may be based on the pattern image. Material data may be extracted using at least one model. Extracting material data may include providing the pattern image to the model and / or receiving material data from the model. Providing the image to the model may include and may be followed by receiving the pattern image at the model's input layer or via a model loss function. The model may be a data-driven model. Data-driven models may include convolutional neural networks and / or encoder-decoder structures, such as autoencoders. Other examples of methods for generating representations may include FFTs, wavelets, deep learning (e.g., CNNs), energy models, normalizing flows, GANs, visual transformers or transformers for natural language processing, autoregressive image modeling, normalizing flows, deep autoencoders, and deep energy-based models. Supervised or unsupervised approaches may be used to generate representations, as well as to generate embeddings in ML languages, such as cosine or Euclidean metrics. The data-driven model may be parameterized based on a training dataset comprising at least one image and material data, preferably at least one pattern image and material data. In another embodiment, extracting material data may include providing an image to a model and / or receiving material data from the model. In another embodiment, the data-driven model may be trained based on a training dataset comprising at least one image and material data. In another embodiment, the data-driven model may be parameterized based on a training dataset comprising at least one image and material data. The data-driven model may be parameterized based on the training dataset to receive an image and provide material data based on the received image. The data-driven model may be trained based on the training dataset to receive an image and provide material data as output based on the received image. The training dataset may include at least one image and material data (preferably material data associated with the at least one image). The image may include a representation of the image. The representation may be a low-dimensional representation of the image. The representation may include at least a portion of data or information associated with the image. The representation of the image may include a feature vector. In an embodiment, determining the representation, particularly the low-dimensional representation, may be based on a principal component analysis (PCA) map or a radial basis function (RBF) map. Determining the representation may also be referred to as generating a representation. Generating the representation based on the PCA map may include clustering based on features in the pattern image and / or the portion of the image. Additionally or alternatively, generating the representation may be based on a neural network structure suitable for dimensionality reduction. A neural network structure suitable for dimensionality reduction may include an encoder and / or a decoder. In an example, the neural network structure may be an autoencoder. In an example, the neural network structure may include a convolutional neural network (CNN). The CNN may include at least one convolutional layer and / or at least one pooling layer.CNN can reduce the dimensionality of parts of an image and / or an image by applying convolution (e.g., based on a convolutional layer) and / or by pooling. Applying convolution can be suitable for selecting features related to material information of the pattern image.

[0071] In embodiments, a model may be adapted to determine an output based on an input. In particular, a model may be adapted to determine material data based on an image as input. The model may be a deterministic model, a data-driven model, or a hybrid model. Preferably, a deterministic model reflects physical phenomena in mathematical form, such as a first-principles model. A deterministic model may include a set of equations that describe the interaction between a material and patterned electromagnetic radiation, thereby generating a condition metric, a vital sign metric, or the like. A data-driven model may be a classification model. A hybrid model may include at least one machine learning architecture and model parameters with deterministic or statistical adjustments. Statistical or deterministic adjustments may be introduced to improve the quality of the results, as these adjustments provide a systematic relationship between empiricism and theory. In embodiments, a data-driven model may be a classification model. The classification model may include at least one machine learning architecture and model parameters. For example, the machine learning architecture may be or may include one or more of the following: linear regression, logistic regression, random forest, piecewise linear classifier, nonlinear classifier, support vector machine, naive Bayesian classification, nearest neighbor, neural network, convolutional neural network, generative adversarial network, support vector machine, or gradient boosting algorithm, among others. In the case of a neural network, the model can be a multi-scale neural network or a recurrent neural network (RNN), such as, but not limited to, a gated recurrent unit (GRU) recurrent neural network or a long short-term memory (LSTM) recurrent neural network. The data-driven model can be parameterized based on a training dataset. The data-driven model can be trained based on the training dataset. Training the model can include parameterizing the model. The term "training" can also be expressed as learning. The term can specifically refer to, but is not limited to, the process of building a classification model, and in particular, determining and / or updating the parameters of the classification model. Updating the parameters of the classification model can also be referred to as retraining. The training referred to herein can include retraining. In an embodiment, the training dataset can include at least one image and material information.

[0072] In an embodiment, extracting material data from an image using a data-driven model may include providing the image to the data-driven model. Additionally or alternatively, extracting material data from an image using a data-driven model may include generating an embedding associated with the image based on the data-driven model. An embedding may refer to a low-dimensional representation associated with an image, such as a feature vector. The feature vector may be suitable for suppressing background while maintaining a material signature indicative of material data. In this context, background may refer to information independent of the material signature and / or material data. Further, background may refer to information related to biometric features such as facial features. Based on the embedding associated with the image, the material data may be determined using the data-driven model. Additionally or alternatively, extracting material data from an image by providing the image to the data-driven model may include transforming the image into material data, in particular, a material feature vector indicative of the material data. Therefore, the material data may further include a material feature vector and / or the material feature vector may be used to determine the material data.

[0073] In an embodiment, the authentication process may be verified based on the extracted material data.

[0074] In an embodiment, verifying based on the extracted material data may include determining whether the extracted material data corresponds to expected material data. Determining whether the extracted material data matches the expected material data may be referred to as verification. Allowing or denying the user and / or subject from performing at least one operation requiring authentication on the device based on the material data may include verifying the authentication or authentication process. Verification may be based on the material data and / or an image. Determining whether the extracted material data corresponds to the expected material data may include determining a similarity between the extracted material data and the expected material data. Determining the similarity between the extracted material data and the expected material data may include comparing the extracted material data with the expected material data. The expected material data may refer to predetermined material data. In an example, the expected material data may be skin. Determining whether the material data corresponds to the expected material data may include comparing the material data with the expected material data. Comparing the material data with the expected material data may result in allowing or denying the user and / or subject from performing the at least one operation requiring authentication. In an example, skin as the expected material data may be compared to non-skin materials or silicon as the material data, and the results may be skewed because silicon or non-skin materials may differ from skin.

[0075] In an embodiment, the authentication process or verification thereof may include generating at least one feature vector from the material data and matching the material feature vector with an associated material reference template vector.

[0076] The authentication unit may be configured to authenticate the user if the user is recognizable and / or if the material data matches the expected material data. The device may include at least one authorization unit configured to allow the user to perform at least one operation on the device, such as unlocking the device if the user is successfully authenticated, or denying the user from performing at least one operation on the device if the user is unsuccessfully authenticated. Thus, the user can be aware of the result of the authentication.

[0077] As already pointed out above, the relative distance between the flood illumination source and the pattern illumination source may be lower than 3.0 mm.

[0078] In another aspect, the present invention discloses a method for authenticating a user of a device to perform at least one operation requiring authentication on the device.

[0079] The apparatus comprises a display, and the method comprises:

[0080] a. illuminating the user with at least one infrared light pattern from at least one pattern illumination source of the device, wherein the number of infrared light spots is less than or equal to 4000 spots,

[0081] b. illuminating the user with infrared flood light from at least one flood illumination source of the device,

[0082] c. generating, with an image generation unit of the device, at least one pattern image showing the user, in particular at least a portion of the user's face, when illuminating the user with an infrared light pattern, and generating, with the image generation unit of the device, at least one image showing the user, when illuminating the user with the infrared flood light, wherein the display of the device is at least partially transparent in at least one continuous area covering the pattern illumination source, the flood illumination source and / or the image generation unit,

[0083] d. identifying the user based on the flood image by using at least one authentication unit of the device,

[0084] e. extracting material data from the at least one pattern image by using the authentication unit, and

[0085] f. Allowing the user to perform at least one operation requiring authentication on the device based on the material data and the identification.

[0086] These method steps may be performed in the order given or in a different order. Furthermore, there may be one or more additional method steps not listed. Furthermore, one, more than one, or even all of the method steps may be repeated. For details, options, and definitions, reference may be made to the optoelectronic devices and apparatus discussed above. Thus, in particular, the method may include using an apparatus according to the present invention (e.g., according to one or more of the embodiments presented above or described in further detail below).

[0087] Identifying the user may include matching the flood image to a template.

[0088] The method may include using a facial recognition authentication process operating on the flood image, the pattern image and / or the extracted material data.The pattern image and / or the image showing the user when illuminated with the infrared flood may show at least a portion of the user's face.

[0089] All described method steps can be performed using the device. Thus, a single processing device can be configured to exclusively execute at least one computer program, in particular at least one line of computer program code configured to execute at least one algorithm, as used in at least one embodiment of the method according to the present invention. In this article, a computer program executed on a single processing device may include all instructions for causing a computer to perform the method. Alternatively or additionally, at least one method step can be performed using at least one remote device, the at least one remote device being particularly selected from at least one of a server or a cloud server, in particular when the device and the remote device can be part of a computer network. In this case, the computer program may include at least one remote component to be executed by at least one remote processing device to perform at least one method step. The remote component may have the function of performing identification of a user and / or extraction of material data. Further, the computer program may include at least one interface configured to forward data to and / or receive data from at least one remote component of the computer program.

[0090] The method may include allowing or denying a user to perform at least one operation on the device. In an embodiment, allowing or denying a user to perform at least one operation requiring authentication on the device based on material data may include: allowing the user to perform at least one operation requiring authentication on the device if the material data matches the expected material data and / or the authentication may be successful. The expected material data may be predetermined material data. The authentication may be successful if the user can be identified and / or if the material data matches the expected material data. Further, allowing or denying an object to perform at least one operation requiring authentication on the device based on material data may include: denying performance of at least one operation requiring authentication on the device if the material data does not match the expected material data and / or the authentication may be unsuccessful. The authentication may be unsuccessful if the pattern image cannot be matched with the image template and / or if the material data does not match the expected material data.

[0091] At least one operation on the device that requires authentication may be accessing the device (e.g., unlocking the device) and / or accessing an application preferably associated with the device and / or accessing a portion of an application preferably associated with the device. In an embodiment, allowing a user to access a resource may include allowing the user to perform at least one operation with the device and / or system. A resource may be a device, a system, a function of a device, a function of a system, and / or an entity. Additionally and / or alternatively, allowing a user to access a resource may include allowing the user to access an entity. The entity may be a physical entity and / or a virtual entity. A virtual entity may be, for example, a database. A physical entity may be an area with restricted access. The area with restricted access may be one of the following: a secure area, a room, an apartment, a vehicle, a portion of the aforementioned examples, etc. The device and / or system may be locked. The device and / or system may only be unlocked by an authorized user.

[0092] The term "user" as used herein is a broad term and is to be given its ordinary and customary meaning to those of ordinary skill in the art and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, a person who intends to use a device and / or uses the device.

[0093] The method may be computer-implemented. As used herein, the term "computer-implemented" is a broad term and is to be given its ordinary and conventional meaning to those of ordinary skill in the art and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, a method involving at least one computer and / or at least one computer network. The computer and / or computer network may include at least one processor that may be configured to perform at least one method step of the method according to the present invention. In particular, each of these method steps is performed by a computer and / or computer network. The method may be performed fully automatically, in particular without user interaction.

[0094] This document further discloses and proposes a computer program comprising computer-executable instructions for executing the method according to the present invention in accordance with one or more embodiments herein when the program is executed on a computer or computer network. Specifically, the computer program may be stored on a computer-readable data carrier and / or computer-readable storage medium. The computer program may be executed on at least one processor included in an optoelectronic device and / or apparatus. The computer program may generate input data by accessing and / or controlling at least one unit of the optoelectronic device and / or apparatus (e.g., a pattern illumination source and / or a flood illumination source and / or an image generation unit). The computer program may generate result data based on the input data, in particular by using an authentication unit.

[0095] As used herein, the terms "computer-readable data carrier" and "computer-readable storage medium" may specifically refer to a non-transitory data storage device, such as a hardware storage medium, on which computer-executable instructions are stored. The stored computer-executable instructions may be associated with a computer program. A computer-readable data carrier or storage medium may specifically be or include a storage medium such as a random access memory (RAM) and / or a read-only memory (ROM).

[0096] Thus, in particular, one, more than one or even all of the method steps a to f as indicated above may be performed by using a computer or a computer network, preferably by using a computer program.

[0097] This document further discloses and proposes a computer program product having program code means for performing the method according to the present invention in accordance with one or more embodiments of the present invention when the program is executed on a computer or computer network. Specifically, the program code means can be stored on a computer-readable data carrier and / or a computer-readable storage medium.

[0098] This document further discloses and proposes a data carrier having a data structure stored thereon, which, after being loaded into a computer or a computer network (for example, into a working memory or main memory of the computer or the computer network), can execute the method according to one or more embodiments disclosed herein.

[0099] This document further discloses and proposes a computer program product having program code means stored on a machine-readable carrier, for performing, when executed on a computer or computer network, a method according to one or more embodiments disclosed herein. As used herein, a computer program product refers to a program that is a tradable product. The product can generally be in any format, such as paper, or on a computer-readable data carrier and / or computer-readable storage medium. Specifically, the computer program product can be distributed via a data network.

[0100] Further disclosed and proposed herein is a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to one or more embodiments disclosed herein.

[0101] Finally, a modulated data signal containing instructions readable by a computer system or a computer network for executing a method according to one or more embodiments disclosed herein is disclosed and proposed herein.

[0102] With reference to the computer-implemented aspects of the present invention, one or more or even all of the method steps of the methods according to one or more of the embodiments disclosed herein can be performed using a computer or computer network. Thus, generally, any of the method steps involving the provision and / or manipulation of data can be performed using a computer or computer network. Generally, these method steps can include any method steps, except for method steps that typically require manual work, such as providing samples and / or performing certain aspects of the actual measurements.

[0103] Specifically, this article further discloses:

[0104] - a computer or a computer network comprising at least one processor, wherein the processor is adapted to perform a method according to one of the embodiments described in this specification,

[0105] - a computer-loadable data structure adapted to perform a method according to one of the embodiments described in this specification when the data structure is executed on a computer,

[0106] - a computer program, wherein the computer program is adapted to perform a method according to one of the embodiments described in this description when the program is executed on a computer,

[0107] - a computer program comprising program means for carrying out the method according to one of the embodiments described in this description when the computer program is executed on a computer or on a computer network,

[0108] - a computer program comprising the program means according to the preceding embodiment, wherein the program means is stored on a computer-readable storage medium,

[0109] a storage medium on which a data structure is stored and wherein the data structure is suitable for carrying out a method according to one of the embodiments described in this specification after being loaded into a main storage device and / or a working storage device of a computer or a computer network, and

[0110] A computer program product having program code means, wherein the program code means can be stored or stored on a storage medium for performing the method according to one of the embodiments described in this description if the program code means is executed on a computer or a computer network.

[0111] As used herein, the terms "having," "including," or "comprising," or any grammatical variations thereof, are used in a non-exclusive manner. Thus, these terms can refer to the absence of additional features in the entity described in that context, in addition to the features introduced by these terms, or the presence of one or more additional features. By way of example, the expressions "A has B," "A includes B," and "A contains B" can refer to the absence of any other elements in A besides B (i.e., A consists solely and solely of B), or the presence of one or more additional elements in entity A, such as element C, elements C and D, or even additional elements, in addition to B.

[0112] Furthermore, it should be noted that the terms "at least one", "one or more", or similar expressions indicating that a feature or element may occur once or more than once are typically used only once when introducing the corresponding feature or element. In most cases, when referring to the corresponding feature or element, the expression "at least one" or "one or more" is not repeated, but in fact, the corresponding feature or element may occur once or more than once.

[0113] Further, as used herein, the terms "preferably", "more preferably", "particularly", "more particularly", "specifically", "more specifically" or similar terms are used in conjunction with optional features, without limiting the possibilities of alternatives. Therefore, the features introduced by these terms are optional features and are not intended to limit the scope of the claims in any way. As the skilled person will recognize, the present invention can be carried out through the use of alternative features. Similarly, features introduced by "in an embodiment of the invention" or similar expressions are intended to be optional features, without any limitation on alternative embodiments of the invention, without any limitation on the scope of the invention, and without any limitation on the possibility of combining features introduced in this manner with other optional or non-optional features of the invention.

[0114] In general, in the context of the present invention, the following embodiments are considered to be preferred:

[0115] In an embodiment, an optoelectronic device is disclosed, the optoelectronic device comprising:

[0116] at least one patterned illumination source configured to emit at least one infrared light pattern comprising a plurality of infrared light spots, wherein the number of infrared light spots is lower than or equal to 4000 spots,

[0117] - at least one flood illumination source configured to emit infrared flood light,

[0118] - at least one image generation unit configured to generate at least one pattern image when the pattern illumination source emits an infrared light pattern, and configured to generate at least one flood image when the flood illumination source emits infrared flood light.

[0119] In an embodiment, the optoelectronic device is included in an apparatus, wherein the apparatus comprises at least one display, wherein the infrared light pattern passes through the display when emitted from the pattern illumination source and / or the infrared flood light passes through the display when emitted from the flood illumination source, wherein the display is at least partially transparent in at least one continuous area covering the pattern illumination source, the flood illumination source and / or the image generating unit.

[0120] In an embodiment, the display comprises:

[0121] a first region having a pixels per inch (PPI) value less than 350, and

[0122] A second region having a PPI value equal to or greater than 400.

[0123] In an embodiment, the infrared light pattern includes equal to or less than 3000 light spots, preferably equal to or less than 2000 light spots.

[0124] In an embodiment, the light spot has a circular shape.

[0125] In an embodiment, the infrared light is coherent, wherein the infrared light pattern is a coherent infrared light pattern.

[0126] In an embodiment, the infrared light pattern is a hexagonal pattern, preferably a hexagonal infrared light pattern, preferably a 2 / 5 hexagonal infrared light pattern.

[0127] In an embodiment, at least one of the infrared light spots is associated with a beam divergence of 0.2° to 0.5°, preferably 0.1° to 0.3°.

[0128] In an embodiment, the infrared light pattern is a near infrared light pattern.

[0129] In an embodiment, the image generating unit has a field of view between 10° × 10° and 75° × 75°, preferably 55° × 65°, more preferably the field of view is between 20° × 20° and 65° × 65°, more preferably between 30° × 30° and 60° × 60°, most preferably 55° × 65°.

[0130] In an embodiment, the image generation unit has a resolution lower than 2 MP, preferably between 0.3 MP and 1.5 MP.

[0131] In an embodiment, the image generation unit comprises at least one CMOS sensor or at least one CCD chip.

[0132] In an embodiment, the patterned illumination source comprises at least one pattern projector configured to generate an infrared light pattern.

[0133] In an embodiment, the pattern illumination source, such as a pattern projector, comprises at least one vertical cavity surface emitting laser (VCSEL), preferably a plurality of VCSELs.

[0134] In an embodiment, the patterned illumination source comprises at least one optical element configured to increase the number of light spots, wherein the at least one optical element comprises at least one diffractive optical element (DOE) and / or at least one metasurface element.

[0135] In an embodiment, the relative distance between the flood illumination source and the pattern illumination source is less than 3.0 mm.

[0136] In an embodiment, the relative distance between the flood illumination source and the pattern illumination source is lower than 2.5 mm, preferably lower than 2.0 mm.

[0137] In an embodiment, the use of an optoelectronic device according to any of the preceding embodiments for authenticating a user of a device comprising the device is disclosed.

[0138] In an embodiment, a device for authenticating a user of a device to perform at least one operation requiring authentication on the device is disclosed, the device comprising:

[0139] - at least one flood illumination source configured to emit infrared flood light;

[0140] at least one patterned illumination source configured to emit at least one infrared light pattern comprising a plurality of infrared light spots, wherein the number of the infrared light spots is less than or equal to 4000 spots;

[0141] at least one image generation unit configured to generate at least one pattern image when the pattern illumination source emits an infrared light pattern, and configured to generate at least one flood image when the flood illumination source emits an infrared flood light;

[0142] at least one display, wherein the infrared light pattern passes through the display when illuminated from the pattern illumination source and / or the infrared flood light passes through the display when illuminated from the flood illumination source, wherein the display of the device is at least partially transparent in at least one continuous area covering the pattern illumination source, the flood illumination source and / or the image generating unit,

[0143] - at least one authentication unit configured to perform at least one authentication process of a user using the flood image and the pattern image.

[0144] In an embodiment, the display is or comprises at least one organic light emitting diode (OLED) display.

[0145] In an embodiment, a display comprises a display area.

[0146] In an embodiment, the display is made of glass and / or is covered by glass.

[0147] In an embodiment, the display of the device is at least partially transparent in at least two consecutive areas.

[0148] In an embodiment, the display has a first area associated with a first pixel density value and a second area associated with a second pixel density value, wherein the first pixel density value is lower than the second pixel density value, preferably the first pixel density value is equal to or lower than 450 PPI.

[0149] In an embodiment, the first pixel density value is associated with at least one continuous area that is at least partially transparent.

[0150] In an embodiment, the first pixel density value is lower than 350 and the second pixel density value is equal to or greater than 400.

[0151] In an embodiment, the number of infrared spots may be lower than 2000 and / or higher than 0, preferably higher than 5, more preferably higher than 10, most preferably higher than 100.

[0152] In an embodiment, the device is selected from the group consisting of: a television device; a game console; a personal computer; a mobile device, in particular a mobile phone, and / or a smartphone, and / or a tablet computer, and / or a laptop computer, and / or a tablet computer, and / or a virtual reality device, and / or a wearable device, such as a smart watch; or other type of portable computer.

[0153] In an embodiment, the authentication unit is configured for using a facial recognition authentication process operating on the pattern image, the flood image and / or the extracted material data.

[0154] In an embodiment, the apparatus comprises at least one optoelectronic device according to any of the preceding embodiments relating to optoelectronic devices.

[0155] In an embodiment, the relative distance between the flood illumination source and the pattern illumination source is less than 3.0 mm.

[0156] In an embodiment, a method for authenticating a user of a device to perform at least one operation requiring authentication on the device is disclosed, the device including a display and the method comprising:

[0157] a. illuminating the user with at least one infrared light pattern from at least one pattern illumination source of the device,

[0158] b. illuminating the user with infrared flood light from at least one flood illumination source of the device,

[0159] c. generating, with an image generation unit of the device, at least one pattern image showing the user when illuminating the user with an infrared pattern, and generating, with the image generation unit of the device, at least one image showing the user when illuminating the user with the infrared flood light, wherein the display of the device is at least partially transparent in at least one continuous area covering the pattern illumination source, the flood illumination source, and / or the image generation unit,

[0160] d. identifying the user based on the flood image by using at least one authentication unit of the device,

[0161] e. extracting material data from the at least one pattern image by using the authentication unit, and

[0162] f. Allowing the user to perform at least one operation requiring authentication on the device based on the material data and the identification.

[0163] In an embodiment, the method comprises using a facial recognition authentication process operating on the pattern image, the flood image and / or the extracted material data.

[0164] In an embodiment, the pattern image shows the user when the user is illuminated with infrared flood light and / or the image shows at least a portion of the user's face.

[0165] In an embodiment, identifying the user comprises matching the flood image to a template.

[0166] In an embodiment, the distance between the flood illumination source and the pattern illumination source is less than 3.0 mm.

[0167] In an embodiment, the method is computer-implemented.

[0168] In an embodiment, a computer program is disclosed comprising instructions which, when executed by a device according to any of the preceding embodiments relating to devices, cause the device to perform a method according to any of the preceding embodiments relating to methods.

[0169] In an embodiment, a computer-readable storage medium comprising instructions is disclosed that, when executed by a device according to any of the preceding device-related embodiments, causes the device to perform a method according to any of the preceding method-related embodiments.

[0170] In an embodiment, a non-transitory computer-readable medium comprising instructions is disclosed that, when executed by one or more processors, cause the one or more processors to perform a method according to any of the preceding method-related embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0171] Further optional details and features of the present invention will become apparent from the following description of preferred exemplary embodiments in conjunction with the dependent claims. In this context, specific features may be implemented individually or in combination with other features. The present invention is not limited to the exemplary embodiments. Exemplary embodiments are schematically illustrated in the accompanying drawings. Identical reference numerals in the various figures refer to identical elements or elements having the same function, or elements that correspond to one another in terms of their function.

[0172] Specifically, in the accompanying drawings:

[0173] Figure 1 An embodiment of the device according to the invention is shown;

[0174] Figure 2 An embodiment of the method according to the present invention is shown; and

[0175] Figure 3 A further embodiment of the device according to the invention is shown. DETAILED DESCRIPTION

[0176] Figure 1 An embodiment of the device 110 of the present invention is shown in a highly schematic manner. For example, the device 110 may be selected from the group consisting of: a television device; a game console; a personal computer; a mobile device, in particular a mobile phone, and / or a smartphone, and / or a tablet computer, and / or a laptop computer, and / or a tablet computer, and / or a virtual reality device, and / or a wearable device, such as a smartwatch; or other types of portable computers.

[0177] In this embodiment, apparatus 110 includes an optoelectronic device 112 according to the present invention. Optoelectronic device 112 includes at least one patterned illumination source 114 configured to emit at least one infrared light pattern comprising a plurality of infrared light spots. The number of infrared light spots is less than or equal to 4000 spots.

[0178] The pattern illumination source 114 can be configured to generate or provide at least one light pattern, in particular at least one infrared light pattern. The light pattern can include multiple light spots. The light spots can be at least partially spatially extended. The infrared light pattern can be a near-infrared light pattern. The infrared light can be coherent. The infrared light pattern can be a coherent infrared light pattern. The pattern illumination source 114 can be configured to emit light of a single wavelength, such as light in the near-infrared region. In other embodiments, the pattern illumination source 114 can be adapted to emit light having multiple wavelengths, for example to allow additional measurements in other wavelength channels. The infrared light pattern can include at least one regular and / or constant and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or a pattern including further convex tessellations. For example, the infrared light pattern is a hexagonal pattern, preferably a hexagonal infrared light pattern, preferably a 2 / 5 hexagonal infrared light pattern. Using a periodic 2 / 5 hexagonal pattern can allow for distinguishing artifacts from usable signals.

[0179] The infrared light pattern may include at least one dot pattern. The infrared light pattern has a low dot density. The number of infrared light spots is less than or equal to 4,000 spots. The infrared light pattern may include equal to or less than 3,000 spots, preferably equal to or less than 2,000 spots. The spots may have a circular shape. Other shapes are possible. The infrared light pattern may have a low dot density, particularly compared to other structured light technologies that typically have 10k-30k dots in a 55 × 38° field of view. Using this low dot density can allow for compensation for the aforementioned diffraction losses. By reducing the number of spots projected onto the object and / or user, the contrast in the pattern image can be increased. Increasing the number of dots reduces the irradiance per dot. Reducing the number of spots may result in an increase in the irradiance of the spots, and therefore an increase in the contrast of the projected pattern image of the infrared light pattern. The infrared light pattern may have a periodic dot pattern with a reduced number of spots, wherein each of the spots has a high irradiance. This light pattern can ensure improved authentication using the patterned illumination source 114 and at least one flood illumination source 116 behind the display 120 as described above, and at least one image generation unit 118. Furthermore, the small number of light spots can ensure compliance with eye safety requirements and stability requirements. The permitted dose can be divided between the light spots of the light pattern.

[0180] At least one of the infrared light spots may be associated with a beam divergence of 0.2° to 0.5°, preferably 0.1° to 0.3°.

[0181] The pattern illumination source 114 may include at least one pattern projector configured to generate an infrared light pattern. The pattern illumination source (e.g., pattern projector) may include at least one emitter, in particular, a plurality of emitters. The emitter may include at least one element selected from the group consisting of at least one laser source and at least one non-laser light source, such as at least one semiconductor laser, at least one double heterostructure laser, at least one external cavity laser, at least one separated confined heterostructure laser, at least one quantum cascade laser, at least one distributed Bragg reflector laser, at least one polariton laser, at least one hybrid silicon laser, at least one extended cavity diode laser, at least one quantum dot laser, at least one volume Bragg grating laser, at least one indium arsenide laser, at least one gallium arsenide laser, at least one transistor laser, at least one diode pump laser, at least one distributed feedback laser, at least one quantum well laser, at least one interband cascade laser, at least one semiconductor ring laser, at least one vertical cavity surface emitting laser (VCSEL); the at least one non-laser light source may be, for example, at least one LED or at least one light bulb. For example, the pattern illumination source 114 (e.g., a pattern projector) includes at least one VCSEL, preferably a plurality of VCSELs. The plurality of VCSELs may be arranged in at least one array, such as a VCSEL matrix. The VCSELs may be arranged on a common substrate or on different substrates. Examples of VCSELs can be found, for example, at en.wikipedia.org / wiki / Verticalcavity_surface-emitting_laser. VCSELs are generally known to those skilled in the art, such as from WO 2017 / 222618 A. Each VCSEL is configured to generate at least one light beam. The VCSEL or VCSELs may be configured to generate a desired number of light spots equal to or less than 4000, preferably equal to or less than 3000, and more preferably equal to or less than 2000. The VCSEL may be configured to emit a light beam having a wavelength in the range of 800 nm to 1000 nm. For example, the VCSEL may be configured to emit a light beam at 808 nm, 850 nm, 940 nm, or 980 nm. Preferably, the VCSEL emits light at 940 nm, since terrestrial solar radiation has a local minimum in irradiance at this wavelength, as described, for example, in CIE 085-1989 "Solar spectral irradiance".

[0182] The pattern illumination source 114 may include at least one optical element (not shown here) configured to increase (e.g., duplicate) the number of light spots (e.g., light spots generated by a pattern projector). The pattern illumination source 114 may include at least one diffractive optical element (DOE) and / or at least one metasurface element. The DOE and / or metasurface element may be configured to generate multiple light beams from a single incident light beam. For example, a VCSEL projecting up to 2,000 light spots and an optical element including multiple metasurface elements may be used to duplicate the number of light spots. Other replications are also possible. For example, one or more VCSELs may be used, and the generated laser light spots may be duplicated using at least one DOE.

[0183] The pattern illumination source 114 may include at least one delivery device (not shown here). The delivery device may include at least one imaging optical device. The delivery device may specifically include one or more of the following: at least one lens, for example, at least one lens selected from the group consisting of at least one focusable lens, at least one aspheric lens, at least one spherical lens, and at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflection element, preferably at least one reflective mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitter; at least one multi-lens system; at least one holographic optical element; at least one meta-optical element. Specifically, the delivery device includes at least one refractive optical lens module. Therefore, the delivery device may include a multi-lens system with refractive properties.

[0184] The optoelectronic device 112 includes at least one flood illumination source configured to emit infrared flood light. Figure 3 It can be concluded that the relative distance between the flood illumination source 116 and the pattern illumination source 114 can be less than 3.0 mm. The relative distance between the flood illumination source 116 and the pattern illumination source 114 can be less than 2.5 mm, preferably less than 2.0 mm. The pattern illumination source 114 and the flood illumination source 116 can be combined into a single module. For example, the pattern illumination source 114 and the flood illumination source 116 can be arranged on the same substrate, in particular with a minimum relative distance. The minimum relative distance can be defined by the physical extension of the flood illumination source 116 and the pattern illumination source 114. Arranging the pattern illumination source 114 and the flood illumination source 116 with a relative distance less than 3.0 mm can reduce the space requirement of the two illumination sources 114, 116. In particular, the illumination sources 114, 116 can even be combined into a single module. This reduced space requirement can allow for a reduction in the transparent area(s) required in the display for operation of the illumination source(s) 114, 116 behind the display 120. Figure 2The reference numerals discussed in the context of may be applied accordingly to Figure 3 .

[0185] The optoelectronic device 112 includes at least one image generation unit 118 configured to generate at least one pattern image when the pattern illumination source 114 emits an infrared light pattern and to generate at least one flood image when the flood illumination source 116 emits an infrared flood light.

[0186] The image generation unit 118 can be at least one unit of the optoelectronic device 112 configured to generate at least one image. Image generation can include capturing and / or generating and / or determining and / or recording at least one image using the image generation unit 118. Image generation can include performing imaging and / or recording images. Image generation can include capturing a single image and / or multiple images, such as a sequence of images. For example, image generation can include continuously recording a sequence of images, such as a video or movie. Image generation can be initiated by a user action or can be initiated automatically, for example, upon automatically detecting the presence of at least one object or user within the field of view of the image generation unit and / or within a predetermined area of ​​the field of view.

[0187] Image generation unit 118 may include at least one optical sensor, in particular at least one pixelated optical sensor. Image generation unit 118 may include at least one CMOS sensor or at least one CCD chip. For example, image generation unit 118 may include at least one CMOS sensor that may be sensitive in the infrared spectral range. The image may be image data recorded using an optical sensor, such as multiple electronic readouts from a CMOS or CCD chip. The image may include raw image data or may be a pre-processed image. For example, pre-processing may include applying at least one filter and / or at least one background correction and / or at least one background subtraction to the raw image data.

[0188] For example, the image generation unit 118 may include a monochrome camera, e.g., comprising monochrome pixels. For example, the image generation unit may include a color camera, e.g., comprising color pixels. The image generation unit may include a color CMOS camera. For example, the camera may include both monochrome and color pixels. Color and monochrome pixels may be combined within the camera. The image generation unit 118 may include at least one color camera (e.g., RGB) and / or at least one monochrome camera, such as a monochrome CMOS. The camera may include at least one monochrome CMOS chip. A camera may typically include a one-dimensional or two-dimensional array of image sensors (e.g., pixels).

[0189] As mentioned above, the image generation unit may be at least one color, such as an RGB camera. For example, the color camera may be a selfie camera of a smartphone.

[0190] The image generating unit may have a field of view between 10°×10° and 75°×75°, preferably 55°×65°.The image generating unit may have a resolution lower than 2 MP, preferably between 0.3 MP and 1.5 MP.

[0191] The image generation unit may include further elements, such as one or more optical elements, for example one or more lenses. As an example, the optical sensor may be a fixed-focus camera, at least one of whose lenses is fixed relative to the camera's adjustment. Alternatively, however, the camera may also include one or more variable lenses that can be adjusted automatically or manually. However, other cameras are also feasible.

[0192] The pattern image may be an image generated by image generation unit 118 when irradiating (e.g., onto an object and / or a user) with an infrared light pattern. The pattern image may include an image showing at least a portion of the user, particularly the user's face, when irradiated with the infrared light pattern. The pattern image may be generated by imaging and / or recording light reflected from the object and / or user illuminated by the infrared light pattern. For example, the illumination by pattern illumination source 114 and the imaging using the optical sensor may be synchronized, for example, using at least one control unit of optoelectronic device 112.

[0193] A flood image may be an image generated by image generation unit 118 when an illumination source, such as an infrared flood light, is emitted at an object and / or a user. The flood image may include an image showing a user, particularly a face of the user, while the flood light is being illuminated. The flood image may be generated by imaging and / or recording light reflected from the object and / or user being illuminated by the flood light. For example, illumination by flood illumination source 116 and imaging using an optical sensor may be synchronized, such as by using at least one control unit of optoelectronic device 112.

[0194] The image generation unit 118 may be configured to image and / or record the pattern image and the flood image simultaneously or at different times.

[0195] Optoelectronic device 112 may be included in device 110. Device 110 may include at least one display 120, wherein the infrared light pattern, when emitted from pattern illumination source 114, passes through display 120, and / or the infrared flood light, when emitted from flood illumination source 116, passes through display 120. Alternatively or additionally, display 120 may be at least partially transparent in at least one continuous area covering pattern illumination source 114 and / or flood illumination source 116 and / or image generation unit 118.

[0196] Display 120 can be a device of any shape configured to display an information item. The information item can be any information, such as at least one image, at least one chart, at least one histogram, at least one graph, text, numbers, at least one symbol, an operation menu, etc. Display 120 can be or include at least one screen. Display 120 can have any shape, such as a rectangular shape. Display 120 can be a front-facing display of device 110.

[0197] The display 120 may be or may include at least one organic light emitting diode (OLED) display. The OLED display may be configured to emit visible light.

[0198] The display 120 is made of glass and / or may be covered by glass. In particular, the display 120 may include at least one glass cover plate.

[0199] Display 120 can be at least partially transparent. For example, display 120 can be semi-transparent in the near-infrared region. For example, display 120 can have a transparency of 20% to 50% in the near-infrared region. Display 120 can have different transparencies for other wavelength ranges. The present invention may provide an optoelectronic device 112 comprising an image generating unit 118 and two illumination sources 114 and 116 that can be positioned behind display 120 of device 110. Transparent regions of display 120 can allow operation of optoelectronic device 112 behind display 120. As described above, display 120 can be at least partially transparent. Display 120 can have a reduced pixel density and / or reduced pixel size and / or can include at least one transparent conductive path. The transparent regions of display 120 can have a pixel density of 360 to 440 pixels per inch (PPI). Other regions of display 120 (e.g., non-transparent regions) can have a higher pixel density, such as 460 to 500 PPI.

[0200] Display 120 includes a display area. The display area can be an active area of ​​display 120, particularly an activatable area. Display 120 can have additional areas, such as recesses or cutouts. Display 120 can be at least partially transparent in at least one continuous area, preferably in at least two continuous areas. At least one of the continuous areas can at least partially cover the image generation unit and / or pattern illumination source 114 and / or flood illumination source 116. Pattern illumination source 114, flood illumination source 116, and image generation unit 118 can be positioned in front of display 120 in the direction of propagation of the infrared light pattern.

[0201] Display 120 may have a first area associated with a first pixel density value and a second area associated with a second pixel density value. The first pixel density value may be lower than the second pixel density value. The first pixel density value may be equal to or lower than 450 PPI, preferably from 300 to 440 PPI, more preferably 350 to 450 PPI. The second pixel density value may be between 400 and 500 PPI, preferably 450 to 500 PPI. The first pixel density value is associated with at least one continuous area that is at least partially transparent. The first pixel density value may be lower than 350 and the second pixel density value may be equal to or greater than 400.

[0202] As mentioned above, existing displays that meet these requirements are less appealing to users, and there's a desire to reduce transparent areas. Furthermore, since the transmittance in the transparent areas is typically 15% to 50%, covering the image generation unit with such displays can result in low contrast. Contrast can be defined as the difference between the signal and the backlight. In this case, contrast can be considered reduced due to transmission. However, contrast can be defined as the ratio of signal to background. In this case, contrast may be reduced not only by transmission but also by diffraction in the projected spot pattern, as higher-order diffraction, in addition to the zeroth-order, also contributes additional intensity. Diffraction may also occur due to the structure of the display wiring, further reducing irradiance. Furthermore, the spot pattern projected through the display may produce a main spot corresponding to the pattern before passing through the display area. This main spot has reduced intensity due to the generation of higher-order spots. These two effects result in a reduction in irradiance to approximately 3%-5% of the initial irradiance and the presence of undesirable additional spots in the pattern. The present invention allows for sufficient irradiance in the pattern image, thereby improving contrast. Contrast can be increased by reducing the number of spots projected onto the user. This can result in an increase in the irradiance of the light spot, and therefore in an increase in contrast in the projected image of the light spot pattern. Furthermore, transparent areas in the display can be reduced. Using this optoelectronic device 112 can allow, for example, the camera to be repositioned from its outermost position on the display. This repositioning can optimize wiring, as it can optimize another location, namely the position of the image generation unit 118. This can reduce wiring usage or achieve improved battery operation.

[0203] Device 110 is configured to authenticate a user of device 110 in order to perform at least one operation requiring authentication on device 110. Authentication may include verifying the identity of the user. Specifically, authentication may include distinguishing the user from other humans or objects, and in particular distinguishing authorized access from unauthorized access. Authentication may include verifying the identity of the respective user and / or assigning an identity to the user. Authentication may include generating and / or providing identity information, for example, to another device or unit (e.g., to at least one authorization unit) for authorizing access to the device 110. The identity information may be proven by authentication. For example, the identity information may be and / or may include at least one identity token. If authentication is successful, the facial image recorded by image generation unit 118 may be verified as an image of the user's face, and / or the user's identity may be verified.

[0204] At least one authentication unit 122 may be used to perform user authentication. Authentication unit 122 may be configured to perform at least one user authentication process. Authentication unit 122 may include at least one processor. A processor may be any logic circuit configured to perform basic operations of a computer or system, and / or generally refers to a device configured to perform computational or logical operations. Specifically, a processor may be configured to process the basic instructions that drive a computer or system. As an example, a processor may include at least one arithmetic logic unit (ALU), at least one floating point unit (FPU), such as a math coprocessor or digital coprocessor, multiple registers, specifically registers configured to provide operands to the ALU and store operation results, and memory such as L1 and L2 cache memory. Specifically, the processor may be a multi-core processor. Specifically, the processor may be or include a central processing unit (CPU). Additionally or alternatively, the processor may be or include a microprocessor, and thus, specifically, the components of the processor may be contained within a single integrated circuit (IC) chip. Additionally or alternatively, the processor may be or may include one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs) and / or one or more tensor processing units (TPUs) and / or one or more chips, such as dedicated machine learning optimization chips, etc. The processor may be specifically configured, for example, through software programming, to perform one or more evaluation operations.

[0205] Authentication unit 122 is configured to identify a user based on the flood image. Identification may include assigning an identity to a detected face and / or at least one identity check and / or verifying the identity of the user. The authentication process may include multiple steps. For example, the authentication process may include performing at least one facial detection. The facial detection step may include analyzing the flood image. Additionally, for example, the authentication process may include recognition. Identification may include assigning an identity to a detected face and / or at least one identity check and / or verifying the identity of the user. Identification may include analyzing the flood image. Analysis of the flood image may include performing facial verification on the imaged face to confirm whether it is the face of the user. Identifying the user may include matching the flood image (e.g., showing a portion of the user, in particular, an outline of a portion of the user's face) with a template. Identifying the user may include determining whether the imaged face is the face of the user, in particular, determining whether the imaged face corresponds to at least one image of the user's face stored, for example, in at least one memory of the device.

[0206] Analysis of the flood image may include one or more of the following: filtering; selecting at least one region of interest; forming a difference image between the flood image and at least one offset; inverting the flood image; background correction; decomposition into color channels; decomposition into hue, saturation, and brightness channels; frequency decomposition; singular value decomposition; applying a Canny edge detector; applying a Laplacian of Gaussian filter; applying a Difference of Gaussian filter; applying a Sobel operator; applying a Laplacian operator; applying a Scharr operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying a Radon transform; applying a Hough transform; applying a wavelet transform; thresholding; creating a binary image. The region of interest may be manually determined by a user or may be automatically determined, such as by identifying a user within the image. In particular, analysis of the flood image may include using at least one image recognition technique, in particular a facial recognition technique. The image recognition technique includes at least one process for identifying a user within the image. Image recognition may include using at least one technique selected from the following: color-based image recognition, for example using features such as template matching; image segmentation and / or connected component analysis, for example using size or shape; machine learning and / or deep learning, for example using at least one convolutional neural network.

[0207] Analysis of the floodlight image may include determining a plurality of facial features. The analysis may include comparing the determined facial features with template features, in particular matching them. Template features may be features extracted from at least one template. The template may be or may include at least one image generated during a registration process (e.g., when initializing device 110). The template may be an image of an authorized user. Template features and / or facial features may include vectors. Matching of features may include determining a distance between vectors. Identifying the user may include comparing the distance of the vectors with at least one predefined limit, wherein the user is successfully identified if the distance is at least within a tolerance ≤ the predefined limit. In other cases, the user is declined and / or rejected.

[0208] For example, image recognition can include using at least one model, in particular a trained model including at least one facial recognition model. Analysis of the flood image can be performed using a facial recognition system, such as FaceNet, as described, for example, in Florian Schroff, Dmitry Kalenichenko, and James Philbin, “FaceNet: A Unified Embedding for face Recognition and Clustering,” arXiv:1503.03832. The trained model can include at least one convolutional neural network. For example, the convolutional neural network can be designed as described in MD Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” CoRR, abs / 1311.2901, 2013; or C. Szegedy et al., “Going deeper with convolutions,” CoRR, abs / 1409.4842, 2014. For more details on convolutional neural networks for facial recognition systems, see Florian Schroff, Dmitry Kalenichenko, and James Philbin, “FaceNet: A Unified Embedding for face Recognition and Clustering,” arXiv:1503.03832. Labeled image data from an image database can be used as training data.In particular, labeled faces from one or more of GB Huang, M. Ramesh, T. Berg, and E. Learned-Miller, "Labeled faces in the wild: A database for studying face recognition in unconstrained environments," Technical Report 07-49, University of Massachusetts Amherst, October 2007; the Youtube® Faces database as described in L. Wolf, T. Hassner, and I. Maoz, "Face recognition in unconstrained videos with matchedbackground similarity," IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2011; or the Google® Facial Expression Comparison Dataset can be used. Convolutional neural networks can be trained as described in Florian Schroff, Dmitry Kalenichenko, James Philbin, “FaceNet: A Unified Embedding for face Recognition and Clustering,” arXiv:1503.03832.

[0209] The authentication unit 122 is configured to use a facial recognition authentication process operating on the pattern image, the flood image and / or the extracted material data.The authentication unit 122 may be configured to extract the material data from the pattern image.

[0210] The authentication unit 122 can be configured to extract material data from the pattern image by analyzing the beam profile of the light spot. Regarding beam profile analysis, reference is made to WO 2018 / 091649 A1, WO 2018 / 091638 A1, and WO 2018 / 091640 A1, the entire contents of which are incorporated herein by reference. Beam profile analysis can allow for reliable classification of a scene based on several light spots. Each light spot in the pattern image can include a beam profile. Extracting material data based on the pattern image can be performed using at least one model.

[0211] The authentication process may include verification based on the extracted material data. Verification based on the extracted material data may include determining whether the extracted material data corresponds to expected material data. Determining whether the extracted material data matches the expected material data may be referred to as verification. Allowing or denying the user and / or subject from performing at least one operation requiring authentication on device 110 based on the material data may include verifying the authentication or authentication process. Verification may be based on the material data and / or an image. Determining whether the extracted material data corresponds to the expected material data may include determining a similarity between the extracted material data and the expected material data. Determining the similarity between the extracted material data and the expected material data may include comparing the extracted material data with the expected material data. The expected material data may refer to predetermined material data. In an example, the expected material data may be skin. Determining whether the material data corresponds to the expected material data may include comparing the material data with the expected material data. Comparing the material data with the expected material data may result in allowing or / and denying the user and / or subject from performing the at least one operation requiring authentication. In an example, skin as the expected material data may be compared to non-skin materials or silicon as the material data, and the results may be biased because silicon or non-skin materials may differ from skin. In an embodiment, the authentication process or verification thereof may include generating at least one feature vector from the material data and matching the material feature vector to an associated material reference template vector.

[0212] The authentication unit 122 may be configured to authenticate the user if the user can be identified and / or if the material data matches the expected material data. The device 110 may include at least one authorization unit 124 configured to allow the user to perform at least one operation on the device 110, for example, unlocking the device 110 if the user is successfully authenticated, or denying the user from performing at least one operation on the device 110 if the authentication is unsuccessful.

[0213] Figure 2 An exemplary embodiment of a method for authenticating a user of device 110 to perform at least one operation on device 110 that requires authentication is shown.

[0214] The method includes:

[0215] a. (reference numeral 126 ) illuminating the user with at least one infrared light pattern from at least one pattern illumination source 114 of the device 110 , in particular wherein the number of infrared light spots is less than or equal to 4000 light spots,

[0216] b. (reference numeral 128) illuminating the user with infrared flood light from at least one flood illumination source 116 of the device 110,

[0217] c. (reference numeral 130) generating, with the image generation unit 118 of the device 110, at least one pattern image showing the user, in particular at least a portion of the user's face, when illuminating the user with the infrared light pattern, and generating, with the image generation unit 118 of the device, at least one image showing the user when illuminating the user with the infrared flood light, wherein the display 120 of the device 110 is at least partially transparent in at least one continuous area covering the pattern illumination source 114, the flood illumination source 116, and / or the image generation unit 118,

[0218] d. (reference numeral 132 ) identifying the user based on the flood image by using at least one authentication unit 122 of the device 110 ,

[0219] e. (reference numeral 134) extracting material data from the at least one pattern image by using the authentication unit 122; and

[0220] f. (reference numeral 136 ) allowing the user to perform at least one operation requiring authentication on the device 110 based on the material data and the identification.

[0221] These method steps may be performed in the order given or in a different order. Furthermore, there may be one or more additional method steps not listed. Furthermore, one, more than one, or even all of the method steps may be repeated. The pattern image and / or image showing the user when illuminated with infrared flood light may show at least a portion of the user's face.

[0222] The method may be computer-implemented.

[0223] List of Reference Numerals

[0224] 110 equipment

[0225] 112 Photoelectric Device

[0226] 114 pattern illumination source

[0227] 116 floodlight source

[0228] 118 Image Generation Unit

[0229] 120 monitors

[0230] 122 certification units

[0231] 124 authorized units

[0232] 126 Illuminate the user with an infrared light pattern

[0233] 128 Illuminate the user with infrared floodlight

[0234] 130 Generate at least one pattern image and generate at least one flood image

[0235] 132 Identify User

[0236] 134 Extracting Material Data

[0237] 136 allowed

Claims

1. A photoelectric device (112), comprising: - at least one patterned illumination source (114), the at least one patterned illumination source being configured to emit at least one infrared light pattern comprising a plurality of infrared light spots, wherein the number of the infrared light spots is less than or equal to 4000 spots, - at least one flood illumination source (116) configured to emit infrared flood light, - at least one image generation unit (118) configured to generate at least one pattern image when the pattern illumination source (114) emits an infrared light pattern, and configured to generate at least one flood image when the flood illumination source (116) emits infrared flood light.

2. The optoelectronic device (112) according to claim 1, wherein The optoelectronic device (112) is included in a device (110), wherein the device (110) includes at least one display (120), wherein the infrared light pattern passes through the display (120) when emitted from the pattern illumination source (114) and / or the infrared flood light passes through the display (120) when emitted from the flood illumination source (116), wherein the display (120) is at least partially transparent in at least one continuous area covering the pattern illumination source (114), the flood illumination source (116) and / or the image generation unit (118).

3. The optoelectronic device (112) according to any one of claims 1 or 2, wherein: The infrared light pattern includes equal to or less than 3000 light spots, preferably equal to or less than 2000 light spots.

4. The optoelectronic device (112) according to any one of claims 1 to 3, wherein: This infrared light is coherent.

5. The optoelectronic device (112) according to any one of claims 1 to 4, wherein: The infrared light pattern is a hexagonal pattern.

6. The optoelectronic device (112) according to any one of claims 1 to 5, wherein: At least one of the infrared light spots is associated with a beam divergence of 0.2° to 0.5°, preferably 0.1° to 0.3°.

7. The optoelectronic device (112) according to any one of claims 1 to 6, wherein: The image generation unit (118) has a field of view between 10°×10° and 75°×75°.

8. The optoelectronic device (112) according to any one of claims 1 or 7, wherein: The pattern illumination source (114) includes at least one vertical cavity surface emitting laser (VCSEL), preferably a plurality of VCSELs, and / or wherein the pattern illumination source includes at least one optical element configured to increase the number of light spots, wherein the at least one optical element includes at least one diffractive optical element and / or at least one metasurface element.

9. Use of an optoelectronic device (112) according to any one of claims 1 to 8 for authenticating a user of an apparatus (110) comprising the device.

10. A device (110) for authenticating a user of a device (110) to perform at least one operation requiring authentication on the device (110), the device (110) comprising: - at least one flood illumination source (116) configured to emit infrared flood light; - at least one patterned illumination source (114), the at least one patterned illumination source being configured to emit at least one infrared light pattern comprising a plurality of infrared light spots, wherein the number of the infrared light spots is less than 4000 spots; - at least one image generation unit (118), the at least one image generation unit being configured to generate at least one pattern image when the pattern illumination source (114) emits an infrared light pattern, and being configured to generate at least one flood image when the flood illumination source (116) emits an infrared flood light; at least one display (120), wherein the infrared light pattern passes through the display (120) when illuminated from the pattern illumination source (114) and / or the infrared flood light passes through the display (120) when illuminated from the flood illumination source (116), wherein the display (120) of the device (110) is at least partially transparent in at least one continuous area covering the pattern illumination source (114), the flood illumination source (116) and / or the image generation unit (118), - at least one authentication unit (122), the at least one authentication unit being configured to perform at least one authentication process of a user using the flood image and the pattern image.

11. The device (110) according to claim 2 or 10, wherein The display (120) has a first area associated with a first pixel per inch (PPI) value and a second area associated with a second PPI value, wherein the first PPI value is lower than the second PPI value.

12. The device (110) according to the preceding claim, wherein The first PPI value is less than 350, and wherein, The second PPI value is equal to or greater than 400.

13. A method for authenticating a user of a device (110) to perform at least one operation requiring authentication on the device (110), the device (110) comprising a display (120), and the method comprising: a. illuminating the user with at least one infrared light pattern from at least one pattern illumination source (114) of the device (110), b. illuminating the user with infrared flood light from at least one flood illumination source (116) of the device (110), c. generating, using the image generation unit (118) of the device (110), at least one pattern image showing the user when illuminating the user with the infrared light pattern, and generating, using the image generation unit (118) of the device (110), at least one image showing the user when illuminating the user with the infrared flood light, wherein the display (120) of the device (110) is at least partially transparent in at least one continuous area covering the pattern illumination source (114), the flood illumination source (116) and / or the image generation unit (118), d. identifying the user based on the flood image by using at least one authentication unit (122) of the device (110), e. extracting material data from the at least one pattern image by using the authentication unit (122); and f. allowing the user to perform at least one operation requiring authentication on the device (110) based on the material data and the identification.

14. A computer program comprising instructions which, when executed by a device (110) according to any of the preceding claims relating to a device (110), cause the device (110) to perform a method according to any of the preceding claims relating to a method.

15. A computer-readable storage medium comprising instructions which, when executed by a device (110) according to any one of the preceding claims relating to a device (110), cause the device (110) to perform a method according to any one of the preceding claims relating to a method.

16. A non-transitory computer readable medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of the preceding method-related claims.

Images