Optical element on floodlight VCSEL for two-in-one projector

Abstract

It is proposed an optoelectronic device (112) configured for emitting at least one infrared light pattern comprising a plurality of infrared light beams and for emitting infrared flood light, the optoelectronic device comprising:-a light emitter structure (114) comprising a plurality of light emitters (116), a first array (118) of light emitters (116) of the plurality of light emitters (116) forms a pattern illumination source (120) configured to emit the infrared light pattern, and wherein a second array (122) of light emitters (116) of the plurality of light emitters (116) different from the light emitters (116) of the first array (118) forms a flood illumination source (124) configured to emit the infrared flood; -a base (125) providing a single plane for mounting the light emitters (116); -at least one optical element system (126) comprising a plurality of optical elements, where the optical element system (126) is configured for focusing the emitted infrared light pattern onto a focal plane, where the optical element system (126) covers the light emitter structure (114); -at least one floodlight optical element (130) configured for defocusing light emitted by the light emitter (116) of the floodlight illumination source (124), thereby forming an overlapping light spot, where the floodlight optical element (130) is configured for unaffecting the emitted infrared light pattern.

Description

Technical Field

[0001] This invention relates to an optoelectronic device, the use of the optoelectronic device, a device for authenticating a user, and a method for authenticating a user of the device. The invention further relates to a computer program, a computer-readable storage medium, and a non-transient computer-readable medium. The device, method, and use according to the invention can be specifically used, for example, in various fields such as daily life, security technology, gaming, transportation technology, production technology, photography (e.g., digital or video photography for artistic, documentation, or technical purposes), safety technology, information technology, agriculture, crop protection, maintenance, cosmetics, and medical technology, or in science. However, other applications are also possible. Background Technology

[0002] Available authentication systems in mobile devices (such as smartphones, tablets, etc.) include cameras. These mobile devices typically have a front-facing display, such as an organic light-emitting diode (OLED) area. For camera integration, a cutout needs to be made in the display at the camera's location. These cutouts reduce the usable display area, a phenomenon known as a notch, thus reducing the display area available to the user. Therefore, secure biometric facial authentication that operates behind the display panel and is invisible (avoiding notches, perforations, etc.) is desirable to maximize the accessible display area. The use of OLED displays offers the possibility of enabling operation behind the display by introducing semi-transparent areas into the display. Optical components (such as light projectors and cameras) can then be placed behind these semi-transparent areas. However, the number and size of these transparent areas need to be limited to the smallest possible number to maintain a homogeneous display appearance. For example, standard components for facial authentication solutions involving 3D and / or material detection involve a flood projector, a dot projector, and a camera. Combining the flood projector and the dot projector into a housing behind a single semi-transparent area (a 2-in-1 concept) reduces the necessary number of semi-transparent areas from three to two.

[0003] A common approach to achieving an integrated projector solution (the 2-in-1 concept) is to place two VCSEL arrays adjacent to each other, one for flood illumination and the other for point projection. For point projection, the point projection VCSEL is positioned within the focal plane of the projector optics for sharp projection. For flood projection, the flood projection VCSEL is placed off-focus to blur the points, resulting in a homogeneous superposition of light spots in the object space. The drawbacks of this concept are twofold: because the two VCSELs will be offset relative to the system's optical axis, their illumination fields will not coincide. To correct this offset, additional optics, such as a DOE, must be used. The two independent VCSELs create a large emission area, which necessitates a large transparent area to accommodate the large optical aperture of such a projector.

[0004] WO 2021 / 259923A1 describes a projector and illumination module configured for scene illumination and pattern projection. The projector and illumination module include at least one array of multiple individual emitters and at least one optical system. Each of the individual emitters is configured to generate at least one illumination beam. The optical system includes at least one array of multiple transmission devices. The transmission device array includes at least one transmission device for each of the individual emitters. The transmission device array includes at least two sets of transmission devices. The two sets of transmission devices differ in at least one characteristic. The transmission devices in one set are configured to generate at least one illumination pattern in response to an illumination beam illuminating the transmission device. The transmission devices in the other set are configured to generate a diverging beam in response to an illumination beam illuminating the transmission device.

[0005] US 11,710,945 describes an optoelectronic device comprising: a heat sink shaped to define a base; a first platform located at a first height above the base; and a second platform located at a second height above the base, the second height being different from the first height. A first monolithic emitter array is mounted on the first platform and configured to emit a first beam. A second monolithic emitter array is mounted on the second platform and configured to emit a second beam. Optical elements are configured to guide both the first and second beams toward a target region.

[0006] US 2023 / 220974 describes an optical device including an array of light-emitting elements, comprising a first subset of light-emitting elements and a second subset of light-emitting elements. The first subset of light-emitting elements is configured to emit light having a wavelength L1. The device includes: a high-refractive-index material selectively disposed on the second subset of light-emitting elements; and an array of optical elements positioned to be illuminated by the first and second subsets of light-emitting elements. The optical elements are regularly arranged at a spacing P in a common plane located at a distance D relative to the array of light-emitting elements.

[0007] P 2 ≈2L1 D / N, where N is an integer greater than or equal to 1. The problem to be solved

[0008] Therefore, the object of the present invention is to provide an apparatus and method that addresses the aforementioned technical challenges posed by known devices and methods. Specifically, the object of the present invention is to provide an apparatus and method that allows the generation of dot patterns and diffuse illumination using only a single projector to save hardware space. Summary of the Invention

[0009] This problem is solved by a photoelectric device having the features of the independent claims, the use of the photoelectric device, a device for authenticating a user, a method for authenticating a user of the device, a computer program, a computer-readable storage medium, and a non-transient computer-readable medium. Advantageous embodiments that can be implemented independently or in any arbitrary combination are set forth in the dependent claims and throughout the specification.

[0010] In a first aspect, a photoelectric device is disclosed, configured to emit at least one infrared light pattern comprising a plurality of infrared beams and to emit infrared floodlight. The photoelectric device includes:

[0011] - A light emitter structure comprising a plurality of light emitters, wherein a first array of the light emitters is configured to form a pattern illumination source for emitting the infrared light pattern, and wherein a second array of the light emitters, which is different from the light emitters in the first array, is configured to form a flood illumination source for emitting the infrared floodlight.

[0012] - Base, which provides a single plane for mounting the light emitter;

[0013] - At least one optical element system, the at least one optical element system comprising a plurality of optical elements, wherein the optical element system is configured to focus the emitted infrared light pattern onto a focal plane, wherein the optical element system covers a light emitter structure;

[0014] - At least one floodlight optical element, the at least one floodlight optical element being configured to defocus light emitted by a light emitter of a floodlight illumination source, thereby forming an overlapping light spot, wherein the floodlight optical element is configured to ensure that the emitted infrared light pattern is unaffected.

[0015] As used herein, the term "optoelectronic device" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, devices or systems that operate on light and electricity. An optoelectronic device may be a light projector.

[0016] As used herein, the term "light" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, electromagnetic radiation in one or more of the infrared, visible, and ultraviolet spectral ranges. In this document, the term "ultraviolet spectral range" generally refers to electromagnetic radiation with wavelengths from 1 nm to 380 nm, preferably from 100 nm to 380 nm. Further, in part according to the standard ISO-21348, the effective version of this document as of the date of this document, the term "visible spectral range" generally refers to the spectral range from 380 nm to 760 nm. The term "infrared spectral range" (IR) generally refers to electromagnetic radiation from 760 nm to 1000 µm, wherein the range from 760 nm to 1.5 µm is generally referred to as the "near-infrared spectral range" (NIR), the range from 1.5 µm to 15 µm is referred to as the "mid-infrared spectral range" (MidIR), and the range from 15 µm to 1000 µm is referred to as the "far-infrared spectral range" (FIR). Preferably, the light used for the typical purposes of this invention is light in the infrared (IR) spectral range, more preferably light in the near-infrared (NIR) and / or mid-infrared spectral range (MidIR), especially light with a wavelength of 1 µm to 5 µm, preferably 1 µm to 3 µm.

[0017] The optoelectronic device can be configured to emit light of a single wavelength, such as in the near-infrared region. In other embodiments, the optoelectronic device can be adapted to emit light with multiple wavelengths, for example, to allow for additional measurements in other wavelength channels.

[0018] As used herein, the term "ray" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom-defined meaning. Specifically, the term may refer to, but is not limited to, a line perpendicular to the wavefront of light and pointing in the direction of energy flow. As used herein, the term "beam" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom-defined meaning. Specifically, the term may refer to, but is not limited to, a collection of rays. In the following text, the terms "ray" and "beam" will be used as synonyms. As used herein, the term "beam" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom-defined meaning. Specifically, the term may refer to, but is not limited to, a quantity of light, specifically a quantity of light traveling substantially in the same direction, including the possibility that the beam has an extension angle or widening angle.

[0019] As used herein, the term "light pattern" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, at least one arbitrary pattern comprising, in particular, coherent electromagnetic radiation (e.g., at least two beams, preferably at least two beams). The light pattern may be projected onto a user and / or object. Projecting the beams of the light pattern onto a surface can produce a light spot. The beams may illuminate at least a portion of the surface. The light spot may refer to a continuous region of coherent electromagnetic radiation on at least a portion of the surface. The light spot may refer to a coherent electromagnetic radiation spot of any shape. The light spot may be the result of projecting beams associated with the light pattern. The light spot may extend at least partially in space. The emitted light pattern can illuminate the surface by a light pattern comprising multiple light spots. The light spots may at least partially overlap. For example, the number of light spots may be equal to the number of beams associated with the emitted light pattern. The intensities associated with the light spots may be substantially similar. "Substantially similar" may mean that the intensity values ​​associated with the light spots may differ by less than 50%, preferably less than 30%, more preferably less than 20%. Using patterned light can be advantageous because it allows light to be avoided in light-sensitive areas such as the eyes.

[0020] As used herein, the term "infrared light pattern" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, light patterns encompassing beams within the infrared spectral range. Infrared light patterns may have wavelengths from 760 nm to 1.5 μm, preferably 940 nm, 1140 nm, or >1400 nm.

[0021] Infrared light patterns can be coherent. As used herein, the term "coherent" electromagnetic radiation is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term can refer to, but is not limited to, light patterns and / or multiple beams of light whose electric field values ​​at different locations and / or different times have at least substantially a fixed phase relationship. In particular, coherent electromagnetic radiation can refer to electromagnetic radiation capable of exhibiting interference effects. The term "coherent" can also include partial coherence, i.e., imperfect correlation between phase values. Electromagnetic radiation can be perfectly coherent, wherein a deviation of approximately ±10% in the phase relationship is possible.

[0022] Infrared patterns may include at least one dot pattern.

[0023] As used herein, the term "floodlight" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, substantially continuous spatial illumination, particularly diffuse and / or uniform illumination. Floodlight has wavelengths in the infrared range, particularly in the near-infrared range. As used herein, the term "substantially continuous spatial illumination" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, uniform spatial illumination, wherein non-uniform areas are possible. The area illuminated by floodlight (e.g., covering a user, a portion of a user, and / or the user's face) can be continuous. Power can be distributed across the entire illumination field. In contrast, illumination provided by a light pattern can include at least two continuous areas, particularly multiple continuous areas, and / or power can be concentrated in a smaller area (compared to the entire illumination field). Infrared floodlight illumination can be suitable for illuminating continuous areas, particularly a single continuous area. Infrared pattern illumination can be suitable for illuminating at least two continuous areas. A floodlight source can illuminate a measurement area, such as a user, a portion of the user, and / or the user's face, with a substantially constant intensity. As used herein, the term "constant" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, the temporal aspect during the exposure time. Floodlight can vary over time and / or can be substantially constant over time. As used herein, the term "substantially constant" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, completely constant illumination, and embodiments that allow deviations from constant illumination of ≤ ± 10%, preferably ≤ ± 5%, more preferably ≤ ± 2%.

[0024] As used herein, the term "base" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, a carrier on which at least one additional element, particularly a light emitter structure, can be mounted. The base may include multiple cavities into which the light emitter structure can be mounted. The base may have any shape, such as rectangular, circular, or hexagonal. The shape may refer to the side of the base oriented perpendicular to the measured height. The base may include at least one semiconductor substrate. The base may be an element and / or additional element of the light emitter structure. The base may be and / or may include a thermally conductive printed circuit board (PCB).

[0025] A light emitter structure can be formed into a light emitter chip, such as a VCSEL die, for example, a die sawn from a wafer. This light emitter chip can be mounted on a base, for example, using at least one thermally conductive adhesive. The base may include at least one thermally conductive material. The base can be the bottom of an optoelectronic device, such as the bottom of the housing of an optoelectronic device. Therefore, the dimensions of the base can be defined by the dimensions of the optics and the housing. Alternatively, the base and the housing can be separate components. For example, the light emitter chip can be mounted on a base (e.g., a PCB) for example, using at least one thermally conductive adhesive, and the housing can be applied to the combined component.

[0026] A light emitter in a light emitter structure can be mounted on a base, for example, using at least one adhesive or at least one glue. For example, the adhesive can be at least one thermally conductive adhesive. For example, a light emitter forming a patterned illumination source can be glued to the base. For example, a light emitter forming a floodlight illumination source can be glued to the base.

[0027] The base may include at least one thermally conductive material, particularly a thermally conductive material. The thermally conductive material may be configured as a heat exchanger. The thermally conductive material may be configured to regulate the temperature of the light emitter. The thermally conductive material may be configured to transfer heat generated by the light emitter out of the light emitter. For example, the thermally conductive material may include at least one composite material. The light emitter structure may be mounted on the thermally conductive material.

[0028] As used herein, the term "plane" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, the surface of a base. The surface may be continuous. The plane may be a flat surface. The plane may be designed without curvature and / or steps. As used herein, the term "providing" a single plane is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, one or more of the following: a surface that includes, has, can be used as, or serves as a light emitter structure on which it may be mounted.

[0029] As used herein, the term "light emitter structure" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, an assembly of at least four light emitters. An light emitter structure includes multiple light emitters. As used herein, the term "light emitter" (also simply "emitter") is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, at least one arbitrary device configured to provide at least one light beam. The light beam may generate an infrared light pattern.

[0030] Each of these optical emitters may include at least one vertical-cavity surface-emitting laser (VCSEL). As used herein, the term "vertical-cavity surface-emitting laser" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, a semiconductor laser diode configured to emit a laser beam perpendicularly to its top surface. Examples of VCSELs can be found, for example, at en.wikipedia.org / wiki / Verticalcavity_surface-emitting_laser. The optical emitter may be configured to emit light in the near-infrared spectral range, preferably with wavelengths from 760 nm to 1.5 µm, preferably 940 nm, 1140 nm, or > 1400 nm.

[0031] As used herein, the term "irradiation" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, the process of exposing at least one element to light. As used herein, the term "irradiation source" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, any device configured to generate or provide light as defined above. As used herein, the term "pattern irradiation source" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, any device configured to generate or provide at least one light pattern, particularly at least one infrared light pattern. As used herein, the term "floodlight irradiation source" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, at least one arbitrary device configured to provide floodlight.

[0032] As used herein, the term "array" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, a one-dimensional (e.g., row) array or a two-dimensional array, particularly in a matrix having m rows and n columns (where m and n are each positive integers). The light emitters of an optical emitter structure can be arranged in a periodic pattern. The light emitters of an optical emitter structure can be arranged in one or more of the following: a grid pattern, a hexagonal pattern, a shifted hexagonal pattern, etc. Multiple light emitters of an optical emitter structure can form a first array of light emitters, and multiple light emitters in the structure that are different from the light emitters of the first array can form a second array of light emitters. An optical emitter structure may include two light emitter arrays, such as two VCSEL arrays. These arrays are located in a plane. For example, the first array and the second array of emitters are produced directly on the plane as a single die, or the first array and the second array of emitters are produced separately and, for example, mounted side-by-side on the plane. However, it is possible to use even more arrays (e.g., configured to provide different functions).

[0033] For example, the first and second arrays can be arranged side-by-side, particularly adjacent to each other, on a plane. For instance, the plane may include the first array first, followed by the second array, along a direction perpendicular to the optical axis of the photoelectric device. However, other arrangements are also possible.

[0034] For example, the cavities of the first array of light emitters and the second array of light emitters can form a combined pattern, wherein the cavities of the first array of light emitters and the second array of light emitters alternate, for example, row by row or line by line.

[0035] The light emitters of the patterned illumination source and the floodlight illumination source can be activated at different times.

[0036] As used herein, the term "system" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, any set of interacting or interdependent components forming a whole. Specifically, these components may interact with each other to achieve at least one common function. At least two components may be processed independently, or may be coupled or connectable. As used herein, the term "optical element system" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, a system comprising at least two optical elements. An optical element system may include one or more of the following: at least one refractive lens, multiple refractive lenses; at least one diffractive optical element (DOE), multiple DOEs, multiple metalenses. For example, an optical element system may include at least one refractive lens and at least one optical element configured to increase (e.g., replicate) the number of light spots (e.g., light spots generated by a light emitter from a patterned illumination source). An optical element system may include at least one diffractive optical element (DOE) and / or at least one metasurface element. DOEs and / or metasurface elements can be configured to generate multiple beams from a single incident beam. For example, a VCSEL projecting up to 2000 beams and an optical element comprising multiple metasurface elements can be used to replicate the number of beams. Further arrangements (specifically including different numbers of projecting VCSELs and / or at least one different optical element configured to increase the number of beams) are possible. Other multiplication factors are also possible. For example, one or more VCSELs can be used, and the generated laser beams can be replicated using at least one DOE.

[0037] The optical element system covers the light emitter structure. As used herein, the term "covers the light emitter structure" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, completely or at least partially covering the light emitting structure, such as at least covering the first array of light emitters. The optical element system can be designed and / or arranged such that it covers the light emitter structure. For example, the first array may be covered only by the optical element system. For example, both the first and second arrays may be covered by the optical element system. The light emitter structure may be located at the focal point of the optical element system. This arrangement allows the emitted light from the light emitter, particularly from a patterned illumination source, to be collimated.

[0038] As used herein, the term "floodlight optical element" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, an optical element assigned to a floodlight source and configured to defocus light emitted by the light emitter of the floodlight source to form overlapping light spots. A floodlight source is configured, in combination with a floodlight optical element, to generate floodlight, particularly diffuse illumination. A floodlight optical element may include at least one element selected from the group consisting of: at least one plate with a refractive index greater than 1.4, such as a glass plate; at least one diffuser plate; at least one lens; at least one microlens; at least one prism; at least one Fresnel lens; at least one diffractive optical element (DOE); at least one superlens. A second array may be completely covered by a floodlight optical element. For example, an optoelectronic device may include a single floodlight optical element covering all light emitters of the second array. For example, an optoelectronic device may include multiple floodlight optical elements. For example, each light emitter in the second array may include at least one assigned floodlight optical element. As described above, the second array may be located at the focal point of the optical element system. Therefore, the light generated by the second array will also be focused. However, the optical imaging is altered by the additional floodlight optics, causing the cavity to be improperly collimated. This allows for the generation of diffuse floodlight illumination.

[0039] The floodlight optics are configured to ensure that the emitted infrared light pattern remains unaffected. As used herein, the term "unaffected" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, the fact that the floodlight optics are arranged and / or designed such that light generated by the pattern illumination source does not interact with the floodlight optics. For example, the floodlight optics may be arranged and / or designed such that the pattern illumination source is omitted and / or excluded from the coverage of the floodlight optics. In particular, the pattern illumination source is not covered by the floodlight optics.

[0040] For example, floodlight optical elements can be mounted on a base, for example, using at least one adhesive or at least one glue. For example, floodlight optical elements can be mounted on a base and / or a light emitter, particularly on the light emitter of a floodlight source, for example, using at least one adhesive or at least one glue.

[0041] Optoelectronic devices may be included in the device. The device may be selected from the group consisting of: television equipment; game consoles; personal computers; mobile devices, especially mobile phones, and / or smartphones, and / or tablet computers, and / or laptop computers, and / or tablet computers, and / or virtual reality devices, and / or wearable devices such as smartwatches; or other types of portable computers.

[0042] The device may include a display, wherein an infrared light pattern passes through the display when illuminated from a pattern illumination source, and / or an infrared floodlight passes through the display when illuminated from a floodlight illumination source. As used herein, the term "display" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, a device of any shape configured to display information items. The information items may be any information, such as at least one image, at least one chart, at least one histogram, at least one graphic, text, numbers, at least one symbol, operation menus, etc. The display may be or may include at least one screen. The display may have any shape, such as a rectangular shape. The display may be a front-facing display of the device. The display may be or may include at least one organic light-emitting diode (OLED) display. As used herein, the term "organic light-emitting diode" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, a light-emitting diode (LED), wherein the emitting electroluminescent layer is an organic compound film configured to emit light in response to an electric current. OLED displays may be configured to emit visible light.

[0043] The display, particularly the display area, may be made of glass and / or covered by glass. In particular, the display may include at least one glass cover.

[0044] The display is at least partially transparent. As used herein, the term "at least partially transparent" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, the property of a display that allows light, particularly light of a specific wavelength range (e.g., light in the infrared spectral region, particularly light in the near-infrared spectral region), to pass through at least partially. For example, the display may be translucent in the near-infrared region. For example, the display may have 20% to 50% transparency in the near-infrared region. The display may have different transparency for other wavelength ranges. The present invention may propose an optoelectronic device comprising an image generating unit and two illumination sources that can be placed behind the display of the device. The transparent areas(s) of the display may allow the optoelectronic device to operate behind the display. As described above, the display is at least partially transparent. The display may have a reduced pixel density and / or a reduced pixel size and / or may include at least one transparent conductive path. The transparent areas(s) of the display may have a pixel density of 360 to 440 PPI (pixels per inch). Other areas of the display (e.g., non-transparent areas) may have a pixel density higher than 400 PPI, such as 460 to 500 PPI.

[0045] A display may include a display area. As used herein, the term "display area" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, the effective area of ​​the display, particularly the activatable area. The display may have additional areas, such as recesses or cutouts. The display may be at least partially transparent in at least one consecutive area, preferably in at least two consecutive areas. At least one of the consecutive areas at least partially covers the image generating unit and / or the pattern illumination source and / or the flood illumination source. The pattern illumination source, flood illumination source, and image generating unit may be positioned in front of the display in the direction of propagation of the infrared light pattern.

[0046] The display may have a first region associated with a first pixel density (pixels per inch (PPI)) value and a second region associated with a second pixel density value. The first pixel density value may be lower than the second pixel density value. The first pixel density value may be equal to or lower than 450 PPI, preferably from 300 to 440 PPI, more preferably from 350 to 450 PPI. The second pixel density value may be from 400 to 500 PPI, preferably from 450 to 500 PPI. The first pixel density value may be associated with at least one continuous region that is at least partially transparent.

[0047] The device may include at least one image generation unit configured to generate at least one pattern image when a pattern illumination source emits an infrared light pattern, and configured to generate at least one floodlight image when a floodlight illumination source emits infrared floodlight.

[0048] As used herein, the term "image generation unit" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, at least one unit of an optoelectronic device configured to generate at least one image. The image may be generated via a hardware and / or software interface, which may be considered the image generation unit. As used herein, the term "image generation" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, capturing and / or generating and / or determining and / or recording at least one image using an image generation unit. Image generation may include imaging and / or recording images. Image generation may include capturing a single image and / or multiple images, such as an image sequence. For the purpose of generating images via a hardware and / or software interface, the capture and / or generation and / or determination and / or recording of images may be caused and / or initiated by the hardware and / or software interface. For example, image generation may include continuously recording a sequence of images, such as video or film. Image generation can be initiated by user action or can be initiated automatically, for example, when at least one object or user is automatically detected within the field of view of the image generation unit and / or a predetermined area of ​​the field of view.

[0049] The image generation unit may include at least one optical sensor, particularly at least one pixelated optical sensor. The image generation unit may include at least one CMOS sensor or at least one CCD chip. For example, the image generation unit may include at least one CMOS sensor that may be sensitive in the infrared spectral range. As used herein, the term "image" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, data recorded using an optical sensor, such as multiple electronic readings from a CMOS or CCD chip. An image may include raw image data or may be a preprocessed image. For example, preprocessing may include applying at least one filter and / or at least one background correction and / or at least one background subtraction to the raw image data.

[0050] For example, the image generation unit may include one or more of the following: at least one monochrome camera (e.g., including monochrome pixels), at least one color (e.g., RGB) camera (e.g., including color pixels), and at least one IR camera. The camera may be a CMOS camera. The camera may include at least one monochrome camera chip, such as a CMOS chip. The camera may include at least one color camera chip, such as an RGB CMOS chip. The camera may include at least one IR camera chip, such as an IR CMOS chip. For example, the camera may include monochrome (e.g., black and white) pixels and color pixels. Color pixels and monochrome pixels may be combined internally within the camera. The camera typically includes a one-dimensional or two-dimensional array of image sensors (e.g., pixels).

[0051] As described above, the image generation unit can be at least one camera. For example, the camera can be an internal camera and / or an external camera of a device that includes an optoelectronic device. As described above, the internal camera and / or external camera of the device can be accessed via a hardware and / or software interface included in the optoelectronic device that serves as the image generation unit. In the case where the device is or includes a smartphone, the image generation unit can be the smartphone's front-facing camera (e.g., a selfie camera) and / or rear-facing camera.

[0052] The image generation unit may have a field of view between 10° × 10° and 75° × 75°, preferably 55° × 65°. The image generation unit may have a resolution of less than 2 MP, preferably between 0.3 MP and 1.5 MP.

[0053] The image generation unit may include additional components, such as one or more optical elements, like one or more lenses. As an example, the optical sensor may be a fixed-focus camera, where at least one lens is fixed relative to the camera's adjustment. Alternatively, the camera may include one or more variable lenses that can be adjusted automatically or manually. The camera may include at least one optical filter, such as at least one bandpass filter. The bandpass filter may be matched to the spectrum of the light emitter. However, other cameras are also feasible.

[0054] The light emitters of the pattern illumination source and the flood illumination source can be arranged at the optical center of the image generation unit. This ensures excellent matching of the illumination field (FoI) of the pattern projection and the flood illumination projection. This allows for minimizing the size of the transparent area within the display.

[0055] As used herein, the term "pattern image" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, an image generated by an image generation unit while the image is illuminated (e.g., on an object and / or a user) with an infrared light pattern. A pattern image may include an image showing at least a portion of the user, particularly the user's face, particularly in a corresponding region of interest included in the image when the user is illuminated with an infrared light pattern. A pattern image can be generated by imaging and / or recording light reflected from an object and / or user illuminated by an infrared light pattern. A pattern image showing the user may include at least a portion of the illuminated infrared light pattern on at least a portion of the user. For example, illumination from the pattern source and imaging using an optical sensor may be synchronized, for example, by using at least one control unit of an optoelectronic device.

[0056] As used herein, the term "floodlight image" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, an image generated by an image generation unit when an illumination source emits infrared floodlight (e.g., on an object and / or a user). A floodlight image may include an image showing a user, particularly the user's face, when the user is illuminated with floodlight. A floodlight image can be generated by imaging and / or recording light reflected from an object and / or user illuminated by floodlight. A floodlight image showing a user may include at least a portion of the floodlight on at least a portion of the user. For example, illumination from the floodlight source and imaging using an optical sensor may be synchronized, for example, by using at least one control unit of an optoelectronic device.

[0057] The image generation unit can be configured to image and / or record patterned images and floodlight images simultaneously or at different times. The image generation unit can also be configured to image and / or record patterned images and floodlight images at at least partially overlapping measurement areas or equivalents of these measurement areas.

[0058] In another aspect of the invention, the use of the optoelectronic device according to the invention for authenticating users of devices including the device is disclosed.

[0059] As used herein, the term "user" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, a person who intends to use the device and / or uses the device.

[0060] As used herein, the term "authentication" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, verifying the identity of a user. Specifically, authentication may include distinguishing a user from other humans or objects, particularly distinguishing authorized access from unauthorized access. Authentication may include verifying the identity of the corresponding user and / or assigning an identity to the user. Authentication may include generating and / or providing identity information, for example, providing it to other devices or units (e.g., providing it to at least one authorized unit) for authorizing access to that device. Identity information can be proven through authentication. For example, identity information may be and / or may include at least one identity token. If authentication is successful, it can be verified that the facial image recorded by at least one image generation unit is the user's facial image, and / or the user's identity is verified. Authentication can be performed using at least one authentication process. The authentication process may include multiple steps, such as at least one face detection (e.g., on at least one floodlight image, as will be described in more detail below), and at least one recognition step, wherein an identity is assigned to the detected face and / or at least one identity check and / or verification of the user's identity is performed.

[0061] Authentication can be and / or may include biometric authentication. As used herein, the term "biometric authentication" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, authentication using at least one biometric identifier (e.g., a unique, measurable characteristic used to identify and describe an individual). Biometric identifiers may be physiological characteristics. Authentication will be described in more detail below.

[0062] In another aspect of the invention, a device is disclosed for authenticating a user of a device to perform at least one operation on the device that requires authentication.

[0063] The device includes:

[0064] - The photoelectric device according to the present invention;

[0065] - At least one image generation unit, the at least one image generation unit being configured to generate at least one pattern image when the pattern illumination source emits an infrared light pattern, and being configured to generate at least one floodlight image when the floodlight illumination source emits infrared floodlight;

[0066] - At least one display, wherein the infrared light pattern passes through the display when illuminated from the pattern illumination source, and / or the infrared floodlight passes through the display when illuminated from the floodlight illumination source, wherein the display of the device is at least partially transparent in at least one continuous area covering the pattern illumination source, the floodlight illumination source, and / or the image generation unit.

[0067] - At least one authentication unit, which is configured to use the floodlight image and the pattern image to perform at least one authentication process for a user.

[0068] In particular, the device may include at least one optoelectronic device according to the invention. Therefore, for details, options, and definitions, reference can be made to the devices and optoelectronic devices as discussed above or further described below.

[0069] The device can be selected from the group consisting of: television equipment; game consoles; personal computers; mobile devices, especially mobile phones, and / or smartphones, and / or tablet computers, and / or laptop computers, and / or tablet computers, and / or virtual reality devices, and / or wearable devices such as smartwatches; or other types of portable computers.

[0070] The device may further include at least one communication interface, such as a user interface, configured to receive requests for accessing one or more functions associated with the device, particularly for performing at least one operation requiring authentication on the device. As used herein, the term "access" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, inputting and / or using one or more functions. As used herein, the term "function associated with the device" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, any function, such as accessing at least one element and / or at least one resource of the device or associated with the device. Functions requiring user authentication may be predefined. One or more functions associated with the device may include unlocking the device and / or accessing an application preferably associated with the device and / or accessing a portion of an application preferably associated with the device. For example, the function may include accessing content of the device, such as content stored in the device's database and / or content that can be retrieved by the device. In embodiments, allowing a user to access resources may include allowing a user to perform at least one operation with the device and / or system. Resources can be devices, systems, device functions, system functions, and / or entities. Additionally and / or alternatively, allowing a user to access a resource can include allowing the user to access an entity. An entity can be a physical entity and / or a virtual entity. A virtual entity can be, for example, a database. A physical entity can be an access-restricted area. An access-restricted area can be one of the following: a secure area, a room, an apartment, a vehicle, a portion of the examples mentioned above, etc. Devices and / or systems can be locked. Devices and / or systems can only be unlocked by an authorized user. As used herein, the term "request for access" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a special or customized meaning. The term can specifically refer to, but is not limited to, at least one action and / or instance of requesting access. As used herein, the term "receiving a request" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a special or customized meaning. The term can specifically refer to, but is not limited to, for example, the process of obtaining a request from a data source. Receiving can be fully or partially automated. Receiving a request for access to one or more functions associated with a device can be performed using at least one communication interface. Receiving may include receiving at least one user input via at least one user interface (e.g., the device’s display) and / or receiving requests from remote devices and / or the cloud (e.g., via device communications, such as via the Internet).For example, a request can be generated or triggered by at least one user input (e.g., via entering a security number or other unlocking action by the user), and / or can be sent from a remote device and / or the cloud (e.g., via a connected account).

[0071] Authentication can be performed using at least one authentication unit configured to perform at least one authentication process for a user using a floodlight image and a pattern image.

[0072] The authentication unit may include at least one processor. As used herein, the term "processor" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, any logic circuit configured to perform basic operations of a computer or system, and / or generally to a device configured to perform computations or logical operations. In particular, a processor or computer processor may be configured to process the basic instructions that drive a computer or system. A processor may be a semiconductor-based processor, a quantum processor, or any other type of processor configured to process instructions. As an example, a processor may be or may include a Central Processing Unit ("CPU"). A processor may be a Graphics Processing Unit ("GPU"), a Tensor Processing Unit ("TPU"), a Complex Instruction Set Computing Microprocessor ("CISC"), a Reduced Instruction Set Computing ("RISC") microprocessor, a Very Long Instruction Word ("VLIW") microprocessor, or a processor implementing other instruction sets or multiple processors implementing combinations of instruction sets. The processing device can also be one or more dedicated processing devices, such as application-specific integrated circuits (“ASICs”), field-programmable gate arrays (“FPGAs”), complex programmable logic devices (“CPLDs”), digital signal processors (“DSPs”), network processors, etc. The methods, systems, and devices described herein can be implemented as software in a DSP, microcontroller, or any other auxiliary processor, or as hardware circuitry within an ASIC, CPLD, or FPGA. It should be understood that the term processor can also refer to one or more processing devices, such as a distributed processing device system located on multiple computer systems (e.g., cloud computing), and is not limited to a single device unless otherwise stated. A processor can also be an interface to a remote computer system, such as a cloud service. A processor can include or be a secure isolated zone processor (SEP). An SEP can be secure circuitry configured to process the spectrum. "Secure circuitry" is circuitry that protects isolated internal resources from direct access by external circuitry. A processor can be an image signal processor (ISP) and can include circuitry suitable for processing images, particularly images containing personal and / or confidential information.

[0073] For example, the authentication unit can perform at least one face detection using a floodlight image. Face detection can be performed locally on the device. However, face recognition (i.e., assigning identity to detected faces) can be performed remotely, for example, in the cloud, especially when identification rather than just verification is required. User templates can be stored at a remote device, such as in the cloud, and do not need to be stored locally. This can be advantageous from a storage and security perspective.

[0074] The authentication unit can be configured to identify a user based on a floodlight image. Therefore, in particular, the authentication unit can forward data to a remote device. Alternatively or additionally, the authentication unit can perform user identification based on the floodlight image, particularly by running an appropriate computer program with corresponding functionality. As used herein, the term "identification" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term can specifically refer to, but is not limited to, assigning identity to a detected face and / or at least one identity check and / or verification of the user's identity.

[0075] Authentication can involve multiple steps.

[0076] For example, authentication may include performing face detection at least once using a floodlight image. Face detection may include analyzing the floodlight image. In particular, the analysis of the floodlight image may include using at least one image recognition technique, especially a face recognition technique. Image recognition techniques include at least one process of identifying a user in an image. Image recognition may include using at least one technique selected from the following: color-based image recognition, for example using features such as template matching; segmentation and / or connected component (blob) analysis, for example using size or shape; machine learning and / or deep learning, for example using at least one convolutional neural network.

[0077] For example, authentication may include identifying a user. Identification may include assigning an identity to a detected face and / or at least one identity check and / or verification of the user's identity. Identification may include performing facial verification on the imaged face to confirm whether it is the user's face. Identifying a user may include matching a flood image (e.g., showing the outlines of various parts of the user, particularly the outlines of various parts of the user's face) with a template (e.g., a template image generated during registration). Identifying a user may include determining whether the imaged face is the user's face, and in particular determining whether the imaged face corresponds to at least one image of the user's face stored in at least one memory of a device, for example. If the flood image can match the image template, authentication may be successful. If the flood image cannot match the image template, authentication may be unsuccessful.

[0078] For example, user identification may include determining multiple facial features. Analysis may include comparing the determined facial features with template features, specifically performing a matching process. Template features may be features extracted from at least one template. The template may be or may include at least one image generated during the registration process (e.g., when initializing the device). The template may be an image of an authorized user. Template features and / or facial features may include vectors. Feature matching may include determining the distance between vectors. User identification may include comparing the distance between the vectors with at least one predefined limit. If the distance is at least within tolerance and ≤ the predefined limit, the user can be successfully identified. Otherwise, the user may be rejected and / or refused.

[0079] Analysis of a flood image may further include one or more of the following: filtering; selecting at least one region of interest; forming a difference image between the flood image and at least one offset; inverting the flood image; background correction; decomposing into color channels; decomposing into hue, saturation, and brightness channels; frequency decomposition; singular value decomposition; applying a Canny edge detector; applying a Laplacian Gaussian filter; applying a difference Gaussian filter; applying the Sobel operator; applying the Laplacian operator; applying the Scharr operator; applying the Prewitt operator; applying the Roberts operator; applying the Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying the Radon transform; applying the Hough transform; applying the wavelet transform; thresholding; and creating a binary image. The region of interest may be manually determined by the user or automatically determined, for example, by identifying the user within the image.

[0080] For example, image recognition may include using at least one model, particularly a trained model that includes at least one face recognition model. Analysis of floodlight images can be performed using a face recognition system, such as FaceNet, as described, for example, in Florian Schroff, Dmitry Kalenichenko, and James Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering,” arXiv:1503.03832. The trained model may include at least one convolutional neural network. For example, a convolutional neural network may be designed as described in the following literature: MD Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” CoRR, abs / 1311.2901, 2013; or C. Szegedy et al., “Going deeper with convolutions,” CoRR, abs / 1409.4842, 2014. For more details on convolutional neural networks for face recognition systems, please refer to: Florian Schroff, Dmitry Kalenichenko, and James Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering,” arXiv:1503.03832. Labeled image data from image databases can be used as training data.Specifically, labeled faces can be used from one or more of the following sources: GB Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” Technical Report 07-49, University of Massachusetts Amherst, October 2007; the YouTube® Faces database as described in L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrained videos with matched background similarity,” IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2011; or the Google® Facial Expression Comparison Dataset. Training of convolutional neural networks can be described as in "FaceNet: A Unified Embedding for Face Recognition and Clustering" by Florian Schroff, Dmitry Kalenichenko, and James Philbin, arXiv:1503.03832.

[0081] The authentication unit can be further configured to consider additional security features extracted from the pattern image. In particular, the authentication unit can be further configured to extract material data from the pattern image.

[0082] Therefore, in particular, the authentication unit can forward data to a remote device. Alternatively or additionally, the authentication unit can perform material determination based on pattern images, especially by running an appropriate computer program with corresponding functions. In particular, by treating the material as a parameter for verifying the authentication process, the authentication process can be robust to prevent deception by using recorded user images.

[0083] The authentication unit can be configured to extract material data from a patterned image by beam profile analysis of the light spots. For information on beam profile analysis, refer to WO 2018 / 091649 A1, WO 2018 / 091638 A1, and WO 2018 / 091640A1, the entire contents of which are incorporated herein by reference. Beam profile analysis can allow for reliable classification of a scene based on several light spots. Each light spot in a patterned image can include a beam profile. As used herein, the term "beam profile" generally refers to at least one intensity distribution of a light spot on an optical sensor as a function of pixels. Beam profiles can be selected from the group consisting of: trapezoidal beam profiles; triangular beam profiles; conical beam profiles; and linear combinations of Gaussian beam profiles.

[0084] The authentication unit can be configured to outsource at least one step of the authentication process (such as user identification) and / or at least one step of the verification process (such as consideration of material data) to a remote device, specifically a server and / or a cloud server. This device and the remote device can be part of a computer network, particularly the Internet. Thus, the device can function as a field device used by the user to generate data required in the authentication process and / or its verification. The device can transmit the generated data and / or data associated with intermediate steps of the authentication process and / or its verification to the remote device. In this scenario, the authentication unit can be and / or may include a connection interface configured to transmit information to the remote device. Data generated by the remote device used in the authentication process and / or its verification can be further transmitted to the device. This data can be received by the connection interface included in the device. The connection interface can be specifically configured to transmit or exchange information. In particular, the connection interface can provide a data transmission connection. As an example, the connection interface can be or may include at least one port, including one or more of a network or Internet port, a USB port, and a disk drive.

[0085] It is important to emphasize that data from a device can be transferred to a specific remote device based on at least one circumstance (e.g., date, day, load of a particular remote device, etc.). A field device may not be able to select a specific remote device. Conversely, another device may choose which specific remote device the data can be transferred to. The authentication process and / or the generation of verification data may involve several different entities using the remote device. At least one entity may generate intermediate data and transfer that intermediate data to at least one other entity.

[0086] The authentication unit can be configured for facial recognition authentication processes that operate on floodlight images, pattern images, and / or extracted material data. The authentication unit can also be configured to extract material data from pattern images.

[0087] In embodiments, extracting material data from a pattern image may include generating material type and / or data derived from the material type. Preferably, material data extraction may be based on a pattern image. Material data can be extracted by using at least one model. Extracting material data may include providing a pattern image to the model and / or receiving material data from the model. Providing the image to the model may include, and subsequently may be, receiving the pattern image at the model's input layer or via a model loss function. The model may be a data-driven model. A data-driven model may include a convolutional neural network and / or an encoder-decoder structure, such as an autoencoder. Other examples for generating representations may be FFT, wavelets, deep learning (such as CNN), energy models, normalized flow, GANs, visual transformers or transformers for natural language processing, autoregressive image modeling, normalized flow, deep autoencoders, and deep energy-based models. Supervised or unsupervised schemes may be applicable to generating representations and also to generating embeddings in ML languages, such as cosine or Euclidean metrics. The data-driven model may be parameterized based on a training dataset comprising at least one image and material data, preferably at least one pattern image and material data. In another embodiment, extracting material data may include providing images to a model and / or receiving material data from a model. In another embodiment, a data-driven model may be trained based on a training dataset comprising at least one image and material data. In another embodiment, a data-driven model may be parameterized based on a training dataset comprising at least one image and material data. A data-driven model may be parameterized based on a training dataset to receive images and provide material data based on the received images. A data-driven model may be trained based on a training dataset to receive images and provide material data as output based on the received images. The training dataset may include at least one image and material data (preferably material data associated with at least one image). An image may include a representation of the image. The representation may be a low-dimensional representation of the image. The representation may include at least a portion of the data or information associated with the image. The image representation may include feature vectors. In embodiments, determining the representation, particularly the low-dimensional representation, may be based on principal component analysis (PCA) mapping or radial basis function (RBF) mapping. Determining the representation may also be referred to as generating the representation. Generating a representation based on a PCA mapping may include clustering based on features in a patterned image and / or a portion of the image. Alternatively or additionally, the generating representation may be based on a neural network structure suitable for dimensionality reduction. Neural network architectures suitable for dimensionality reduction may include encoders and / or decoders. In the example, the neural network architecture may be an autoencoder. In the example, the neural network architecture may include a convolutional neural network (CNN). A CNN may include at least one convolutional layer and / or at least one pooling layer.CNNs can reduce the dimensionality of parts and / or parts of an image by applying convolutions (e.g., convolutional layers) and / or by pooling. Applying convolutions can be adapted to select features that are relevant to the material information of the patterned image.

[0088] In embodiments, the model may be adapted to determine the output based on the input. Specifically, the model may be adapted to determine material data based on an image as input. The model may be a deterministic model, a data-driven model, or a hybrid model. Preferably, the deterministic model reflects the physical phenomenon in a mathematical form, for example, including a first-principles model. The deterministic model may include a set of equations describing the interaction between the material and patterned electromagnetic radiation, thereby producing measures of condition, vital signs, etc. The data-driven model may be a classification model. The hybrid model may be a classification model including at least one machine learning architecture and model parameters with deterministic or statistical adjustments. Statistical or deterministic adjustments may be introduced to improve the quality of the results because these adjustments provide a systematic relationship between empiricism and theory. In embodiments, the data-driven model may be a classification model. The classification model may include at least one machine learning architecture and model parameters. For example, the machine learning architecture may be or may include one or more of the following: linear regression, logistic regression, random forest, piecewise linear, nonlinear classifier, support vector machine, Naive Bayes classification, nearest neighbor, neural network, convolutional neural network, generative adversarial network, support vector machine, or gradient boosting algorithm, etc. In the case of neural networks, the model can be a multi-scale neural network or a recurrent neural network (RNN), such as, but not limited to, a gated recurrent unit (GRU) recurrent neural network or a long short-term memory (LSTM) recurrent neural network. The data-driven model can be parameterized based on a training dataset. The data-driven model can be trained based on the training dataset. Training the model can include parameterizing the model. The term "training" can also mean learning. Specifically, the term can refer to, but is not limited to, the process of building a classification model, and particularly determining and / or updating the parameters of a classification model. Updating the parameters of a classification model can also be referred to as retraining. Training as discussed herein can include retraining. In embodiments, the training dataset can include at least one image and material information.

[0089] In an embodiment, extracting material data from an image using a data-driven model may include providing the image to the data-driven model. Alternatively or additionally, extracting material data from an image using a data-driven model may include generating an embedding associated with the image based on the data-driven model. The embedding may refer to a low-dimensional representation associated with the image, such as a feature vector. The feature vector may be adapted to suppress background while preserving a material signature indicating the material data. In this context, the background may refer to information independent of the material signature and / or the material data. Further, the background may refer to information related to biometric features, such as facial features. Based on the embedding associated with the image, the material data can be determined using the data-driven model. Alternatively or additionally, extracting material data from an image by providing the image to the data-driven model may include transforming the image into material data, particularly material feature vectors indicating the material data. Therefore, the material data may further include material feature vectors and / or the material feature vectors may be used to determine the material data.

[0090] In this embodiment, the authentication process can be verified based on the extracted material data.

[0091] In an embodiment, verification based on extracted material data may include determining whether the extracted material data corresponds to expected material data. Determining whether the extracted material data matches expected material data can be termed verification. Allowing or denying a user and / or object from performing at least one operation requiring authentication on the device based on material data may include verifying the authentication or authentication process. Verification may be based on material data and / or images. Determining whether the extracted material data corresponds to expected material data may include determining the similarity between the extracted material data and expected material data. Determining the similarity between the extracted material data and expected material data may include comparing the extracted material data with the expected material data. Expected material data may refer to predetermined material data. In an example, expected material data may be skin. It may be determined whether material data corresponds to expected material data. In an example, material data may be a non-skin material or silicon. Determining whether material data corresponds to expected material data may include comparing the material data with the expected material data. The comparison of material data with expected material data may result in allowing and / or denying a user and / or object from performing at least one operation requiring authentication. In the example, skin as the desired material data can be compared with non-skin materials or silicon as material data, and the result can be rejection because silicon or non-skin materials may differ from skin.

[0092] In an embodiment, the authentication process or its verification may include generating at least one feature vector from the material data and matching the material feature vector with an associated material reference template vector.

[0093] The authentication unit can be configured to authenticate a user if the user can be identified and / or if the material data matches the expected material data.

[0094] Authentication may include the use of even additional security features, such as 3D information and / or additional liveness data.

[0095] For example, beam profiling analysis can be used to determine three-dimensional information (e.g., the longitudinal coordinates z of one or more points on a user's face), as described, for example, in WO 2018 / 091640 A1, the entire contents of which are incorporated herein by reference. For example, an authentication unit can be configured to determine at least one longitudinal coordinate z by evaluating a quotient signal Q of sensor signals detected by an image generation unit. The quotient signal Q can be generated by combining sensor signals, particularly by one or more of the following: dividing sensor signals, dividing sensor signals by multiples, or dividing a linear combination of sensor signals. The authentication unit can be configured to determine the longitudinal coordinate using at least one predetermined relationship between the quotient signal Q and the longitudinal coordinate z. For example, the authentication unit is configured to obtain the quotient signal Q by the following formula:

[0096]

[0097] Where x and y are the lateral coordinates, A1 and A2 are the regions of the beam profile at the sensor location, and E(x,y,z) o ) indicates the distance z between objects o The beam profile is given below. Regions A1 and A2 may differ. In particular, A1 and A2 are not identical. Therefore, A1 and A2 may differ in one or more aspects of shape or content. As used herein, the term "beam profile" refers to the spatial distribution of beam intensity, particularly the spatial distribution in at least one plane perpendicular to the beam propagation. A beam profile may be a cross-section of the beam. For further details on determining the longitudinal coordinate z using beam profile analysis, refer to WO2018 / 091640 A1, the entire contents of which are incorporated herein by reference.

[0098] The authentication unit can be configured to determine the longitudinal coordinates z at multiple locations on a user's face and to determine a depth map. The determined depth map can be compared to a predetermined depth map of the user, for example, determined during the registration process. The authentication unit can be configured to authenticate the user if the determined depth map matches (in particular, at least within tolerance) the user's predetermined depth map. Otherwise, the user can be rejected.

[0099] In addition, other security features (such as surface roughness) can be used for certification.

[0100] For example, the device can be configured to determine a measure of surface roughness, such as by means of...

[0101] a) Receive a speckle image showing an object irradiated with coherent electromagnetic radiation associated with wavelengths between 850 nm and 1400 nm.

[0102] b) Determine the surface roughness measure based on the speckle image.

[0103] c) Provide the surface roughness measurement.

[0104] The authentication device can be configured to authenticate or deny a user by using a surface roughness metric.

[0105] As used herein, the term "speckle image" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, an image showing multiple speckles. A speckle image can show multiple speckles. A speckle image can include an image showing at least a portion of a user, particularly the user's face, when irradiated with coherent electromagnetic radiation, particularly within a corresponding region of interest included in the image. A speckle image can be generated when a user is irradiated with coherent electromagnetic radiation associated with wavelengths between 850 nm and 1400 nm. A speckle image can show a speckle pattern. A speckle pattern can specify the distribution of speckles. A speckle image can indicate the spatial extent of the speckles. A speckle image can be suitable for determining a measure of surface roughness. A speckle image can be generated using at least one camera. To generate a speckle image, a user can be irradiated by an illumination source. As used herein, the term "speckle" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. The term can specifically refer to, but is not limited to, optical phenomena caused by coherent electromagnetic radiation interference due to the irregularity or irregularity of the surface. Speckle can manifest as a variation in contrast in an image (e.g., a speckle image). As used herein, the term "speckle pattern" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term can specifically refer to, but is not limited to, the distribution of multiple speckles. The distribution of multiple speckles can refer to the spatial distribution of at least one speckle among multiple speckles and / or the spatial distribution of at least two speckles relative to each other. The spatial distribution of at least one speckle among multiple speckles can refer to and / or specify the spatial extent of the at least one speckle among multiple speckles. The spatial distribution of at least two speckles among multiple speckles can refer to and / or specify the spatial extent of the first speckle among the at least two speckles relative to the second speckle among the at least two speckles, and / or the distance between the first speckle among the at least two speckles and the second speckle among the at least two speckles. The generation of speckle images can be initiated by user action or can be initiated automatically, for example, when the presence of a user is automatically detected within the camera's field of view and / or a predetermined area of ​​the field of view. As used herein, the term "field of view" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, the angular range of the observable world and / or at least one scene that can be captured or viewed by an optical system (such as an image generation unit). The field of view is typically expressed in degrees and / or radians, and exemplaryly may represent the total angle spanned by the image and / or the visible area.

[0106] As mentioned above, since speckle patterns may be caused by surface irregularities, they reflect the surface roughness. Therefore, determining surface roughness metrics based on speckle patterns in speckle images utilizes the relationship between speckle distribution and surface roughness. This provides a low-cost, efficient, and readily available solution for surface roughness assessment.

[0107] For example, a speckle image can show a user under coherent electromagnetic radiation and determine the surface roughness of the user's skin. Preferably, the user may have generated the speckle image and / or initiated its generation. Preferably, the speckle image can be initiated by the user-operated application of a mobile electronic device. By doing so, the user can decide for themselves when to determine the surface roughness of their skin. This allows non-expert users to determine surface roughness, and the measurement can be performed in a more natural and less artificial context. Consequently, surface roughness can be assessed more realistically, which in turn provides a more accurate measure of surface roughness. For example, depending on a person's activities, the surface roughness of the skin may vary throughout the day. Exercise can affect surface roughness and cause skin creaming. This effect can be verified using the methods and systems described herein.

[0108] For example, a speckle image can be associated with a resolution of less than 5 megapixels. Preferably, the speckle image can be associated with a resolution of less than 3 megapixels, more preferably less than 2.5 megapixels, and most preferably less than 2 megapixels. Such a speckle image can be generated using readily available, small, and inexpensive smartphone cameras. Furthermore, the storage and processing power required to evaluate surface roughness is relatively small. Therefore, the low resolution of the speckle image used to evaluate surface roughness enables the use of mobile electronic devices, particularly devices like smartphones or wearable devices, which have strictly limited size, memory, and processing power.

[0109] A speckle image can be reduced to a predefined size before determining surface roughness metrics. Reducing the speckle image to a predefined size can be based on applying one or more image enhancement techniques. Reducing the speckle image to a predefined size can include: selecting a speckle image region of a predefined size, and segmenting the speckle image into regions of predefined sizes. The predefined-sized speckle image region can be associated with a living organism, such as a human being, and particularly with the skin of a living organism, such as human skin. The portion of the image outside the predefined-sized speckle image region may be associated with the background and / or may be unrelated to a living organism, such as a human being. By doing so, the amount of data that needs to be processed is reduced, which reduces the time required to determine surface roughness, or allows for the need for less storage and processors. Furthermore, the image portions useful for analysis are selected. Therefore, reducing the size allows for the neglect of portions of the speckle image that are unrelated to the object or living organism (such as a human). Thus, surface roughness metrics can be easily determined, and for analysis purposes, obstructive portions irrelevant to the user are ignored.

[0110] For example, image enhancement techniques may include at least one of the following: scaling, cropping, rotating, blurring, distorting, shearing, resizing, folding, changing contrast, changing brightness, adding noise, multiplying by at least a portion of the pixel value, filtering, adjusting color, applying convolution, imprinting, sharpening, flipping, averaging pixel values, etc.

[0111] Reducing a speckle image to a predefined size can be based on detecting a user within the speckle image. Specifically, the speckle image can be reduced to a predefined size based on detecting a user before determining a surface roughness metric. Specifically, reducing the speckle image to a predefined size based on detecting a user can include: detecting the user's contour (e.g., detecting the contour of the user's face), and reducing the speckle image to a region associated with the user (particularly a region associated with the user's face). Preferably, the region associated with the user can be within the contour of the user (particularly the user) and / or the contour of the user's face.

[0112] A floodlight image can reveal the contours of a user. The user's contours can be detected based on the floodlight image. Preferably, the user's contours can be detected by providing the floodlight image to an object detection data-driven model, particularly a user detection model, wherein the object detection data-driven model can be parameterized and / or trained based on a training dataset to receive the floodlight image and provide indications of the user's contours. The training dataset may include the floodlight image and indications of the contours of objects and / or people. The contour indications may include multiple points indicating the location of specific landmarks associated with the user. For example, where a speckle image can be associated with a user's face, the user's face can be detected based on the contours, where the contours may indicate facial landmarks such as the nose point, lip corners, or eyebrow tails.

[0113] As used herein, the term "surface roughness" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, the characteristics of a surface in relation to a user. In particular, surface roughness may characterize the lateral and / or vertical extent of a surface feature. Surface roughness can be evaluated based on surface roughness metrics. Surface roughness metrics can quantify surface roughness.

[0114] As used herein, the term "surface feature" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, any structure of any shape associated with a surface, particularly a user's surface. In particular, a surface feature may refer to a substructure of a surface associated with a user. A surface may include multiple surface features. For example, a ridge or depression may be a surface feature. Preferably, a surface feature may refer to a portion of a surface associated with an angle not equal to 90° relative to the surface normal.

[0115] As used herein, the term "surface roughness metric" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, a measure suitable for quantifying surface roughness. A surface roughness metric may be associated with a speckle pattern. For example, a surface roughness metric may include at least one of the following: fractal dimension, speckle size, speckle contrast, speckle modulation, roughness index, standard deviation of the height associated with a surface feature, lateral correlation length, mean median height, root mean square height, or a combination thereof. Preferably, a surface roughness metric may be suitable for describing vertical and lateral surface features. A surface roughness metric may include values ​​associated with the surface roughness metric. A surface roughness metric may refer to a term used to measure a quantity of surface roughness and / or a value associated with a quantity used to measure surface roughness. Determining a surface roughness metric based on a speckle image may mean determining the surface roughness metric based on the speckle pattern in the speckle image.

[0116] Surface roughness metrics can be determined based on speckle images by providing them to the model and receiving surface roughness metrics from the model. For example, the model can be adapted to determine the output based on the input. In particular, the model can be adapted to determine surface roughness metrics based on speckle images, preferably based on the received speckle images.

[0117] For example, a model can be, or may include, one or more of a physical model, a data-driven model, or a hybrid model. A hybrid model can be a model that includes at least one data-driven model with physical or statistical adjustments and model parameters. Statistical or physical adjustments can be introduced to improve the quality of the results because these adjustments provide a systematic relationship between empiricism and theory. For example, a data-driven model can represent the correlation between surface roughness metrics and speckle images. A data-driven model can obtain the correlation between surface roughness metrics and speckle images based on a training dataset that includes multiple speckle images and multiple surface roughness metrics. For example, a data-driven model can be parameterized based on a training dataset to receive speckle images and provide surface roughness metrics. A data-driven model can be trained based on a training dataset. A training dataset can include at least one speckle image and at least one corresponding surface roughness metric. A training dataset can include multiple speckle images and multiple surface roughness metrics. Training the model can include parameterizing the model. A data-driven model can be parameterized and / or trained to provide surface roughness metrics based on speckle images, particularly received speckle images. Determining a surface roughness measure based on a speckle image can include: providing the speckle image to a data-driven model, and receiving the surface roughness measure from the data-driven model. Providing a surface roughness measure based on a speckle image can include: mapping the speckle image to a surface roughness measure. The data-driven model can be parameterized and / or trained to receive the speckle image. The data-driven model can receive the speckle image at an input layer. The term "training" can also refer to learning. This term can specifically refer to, but is not limited to, the process of constructing a data-driven model, and in particular, determining and / or updating the parameters of the data-driven model. Updating the parameters of the data-driven model can also be referred to as retraining. Training as discussed herein can include retraining. During training, the data-driven model can be adjusted to achieve a best fit with the training data, for example, to best fit at least one input value to at least one desired output value. For example, if the neural network is a feedforward neural network (such as a CNN), a backpropagation algorithm can be applied to train the neural network. In the case of an RNN, a gradient descent algorithm or a backpropagation algorithm over time can be used to achieve the training objective. Training a data-driven model can include or can refer to calibrating a model, but is not limited to calibrating a model.

[0118] For example, a physical model can reflect physical phenomena mathematically, including, for instance, a first-principles model. A physical model can include a set of equations describing the interaction between an object and coherent electromagnetic radiation, thereby generating a measure of surface roughness. The physical model can be based on at least one of the following: fractal dimension, speckle size, speckle contrast, speckle modulation, roughness index, standard deviation of the height associated with a surface feature, lateral correlation length, mean median height, root mean square height, or a combination thereof. Specifically, a physical model can include one or more equations relating a speckle image to a measure of surface roughness, based on equations related to fractal dimension, speckle size, speckle contrast, speckle modulation, roughness index, standard deviation of the height associated with a surface feature, lateral correlation length, mean median height, root mean square height, or a combination thereof.

[0119] For example, the fractal dimension can be determined based on the Fourier transform and / or the inverse Fourier transform of the speckle image. For instance, the fractal dimension can be determined based on the slope of a linear function fitted to a double log-log plot of power spectral density versus frequency obtained through the Fourier transform. The speckle size can refer to the spatial extent of one or more specks. Where the speckle size can refer to the spatial extent of more than one speckle, the speckle size can be determined based on the average of more than one speckle size and / or a weighted average of more than one speckle size. The speckle contrast can refer to a measure of the standard deviation of the average intensity of at least a portion of the speckle image relative to at least a portion of the average intensity of the speckle image. The speckle modulation can refer to a measure of the intensity fluctuations associated with speckles in at least a portion of the speckle image. The roughness index, the standard deviation of the height associated with surface features, the lateral correlation length, or a combination thereof can be determined based on an autocorrelation function associated with a double log-log plot of power spectral density versus frequency obtained through the Fourier transform.

[0120] For example, determining a surface roughness metric based on a speckle image can include determining the surface roughness metric based on a speckle pattern. For example, determining surface roughness based on a speckle pattern can include determining surface roughness based on the distribution of multiple specks in a speckle image. Determining a surface roughness metric based on the distribution of multiple specks in a speckle image can refer to determining the distribution of multiple specks in a speckle image. Determining the speckle distribution can include determining at least one of the following: the fractal dimension associated with the speckle image, the speckle size associated with the speckle image, the speckle contrast associated with the speckle image, the speckle modulation associated with the speckle image, the roughness index associated with the speckle image, the standard deviation of the height associated with the surface features associated with the speckle image, the lateral correlation length associated with the speckle image, the mean median height associated with the speckle image, the root mean square height associated with the speckle image, or a combination thereof.

[0121] Alternatively or concurrently, determining a surface roughness metric may include determining at least one of the following: fractal dimension associated with a speckle image, speckle size associated with a speckle image, speckle contrast associated with a speckle image, speckle modulation associated with a speckle image, roughness index associated with a speckle image, standard deviation of height associated with surface features associated with a speckle image, lateral correlation length associated with a speckle image, mean median height associated with a speckle image, root mean square height associated with a speckle image, or a combination thereof.

[0122] For example, determining a surface roughness metric can be based on the distribution of speckles in a speckle image. Determining a surface roughness metric based on the distribution of speckles in a speckle image can include determining at least one of the following: speckle size distribution, power spectral density associated with the speckle image, fractal dimension associated with the speckle image, speckle contrast, speckle modulation, or a combination thereof.

[0123] Alternatively or alternatively, determining surface roughness metrics based on the distribution of speckles in a speckle image may include providing the speckle image to a model, particularly a data-driven model, wherein the data-driven model may be parameterized and / or trained based on a training dataset comprising one or more speckle images and one or more corresponding surface roughness metrics.

[0124] For example, a surface roughness metric can be determined based on a speckle image by providing the speckle image to the model and receiving the surface roughness metric from the model. The model can be a data-driven model and can be parameterized and / or trained based on a training dataset that includes multiple speckle images and corresponding surface roughness metrics or indicators of surface roughness metrics. Alternatively or concurrently, the model can be a physical model.

[0125] For example, the method may further include generating a partial speckle image. A partial speckle image can refer to a portion of the image generated based on the speckle image. A partial speckle image can be generated by applying one or more image enhancement techniques to the speckle image.

[0126] For example, the method may further include generating a first speckle image and a second speckle image. The speckle image may include a first speckle image and a second speckle image. The first speckle image may refer to a first portion of the speckle image. The second speckle image may refer to a second portion of the speckle image. Preferably, the first speckle image and the second speckle image may be different from each other. In particular, the first speckle image and the second speckle image may be non-overlapping. The first speckle image and the second speckle image can be generated by applying one or more image enhancement techniques to the speckle image. Determining a surface roughness measure based on the speckle image may include: determining a first surface roughness measure based on the first speckle image, and determining a second surface roughness measure based on the second speckle image. Providing a surface roughness measure may include providing a first surface roughness measure and a second surface roughness measure. In particular, the first surface roughness measure and the second surface roughness measure may be provided together. Preferably, the first surface roughness measure and the second surface roughness measure may be provided in a surface roughness measure map indicating the spatial distribution of the surface roughness measure. For example, a surface roughness metric map can indicate a first surface roughness metric associated with a first region in the surface roughness metric map, and a second surface roughness metric map can indicate a second surface roughness metric associated with a second region in the surface roughness metric map. In particular, the surface roughness metric map can resemble a heatmap, wherein the surface roughness metric can be plotted for the region associated with the corresponding surface roughness metric.

[0127] The surface roughness metric is determined using at least one processor. The authentication unit may be or may include at least one processor and / or may be designed as software or an application. User authentication or denial using the surface roughness metric is performed using the device's authentication unit and / or a remote authentication unit.

[0128] In this embodiment, the device may be a mobile electronic device. The surface roughness measurement may be determined by the mobile electronic device, and / or the speckle image is generated using a camera on the mobile electronic device. Specifically, a person may initiate the generation of the speckle image based on the mobile electronic device. This is advantageous because many people own mobile electronic devices such as smartphones. These devices accompany people, therefore, surface roughness measurements can be performed at any time and in a more natural and less artificial context. This allows for a more realistic assessment of surface roughness, thus providing a more accurate measure of surface roughness.

[0129] Surface roughness metrics can be used as biometric identifiers to uniquely identify users. For example, authentication may include determining whether a surface roughness metric corresponds to a human surface roughness metric. For example, the method may include determining whether a surface roughness metric corresponds to a specific user's surface roughness metric. Determining whether a surface roughness metric corresponds to a human and / or a specific user's surface roughness metric may include comparing the surface roughness metric to at least one predefined or predetermined value range of the surface roughness metric (e.g., stored in at least one database on a device or in a remote database such as in the cloud). The user is authenticated if the determined surface roughness metric is at least within the tolerance of the redefined or predetermined value range of the surface roughness metric; otherwise, authentication fails. For example, a surface roughness metric may be human skin roughness. If the determined human skin roughness is in the range of 10 µm to 150 µm, the user is authenticated. However, other ranges are also possible.

[0130] Alternatively or concurrently, the device can be configured to extract blood perfusion data. For example, the beam projected by an irradiation source (e.g., a projector) can be coherent patterned infrared irradiation. Extracting blood perfusion data can include determining the speckle contrast of the patterned image and determining a blood perfusion metric based on the determined speckle contrast. The speckle contrast can be a measure of the average contrast of the intensity distribution within a region of the speckle pattern. In particular, the speckle contrast K over the speckle pattern region can be expressed as the standard deviation σ versus the average speckle intensity. The ratio, that is,

[0131]

[0132] Speckle contrast can include speckle contrast values. Speckle contrast values ​​can range from 0 to 1. Blood perfusion can be determined based on speckle contrast.

[0133] Blood perfusion measurements can depend on the determined speckle contrast. If the speckle contrast changes, the blood perfusion measurement obtained based on the speckle contrast will also change accordingly. Blood perfusion measurements can be a single number or value, representing the likelihood that the object is a living organism. This information can be used as an additional security feature for user authentication.

[0134] For example, to determine speckle contrast, a complete pattern image can be used. Alternatively, a portion of the pattern image can be used to determine speckle contrast. Preferably, a portion of the pattern image represents a smaller area of ​​the pattern image than the area of ​​the complete pattern image. A portion of the pattern image can be obtained by cropping the pattern image.

[0135] In this embodiment, a data-driven model can be used to determine blood perfusion measurements. The data-driven model is parameterized and / or trained based on a training dataset. The training dataset may include pattern images and blood perfusion measurements. The data-driven model can be parameterized and / or trained based on the training dataset to output blood perfusion measurements based on received pattern images.

[0136] The authentication process can be verified based on blood perfusion measurements. In an embodiment, verification based on blood perfusion measurements may include determining whether the blood perfusion measurement corresponds to a human blood perfusion measurement. Determining whether a blood perfusion measurement corresponds to a human can be termed verification. Allowing or denying a user and / or object to perform at least one operation requiring authentication on the device based on blood perfusion measurements may include verifying the authentication or authentication process. Verification may be based on blood perfusion measurements. Determining whether a blood perfusion measurement corresponds to a human may include comparing the blood perfusion measurement to at least one predefined or predetermined range of blood perfusion measurement values, for example, stored in at least one database. If the extracted blood perfusion measurement is at least within the tolerance range of the redefined or predetermined range of blood perfusion measurement values, then the authentication passes verification; otherwise, the authentication fails verification. If the authentication passes verification, the method may include allowing the user to perform at least one operation requiring authentication. Otherwise, if the authentication fails verification, the method may include denying the user from performing at least one operation requiring authentication.

[0137] The device may include at least one authorization unit configured to allow a user to perform at least one operation on the device, such as unlocking the device upon successful user authentication, or denying the user from performing at least one operation on the device upon unsuccessful authentication. Thus, the user is aware of the authentication result. The authorization unit may be configured to allow or deny the user to perform at least one operation requiring authentication on the device based on material data and identification using floodlight images. The authorization unit may be configured to allow or deny a user access to one or more functions associated with the device, depending on authentication or denial. Allowing may include granting permission to access the one or more functions. The authorization unit may be configured to determine whether the user corresponds to an authorized user, wherein allowing or denying is further based on determining whether the user corresponds to an authorized user. As used herein, the term "authorization" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a special or custom meaning. The term may specifically refer to, but is not limited to, the process of assigning access rights to a user (particularly selective permission or selective restriction of access to the device and / or at least one resource of the device). The authorization unit may be configured for access control. As used herein, the term "authorization unit" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, a unit configured to authorize a user, such as a processor. An authorization unit may include at least one processor or may be designed as software or an application. The authorization unit and the authentication unit may be integrated, for example, by using the same processor. The authorization unit may be configured to allow a user to access, for example, one or more functions on the device, such as unlocking the device, upon successful authentication, or to deny the user access to, for example, one or more functions on the device, upon unsuccessful authentication.

[0138] The device (e.g., by using a display) can be configured to display the results of authentication and / or authorization, for example, by using at least one communication interface (e.g., a user interface, such as a display).

[0139] The device used to authenticate users may be included in the user's device, for example, it may be a component of the user's device, or it may be the user's device itself.

[0140] In another aspect, the present invention discloses a method for authenticating a user of a device to perform at least one operation requiring authentication on the device. The device is an embodiment of the present invention (e.g., according to one or more embodiments given above or further detailed below).

[0141] The method includes:

[0142] - Illuminate the user with an infrared light pattern from a pattern illumination source.

[0143] - Illuminate the user with infrared floodlight from a floodlight source.

[0144] - The image generation unit generates at least one pattern image showing at least a portion of the user when illuminated by the infrared light pattern, and generates at least one floodlight image showing at least a portion of the user when illuminated by the infrared floodlight.

[0145] - Identify the user based on this floodlight image.

[0146] - Extract at least one security feature from the at least one pattern image; and

[0147] - Based on this security feature and this identification, the user may be allowed or denied to perform at least one operation on the device that requires authentication.

[0148] These method steps can be performed in a given order or in a different order. Furthermore, there may be one or more additional method steps not listed. Furthermore, one, more than one, or even all of the method steps may be performed repeatedly. For details, options, and definitions, refer to the optoelectronic devices and apparatus discussed above.

[0149] Extracting security features includes extracting material data from the at least one pattern image. The method may include allowing or denying a user to perform at least one authentication-required operation on the device based on the material data and identification and / or even additional security features (e.g., 3D information and / or additional liveness data).

[0150] This method can be computer-implemented. As used herein, the term "computer-implemented" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, methods involving at least one computer and / or at least one computer network. The computer and / or computer network may include at least one processor configured to perform at least one method step of the method according to the invention. Specifically, each of these method steps is performed via a computer and / or computer network. The method can be performed entirely automatically, specifically without user interaction. For example, irradiation and / or image generation can be triggered and / or performed using at least one processor.

[0151] All described method steps can be performed using the device. Therefore, a single processing device can be configured to exclusively execute at least one computer program, particularly at least one line of computer program code configured to execute at least one algorithm, as used in at least one embodiment of the method according to the invention. In this document, the computer program executed on the single processing device can include all instructions that cause the computer to perform the method. Alternatively or additionally, at least one method step can be performed using at least one remote device, particularly selected from at least one of a server or a cloud server, especially when the device and the remote device are part of a computer network. In this case, the computer program can include at least one remote component to be executed by at least one remote processing device to perform at least one method step. The remote component may, for example, have the function of performing user identification. Further, the computer program can include at least one interface configured to forward data to and / or receive data from at least one remote component of the computer program.

[0152] In another aspect, a computer program is disclosed, comprising instructions that, when executed by a device, cause the device to perform the method according to any one of the foregoing embodiments relating to the method. Specifically, the computer program may be stored on a computer-readable data carrier and / or a computer-readable storage medium. The computer program may execute on at least one processor included in the device. The computer program may generate input data by accessing and / or controlling at least one unit of the device (e.g., a projector and / or a floodlight source and / or an image generation unit). The computer program may generate result data based on the input data, particularly by using an authentication unit.

[0153] As used herein, the terms "computer-readable data carrier" and "computer-readable storage medium" specifically refer to non-transitory data storage devices, such as hardware storage media on which computer-executable instructions are stored. The stored computer-executable instructions may be associated with a computer program. Computer-readable data carriers or storage media may specifically be or may include storage media such as random access memory (RAM) and / or read-only memory (ROM).

[0154] In embodiments, a computer-readable storage medium can refer to any suitable data storage device or computer-readable memory on which one or more sets of instructions (e.g., software) are stored, embodying any one or more of the methods or functions described herein. The instructions may also reside wholly or at least partially within main memory and / or processor during execution by a computer, main memory, and processing device that may constitute a computer-readable storage medium. These instructions may further be transmitted or received over a network via a network interface device. Computer-readable storage media include, for example, hard disk drives on a server, USB storage devices, CDs, DVDs, or Blu-ray discs.

[0155] The computer-readable storage medium, particularly the non-transitory computer-readable storage medium, includes instructions that, when executed by a computer, particularly a processor of the device according to the invention, cause the computer to perform the following operations:

[0156] - Receive requests for accessing one or more functions associated with the device;

[0157] - Perform at least one authentication process, which includes the following steps:

[0158] - Trigger infrared pattern illumination of the user from a pattern illumination source, and infrared flood illumination of the user from a flood illumination source.

[0159] - Trigger to generate at least one pattern image and at least one floodlight image for the user simultaneously.

[0160] - Identify the user based on the floodlight image;

[0161] - Extract at least one security feature from the at least one pattern image;

[0162] - Authenticate or deny the user based on this security feature and this identification.

[0163] Therefore, specifically, one, more, or even all of the method steps indicated above can be performed by using a computer or computer network, preferably by using a computer program.

[0164] This document further discloses and proposes a computer program product having program code means so that, when the program is executed on a computer or computer network, it performs the method according to the invention in one or more of the embodiments included herein. Specifically, the program code means may be stored on a computer-readable data carrier and / or a computer-readable storage medium.

[0165] This document further discloses and proposes a data carrier having a data structure stored thereon, which, after being loaded into a computer or computer network (e.g., into the working memory or main memory of the computer or computer network), can perform methods according to one or more embodiments disclosed herein.

[0166] This document further discloses and proposes a computer program product having program code means stored on a machine-readable medium to perform methods according to one or more embodiments disclosed herein when the program is executed on a computer or computer network. As used herein, a computer program product refers to a program that is a tradable product. The product can generally exist in any format, such as in paper format, or on a computer-readable data carrier and / or computer-readable storage medium. Specifically, the computer program product can be distributed via a data network.

[0167] This document further discloses and proposes a non-transient computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to one or more embodiments disclosed herein.

[0168] Finally, this document discloses and proposes a modulated data signal containing computer system or computer network readable instructions for performing methods according to one or more embodiments disclosed herein.

[0169] Referring to the computer implementation aspects of the present invention, one or more, or even all, of the method steps in one or more of the methods disclosed in the embodiments herein can be performed using a computer or computer network. Therefore, typically, any of the method steps involving the provision and / or manipulation of data can be performed using a computer or computer network. Generally, these method steps can include any method steps, except for those that typically require manual work, such as providing samples and / or performing certain aspects of actual measurements.

[0170] Specifically, this article further discloses:

[0171] - A computer or computer network including at least one processor, wherein the processor is adapted to perform a method according to one of the embodiments described in this specification.

[0172] - A computer-loadable data structure adapted to perform a method according to one of the embodiments described in this specification when the data structure is executed on a computer.

[0173] - A computer program, wherein the computer program is adapted, when executed on a computer, to perform a method according to one of the embodiments described in this specification.

[0174] A computer program comprising program means for performing a method according to one of the embodiments described herein when the computer program is executed on a computer or a computer network.

[0175] - A computer program comprising program means according to a previous embodiment, wherein the program means is stored on a computer-readable storage medium.

[0176] - A storage medium wherein a data structure is stored on the storage medium, and wherein the data structure is adapted to perform a method according to one of the embodiments described herein after being loaded into the main storage device and / or working storage device of a computer or computer network.

[0177] - A computer program product having program code means, wherein the program code means may be stored or stored on a storage medium for performing a method according to one of the embodiments described herein if the program code means is executed on a computer or a computer network.

[0178] As used herein, the terms “have,” “include,” or “contain,” or any of their grammatical variations, are used in a non-exclusive manner. Thus, these terms can refer either to a situation where no other features exist in the entity described in the context besides those introduced by these terms, or to a situation where one or more other features exist. For example, the statements “A has B,” “A includes B,” and “A contains B” can refer either to a situation where no other elements exist in A besides B (i.e., A consists solely of B), or to a situation where entity A contains one or more other elements besides B (such as element C, elements C and D, or even other elements).

[0179] Furthermore, it should be noted that the terms "at least one," "one or more," or similar expressions indicating a feature or element may appear once or more, but are typically used only once when describing the corresponding feature or element. In most cases, the expressions "at least one" or "one or more" are not repeated when referring to the corresponding feature or element, but in fact, the corresponding feature or element may appear once or more.

[0180] Furthermore, as used herein, the terms “preferredly,” “more preferably,” “particularly,” “more particularly,” “specifically,” “more specifically,” or similar terms are used in combination with optional features without limiting the possibility of alternatives. Therefore, the features introduced by these terms are optional features and are not intended to limit the scope of the claims in any way. As those skilled in the art will recognize, the invention can be practiced by using alternative features. Similarly, features introduced by phrases such as “in embodiments of the invention” are intended to be optional features and do not limit any alternative embodiments of the invention, the scope of the invention, or the possibility of combining features introduced in this way with other optional or non-optional features of the invention.

[0181] In summary, and without excluding other possible embodiments, the following embodiments are conceivable:

[0182] Example 1. A photoelectric device configured to emit at least one infrared light pattern comprising a plurality of infrared beams and to emit infrared floodlight, the photoelectric device comprising:

[0183] - A light emitter structure comprising a plurality of light emitters, wherein a first array of the light emitters is configured to form a pattern illumination source for emitting the infrared light pattern, and wherein a second array of the light emitters, which is different from the light emitters in the first array, is configured to form a flood illumination source for emitting the infrared floodlight.

[0184] - Base, which provides a single plane for mounting the light emitter;

[0185] - At least one optical element system, the at least one optical element system comprising a plurality of optical elements, wherein the optical element system is configured to focus the emitted infrared light pattern onto a focal plane, wherein the optical element system covers a light emitter structure;

[0186] - At least one floodlight optical element, the at least one floodlight optical element being configured to defocus light emitted by a light emitter of a floodlight illumination source, thereby forming an overlapping light spot, wherein the floodlight optical element is configured to ensure that the emitted infrared light pattern is unaffected.

[0187] Example 2. According to the optoelectronic device of Example 1, the first array and the second array are arranged side by side on the plane, or the cavities of the light emitters of the first array and the light emitters of the second array form a combined pattern, in which the cavities of the light emitters of the first array and the light emitters of the second array alternate, for example, row by row or line by line.

[0188] Example 3. The optoelectronic device according to any one of the foregoing embodiments, wherein the floodlight optical element comprises at least one element selected from the group consisting of: at least one plate with a refractive index greater than 1.4, such as a glass plate; at least one diffuser plate; at least one lens; at least one microlens; at least one prism; at least one Fresnel lens; at least one diffractive optical element (DOE); at least one superlens.

[0189] Example 4. The optoelectronic device according to any one of the foregoing embodiments, wherein the second array is completely covered by the floodlight optical element.

[0190] Example 5. The optoelectronic device according to any one of the foregoing embodiments, wherein the optoelectronic device includes a single floodlight optical element covering all light emitters of the second array, or wherein the optoelectronic device includes a plurality of floodlight optical elements, wherein each light emitter of the second array includes at least one assigned floodlight optical element.

[0191] Example 6. The optoelectronic device according to any one of the foregoing embodiments, wherein the optical element system includes one or more of the following: at least one refractive lens, a plurality of refractive lenses; at least one diffractive optical element (DOE), a plurality of DOEs, and a plurality of superlenses.

[0192] Example 7. The optoelectronic device according to any one of the foregoing embodiments, wherein the light emitter structure is located at the focal point of the optical element system.

[0193] Example 8. The optoelectronic device according to any one of the foregoing embodiments, wherein the optoelectronic device is included in a device, wherein the device includes a display, and the infrared light pattern passes through the display when illuminated from the pattern irradiation source, and / or the infrared floodlight passes through the display when illuminated from the floodlight irradiation source, wherein the display of the device is at least partially transparent in at least one continuous region covering the pattern irradiation source and the floodlight irradiation source.

[0194] Example 9. The optoelectronic device according to the previous embodiment, wherein the device is selected from the group consisting of: television equipment; game consoles; personal computers; mobile devices, particularly mobile phones, and / or smartphones, and / or tablet computers, and / or laptop computers, and / or tablet computers, and / or virtual reality devices, and / or wearable devices such as smartwatches; or other types of portable computers.

[0195] Example 10. An optoelectronic device according to any one of the foregoing two embodiments, wherein the device includes at least one image generation unit configured to generate at least one pattern image when the pattern illumination source emits the infrared light pattern, and configured to generate at least one floodlight image when the floodlight illumination source emits infrared floodlight, wherein the light emitter of the pattern illumination source and the light emitter of the floodlight illumination source are arranged at the optical center of the image generation unit.

[0196] Example 11. The optoelectronic device according to any one of the foregoing embodiments, wherein each of these light emitters includes at least one vertical cavity surface-emitting laser (VCSEL).

[0197] Example 12. The optoelectronic device according to any one of the foregoing embodiments, wherein the light emitter is configured to emit light in the near-infrared spectral range, preferably, the emitted light has a wavelength of 760 nm to 1.5 μm, preferably 940 nm, 1140 nm or > 1400 nm.

[0198] Example 13. The optoelectronic device according to any one of the foregoing embodiments, wherein the light emitter of the light emitter structure is arranged in one or more of the following: a grid pattern, a hexagonal pattern, a shifted hexagonal pattern, etc.

[0199] Example 14. The optoelectronic device according to any one of the foregoing embodiments, wherein the first array and the second array of the emitter are produced directly on the plane as a single die, or the first array and the second array of the emitter are produced separately and mounted side by side on the plane.

[0200] Example 15. The optoelectronic device according to any one of the foregoing embodiments, wherein the light emitters and / or the floodlight optical elements are mounted on the base, for example, by using at least one adhesive or at least one glue.

[0201] Example 16. The optoelectronic device according to any one of the foregoing embodiments, wherein the floodlight optical elements are mounted on the base and / or the light emitters, for example, by using at least one glue or at least one adhesive.

[0202] Example 17. The use of the optoelectronic device according to any one of the foregoing embodiments for authenticating users of devices including the device.

[0203] Example 18. A device for authenticating a user of a device to perform at least one operation requiring authentication on the device, the device comprising:

[0204] -The optoelectronic device according to any one of the foregoing embodiments relating to optoelectronic devices;

[0205] - At least one image generation unit, the at least one image generation unit being configured to generate at least one pattern image when the pattern illumination source emits an infrared light pattern, and being configured to generate at least one floodlight image when the floodlight illumination source emits infrared floodlight;

[0206] - At least one display, wherein the infrared light pattern passes through the display when illuminated from the pattern illumination source, and / or the infrared floodlight passes through the display when illuminated from the floodlight illumination source, wherein the display of the device is at least partially transparent in at least one continuous area covering the pattern illumination source, the floodlight illumination source, and / or the image generation unit.

[0207] - At least one authentication unit, which is configured to use the floodlight image and the pattern image to perform at least one authentication process for a user.

[0208] Example 19. A method for authenticating a user of a device to perform at least one operation requiring authentication on the device, wherein the device is the device according to the previous embodiment, and the method includes:

[0209] - Illuminate the user with an infrared light pattern from a pattern illumination source.

[0210] - Illuminate the user with infrared floodlight from a floodlight source.

[0211] - At least one pattern image is generated using the image generation unit, the at least one pattern image showing at least a portion of the user when the user is illuminated with the infrared light pattern, and at least one image is generated using the image generation unit, the at least one image showing at least a portion of the user when the user is illuminated with the infrared floodlight.

[0212] - Identify the user based on this floodlight image.

[0213] - Extract material data from the at least one pattern image; and

[0214] - Based on the material data and identification, allow the user to perform at least one operation on the device that requires authentication.

[0215] Example 20. A computer program including instructions that, when executed by a device according to Example 18, cause the device to perform the method according to the preceding embodiment.

[0216] Example 21. A computer-readable storage medium including instructions that, when executed by a device according to Example 18, cause the device to perform the method according to Example 19.

[0217] Example 22. A non-transient computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to Example 19. Attached Figure Description

[0218] Further optional features and embodiments will be disclosed in more detail, preferably in conjunction with the dependent claims, in the following embodiments. As those skilled in the art will recognize, the corresponding optional features can be implemented independently and in any feasible combination. The scope of the invention is not limited to the preferred embodiments. Embodiments are schematically depicted in the accompanying drawings. The same reference numerals in these drawings denote the same or functionally equivalent elements.

[0219] In the attached diagram:

[0220] Figure 1 An embodiment of the device is shown;

[0221] Figure 2 An embodiment of the method is shown;

[0222] Figure 3 A and Figure 3 B illustrates an embodiment of the optoelectronic device; and

[0223] Figure 4 A and Figure 4 B illustrates an embodiment of the optoelectronic device. Detailed Implementation

[0224] Figure 1 An embodiment of the device 110 of the present invention is shown in a highly illustrative manner. For example, device 110 may be selected from the group consisting of: television equipment; game consoles; personal computers; mobile devices, particularly mobile phones, and / or smartphones, and / or tablet computers, and / or laptop computers, and / or tablet computers, and / or virtual reality devices, and / or wearable devices such as smartwatches; or other types of portable computers. In this embodiment, device 110 includes an optoelectronic device 112 according to the present invention. Figure 3 A, Figure 3 B and Figure 4 A and Figure 4 B illustrates an embodiment of the photoelectric device 112.

[0225] like Figure 3 A, Figure 3 B and Figure 4 A and Figure 4 As shown in Figure B, the photoelectric device 112 includes:

[0226] - Light emitter structure 114, which includes a plurality of light emitters 116, wherein a first array 118 of the light emitters in the plurality of light emitters 116 forms a pattern illumination source 120 configured to emit infrared light patterns, wherein a second array 122 of the light emitters in the plurality of light emitters 116 that is different from the light emitters in the first array 118 forms a flood illumination source 124 configured to emit infrared floodlight;

[0227] - Base 125, which provides a single plane for mounting the light emitter 116;

[0228] - At least one optical element system 126, the at least one optical element system comprising a plurality of optical elements, wherein the optical element system 126 is configured to focus the emitted infrared light pattern onto a focal plane, wherein the optical element system 126 covers the light emitter structure 114;

[0229] - At least one floodlight optical element 130, which is configured to defocus the light emitted by the light emitter of the floodlight illumination source 124 to form an overlapping light spot, wherein the floodlight optical element 130 is configured to ensure that the emitted infrared light pattern is unaffected.

[0230] The base 125 may include multiple cavities into which the light emitter structure 114 can be mounted. The base 125 may have any shape, such as rectangular, circular, or hexagonal. The shape may refer to the side of the base oriented perpendicular to the measurement height. The base 125 may include at least one semiconductor substrate. The base 125 may be an element and / or additional element of the light emitter structure. The base 125 may be and / or may include a thermally conductive printed circuit board (PCB).

[0231] The light emitter 116 of the light emitter structure 114 can be formed into a light emitter chip, such as a VCSEL die, for example, a die sawn from a wafer. This light emitter 116 chip can be mounted on a base, for example, by using at least one thermally conductive adhesive. The base 125 can include at least one thermally conductive material. The base 125 can be the bottom of the optoelectronic device 112, for example, the bottom of the housing of the optoelectronic device 112. Therefore, the dimensions of the base 125 can be defined by the dimensions of the optics and the housing. Alternatively, the base 125 and the housing can be separate components. For example, the light emitter 116 chip can be mounted on the base 125 (e.g., a PCB), for example, by using at least one thermally conductive adhesive, and the housing can be applied to the combined component.

[0232] The base 125 may include at least one thermally conductive material, particularly multiple thermally conductive materials. The thermally conductive material may be configured as a heat exchanger. The thermally conductive material may be configured to regulate the temperature of the light emitter. The thermally conductive material may be configured to transfer heat generated by the light emitter out of the light emitter. For example, the thermally conductive material may include at least one composite material. The light emitter structure may be mounted on the thermally conductive material.

[0233] The light emitter 116 of the light emitter structure 114 can be mounted on the base 125, for example, by using at least one adhesive or at least one glue. For example, the adhesive can be at least one thermally conductive adhesive. For example, the light emitter 116 forming the patterned irradiation source 120 can be glued to the base 125. For example, the light emitter 116 forming the floodlight irradiation source 124 can be glued to the base 125.

[0234] Each of these optical emitters 116 may include at least one vertical-cavity surface-emitting laser (VCSEL). The optical emitter 116 may be configured to emit light in the near-infrared spectral range, preferably with a wavelength of 760 nm to 1.5 µm, preferably 940 nm, 1140 nm or > 1400 nm.

[0235] The light emitters 116 of the pattern illumination source 120 and the floodlight illumination source 124 can be activated at different times.

[0236] The light emitters 116 of the light emitter structure 114 can be arranged in a periodic pattern. The light emitters 116 of the light emitter structure 114 can be arranged in one or more of the following patterns: grid pattern, hexagonal pattern, shifted hexagonal pattern, etc. Multiple light emitters 116 of the light emitter structure 114 can form a first array 118 of light emitters 116, and multiple light emitters 116 in the light emitter structure 114 that are different from the light emitters 116 in the first array 118 form a second array 122 of light emitters 116. The light emitter structure 114 can include two light emitter arrays (118, 122), such as two VCSEL arrays. These arrays (118, 122) are located in a plane.

[0237] The first array 118 and the second array 122 of the transmitter 116 can be manufactured directly on a plane as a single die, or the first array 118 and the second array 122 of the transmitter 116 can be manufactured separately and, for example, mounted side by side on a plane. However, embodiments in which even more arrays are used (e.g., configured to provide different functions) are possible.

[0238] The optical element system 126 may include one or more of the following: at least one refractive lens, multiple refractive lenses; at least one diffractive optical element (DOE), multiple DOEs, and multiple metalenses. For example, the optical element system 126 may include at least one refractive lens and at least one optical element configured to increase (e.g., replicate) the number of light spots (e.g., light spots generated by a light emitter from a patterned illumination source). Specifically, the optical element system 126 may include at least one diffractive optical element (DOE) and / or at least one metasurface element. The DOE and / or metasurface element may be configured to generate multiple beams from a single incident beam. For example, a VCSEL projecting up to 2000 light spots and an optical element including multiple metasurface elements may be used to replicate the number of light spots. Further arrangements (particularly including different numbers of projecting VCSELs and / or at least one different optical element configured to increase the number of light spots) may be possible. Other multiplication factors are also possible. For example, one or more VCSELs may be used, and the generated laser light spots may be replicated by using at least one DOE.

[0239] Optical element system 126 covers light emitter structure 114. Optical element system 126 can be designed and / or arranged such that it covers light emitter structure 114. For example, the first array 118 can be covered only by optical element system 126. For example, both the first array 118 and the second array 122 can be covered by optical element system 126. Light emitter structure 114 can be located at the focal point of optical element system 126. This arrangement allows the emitted light from light emitter 116, particularly the light emitter 116 of patterned illumination source 120, to be collimated.

[0240] The floodlight optics 130 can be configured to defocus the light emitted by the light emitter 116 of the floodlight source 124, thereby forming overlapping light spots. The floodlight source 124 is configured, in combination with the floodlight optics 130, to generate floodlight, particularly diffuse illumination. The floodlight optics 130 may include at least one element selected from the group consisting of: at least one plate with a refractive index greater than 1.4, such as a glass plate; at least one diffuser plate; at least one lens; at least one microlens; at least one prism; at least one Fresnel lens; at least one diffractive optical element (DOE); at least one superlens. The second array 122 may be completely covered by the floodlight optics 130.

[0241] For example, the floodlight optical element 130 can be mounted on the base, for example, by using at least one adhesive or at least one glue. For example, the floodlight optical element 130 can be mounted on the base 125 and / or the light emitter 116, particularly the light emitter of the floodlight source 124, by using at least one adhesive or at least one glue.

[0242] Figure 3 A and Figure 3 B shows an example in which the first array 118 and the second array 122 can be arranged side by side, or in particular adjacent to each other, on a plane. Figure 3 A shows a top view of an exemplary layout of a light emitter structure 114. In this embodiment, the light emitter structure 114 may include two VCSEL arrays 118, 122. In this case, the VCSEL arrays 118, 122 are located side by side on a plane. Figure 3 B targets Figure 3 The layout of the light emitter structure 114 of A illustrates an embodiment of the operation of the optoelectronic device 112. Figure 3 To the left of B, the photoelectric device 112 is shown as having an activated irradiation source 120. Figure 3 To the right of B, the photoelectric device 112 is shown as having an activated floodlight source 124. The plane of the base 125 may first include a first array 118 along a direction perpendicular to the optical axis of the photoelectric device 112, followed by a second array 122. The VCSEL array 118 may define a dot pattern, such as a periodic regular pattern, like a simple grid pattern, a hexagonal pattern, a shifted hexagonal pattern, etc. The VCSEL array 118 may be located at the focal point of the optical element system 126, i.e., all cavities are collimated. The second array 122 may be responsible for floodlight illumination, particularly diffuse illumination. This array 122 may be completely covered by the floodlight optical element 130. Figure 3 As shown in Figure B, the second array 122 can also be located at the focal point of the optical element system 126. Therefore, the light generated by the second array 122 will also be focused. However, the addition of a floodlight optics alters the optical imaging, causing the cavity to be improperly collimated. This can allow for the generation of diffuse floodlight illumination. The floodlight optics 130 is configured to ensure that the emitted infrared light pattern is unaffected. The VCSEL array 118 may not be covered by this floodlight optics 130.

[0243] like Figure 3 As shown in Figure B, the optoelectronic device 112 includes a single floodlight optical element 130 covering all the light emitters of the second array 122. Other embodiments are possible, such as... Figure 4 As shown in B. For example, in Figure 4In B, the optoelectronic device 112 includes a plurality of floodlight optical elements 130. For example, each light emitter 116 of the second array 122 includes at least one assigned floodlight optical element 130.

[0244] Figure 4 A shows a top view of an exemplary layout of an optical emitter structure 114. In this embodiment, the optical emitter structure 114 may include two VCSEL arrays 118 and 122. The VCSEL arrays 118 and 122 are located on a single plane. Figure 4 A and Figure 4 B shows an example in which the cavities of the light emitters 116 of the first array 118 and the light emitters 116 of the second array 122 can form a combined pattern, in which the cavities of the light emitters 116 of the first array 118 and the light emitters 116 of the second array 122 alternate, for example, row by row. Figure 4 In A, the white circles represent cavities of illumination source 120, and the black circles represent cavities of floodlight illumination source 124, each cavity having an additional microlens. Figure 4 To the left of B, the photoelectric device 112 is shown as having an activated irradiation source 120. Figure 4 To the right of B, the photoelectric device 112 is shown as having an activated floodlight source 124.

[0245] Back Figure 1 Device 110 may include display 132, and infrared light pattern passes through display 132 when illuminated from pattern illumination source 120, and / or infrared floodlight passes through display 132 when illuminated from floodlight illumination source 124. Display 132 may be a front display of device 110. Display 132 may be or may include at least one organic light-emitting diode (OLED) display.

[0246] The device 110 includes at least one image generation unit 134 configured to generate at least one pattern image when the pattern illumination source 120 emits an infrared light pattern, and configured to generate at least one floodlight image when the floodlight illumination source 124 emits infrared floodlight. The image generation unit 134 may include at least one optical sensor, particularly at least one pixelated optical sensor. The image generation unit 134 may include at least one CMOS sensor or at least one CCD chip.

[0247] Device 110 includes at least one authentication unit 136 configured to perform at least one authentication process for a user using a floodlight image and a pattern image. Authentication unit 136 may include at least one processor.

[0248] Authentication may include verifying a user's identity. Specifically, authentication may include distinguishing a user from other humans or objects, particularly distinguishing authorized access from unauthorized access. Authentication may include verifying the identity of the corresponding user and / or assigning an identity to the user. Authentication may include generating and / or providing identity information, such as providing it to other devices or units (e.g., to at least one authorized unit) for authorizing access to that device. The identity information can be proven through authentication. For example, the identity information may be and / or may include at least one identity token. If authentication is successful, it may be verified that the facial image recorded by at least one image generation unit is the user's facial image, and / or the user's identity is verified. Authentication may be performed using at least one authentication process. The authentication process may include multiple steps, such as at least one face detection (e.g., on at least one floodlight image, as will be described in more detail below), and at least one recognition step, wherein an identity is assigned to the detected face and / or at least one identity check and / or verification of the user's identity is performed.

[0249] For example, authentication unit 136 can use a floodlight image to perform at least one face detection. Authentication unit 136 can be configured to identify a user based on the floodlight image. Authentication may include multiple steps.

[0250] For example, authentication may include performing face detection at least once using a floodlight image. Face detection may include analyzing the floodlight image. In particular, the analysis of the floodlight image may include using at least one image recognition technique, especially a face recognition technique. Image recognition techniques include at least one process of identifying a user in an image. Image recognition may include using at least one technique selected from the following: color-based image recognition, for example using features such as template matching; segmentation and / or connected component (blob) analysis, for example using size or shape; machine learning and / or deep learning, for example using at least one convolutional neural network.

[0251] For example, authentication may include identifying a user. Identification may include assigning an identity to a detected face and / or at least one identity check and / or verification of the user's identity. Identification may include performing facial verification on the imaged face to confirm whether it is the user's face. Identifying a user may include matching a flood image (e.g., showing the outlines of various parts of the user, particularly the outlines of various parts of the user's face) with a template (e.g., a template image generated during registration). Identifying a user may include determining whether the imaged face is the user's face, and in particular determining whether the imaged face corresponds to at least one image of the user's face stored in at least one memory of a device, for example. If the flood image can match the image template, authentication may be successful. If the flood image cannot match the image template, authentication may be unsuccessful.

[0252] For example, user identification may include determining multiple facial features. Analysis may include comparing the determined facial features with template features, specifically performing a matching process. Template features may be features extracted from at least one template. The template may be or may include at least one image generated during the registration process (e.g., when initializing the device). The template may be an image of an authorized user. Template features and / or facial features may include vectors. Feature matching may include determining the distance between vectors. User identification may include comparing the distance between the vectors with at least one predefined limit. If the distance is at least within tolerance and ≤ the predefined limit, the user can be successfully identified. Otherwise, the user may be rejected and / or refused.

[0253] For example, image recognition may include using at least one model, particularly a trained model that includes at least one face recognition model. Analysis of floodlight images can be performed using a face recognition system, such as FaceNet, as described, for example, in Florian Schroff, Dmitry Kalenichenko, and James Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering,” arXiv:1503.03832. The trained model may include at least one convolutional neural network. For example, a convolutional neural network may be designed as described in the following literature: MD Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” CoRR, abs / 1311.2901, 2013; or C. Szegedy et al., “Going deeper with convolutions,” CoRR, abs / 1409.4842, 2014. For more details on convolutional neural networks for face recognition systems, please refer to: Florian Schroff, Dmitry Kalenichenko, and James Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering,” arXiv:1503.03832. Labeled image data from image databases can be used as training data.Specifically, labeled faces can be used from one or more of the following sources: GB Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” Technical Report 07-49, University of Massachusetts Amherst, October 2007; the YouTube® Faces database as described in L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrained videos with matched background similarity,” IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2011; or the Google® Facial Expression Comparison Dataset. Training of convolutional neural networks can be described as in "FaceNet: A Unified Embedding for Face Recognition and Clustering" by Florian Schroff, Dmitry Kalenichenko, and James Philbin, arXiv:1503.03832.

[0254] The authentication unit 136 may be further configured to consider additional security features extracted from the pattern image. Specifically, the authentication unit 136 may be further configured to extract material data from the pattern image. The authentication unit 136 may be configured to authenticate a user if the user can be identified and / or if the material data matches expected material data. Authentication may include the use of even additional security features, such as three-dimensional information and / or additional liveness data.

[0255] Authentication 136 may include the use of even additional security features, such as 3D information and / or additional liveness data.

[0256] For example, beam profiling analysis can be used to determine three-dimensional information (e.g., the longitudinal coordinate z of one or more points on a user's face), as described, for example, in WO 2018 / 091640 A1, the entire contents of which are incorporated herein by reference. For example, authentication unit 136 can be configured to determine at least one longitudinal coordinate z by evaluating a quotient signal Q of sensor signals detected by image generation unit 134. The quotient signal Q can be generated by combining sensor signals, particularly by one or more of the following: dividing sensor signals, dividing sensor signals by multiples, or dividing a linear combination of sensor signals. Authentication unit 136 can be configured to determine the longitudinal coordinate using at least one predetermined relationship between the quotient signal Q and the longitudinal coordinate z. For example, authentication unit 136 is configured to obtain the quotient signal Q by the following formula:

[0257]

[0258] Where x and y are the lateral coordinates, A1 and A2 are the regions of the beam profile at the sensor location, and E(x,y,z) o ) indicates the distance z between objects o The beam profile is given below. Regions A1 and A2 may differ. In particular, A1 and A2 are not identical. Therefore, A1 and A2 may differ in one or more aspects of shape or content. For further details on determining the longitudinal coordinate z using beam profile analysis, refer to WO 2018 / 091640 A1, the entire contents of which are incorporated herein by reference.

[0259] The authentication unit 136 can be configured to determine the longitudinal coordinates z at multiple locations on the user's face and to determine a depth map. The determined depth map can be compared with a predetermined depth map of the user, for example, determined during the registration process. The authentication unit 136 can be configured to authenticate the user if the determined depth map matches (in particular, at least within tolerance) the user's predetermined depth map. Otherwise, the user can be rejected.

[0260] Device 110 may include at least one authorization unit, which in Figure 1The authorization unit is designed to be integrated with the authentication unit 136. This at least one authorization unit is configured to allow a user to perform at least one operation on the device 110, such as unlocking the device 110, if the user is successfully authenticated, or to deny the user from performing at least one operation on the device 110 if authentication is unsuccessful. Thus, the user is aware of the authentication result. The authorization unit can be configured to allow or deny the user to perform at least one operation requiring authentication on the device based on material data and identification using floodlight images. The authorization unit can be configured to allow or deny the user access to one or more functions associated with the device, depending on authentication or denial. Allowing may include granting permission to access the one or more functions. The authorization unit can be configured to determine whether the user corresponds to an authorized user, wherein allowing or denying is further based on determining whether the user corresponds to an authorized user. Authorization may include assigning access rights to the user, particularly selective permission or selective restriction of access to the device 110 and / or at least one resource of the device 110. The authorization unit can be configured for access control. The authorization unit may include at least one processor or may be designed as software or an application. The authorization unit and the authentication unit may be integrated, for example, by using the same processor. The authorization unit can be configured to allow a user to access one or more functions on device 110, such as unlocking the device, if the user is successfully authenticated, or to deny the user access to one or more functions on device 110, such as unlocking the device, if the user is not successfully authenticated.

[0261] Device 110 (e.g., by using display 132) can be configured to display the results of authentication and / or authorization, for example by using at least one communication interface (e.g., a user interface, such as a display).

[0262] exist Figure 2 In the middle, a type of authentication device 110 (especially as regarding Figure 1 , Figure 3 and Figure 4 The described method for a user of device 110 to perform at least one operation requiring authentication on device 110. The method includes:

[0263] - (138) Illuminate the user with an infrared light pattern from pattern illumination source 120.

[0264] - (140) Illuminate the user with infrared floodlight from floodlight source 124,

[0265] - (142) At least one pattern image is generated using the image generation unit 134, the at least one pattern image showing at least a portion of the user when the user is illuminated with an infrared light pattern, and at least one floodlight image is generated using the image generation unit 134, the at least one floodlight image showing at least a portion of the user when the user is illuminated with infrared floodlight.

[0266] - (144) Identify the user based on the floodlight image,

[0267] - (146) Extract at least one security feature from the at least one pattern image; and

[0268] - (148) Based on the security feature and the identification, allow or deny the user to perform at least one operation that requires authentication on device 110.

[0269] These method steps can be executed in a given order or in a different order. Furthermore, there may be one or more additional method steps not listed. Furthermore, one, more than one, or even all of the method steps may be executed repeatedly.

[0270] Extracting security features includes extracting material data from the at least one pattern image. The method may include allowing or denying a user to perform at least one authentication-required operation on the device based on the material data and identification and / or even additional security features (e.g., 3D information and / or additional liveness data).

[0271] List of reference numerals

[0272] .

Claims

1. A photoelectric device (112) configured to emit at least one infrared light pattern comprising a plurality of infrared beams and to emit infrared floodlight, the photoelectric device comprising: - A light emitter structure (114) comprising a plurality of light emitters (116), wherein a first array (118) of the plurality of light emitters (116) forms a pattern irradiation source (120) configured to emit the infrared light pattern, wherein a second array (122) of the plurality of light emitters (116) different from the light emitters (116) of the first array (118) forms a flood irradiation source (124) configured to emit the infrared floodlight. - Base (125), which provides a single plane for mounting these light emitters (116); - At least one optical element system (126), the at least one optical element system comprising a plurality of optical elements, wherein the optical element system (126) is configured to focus the emitted infrared light pattern onto a focal plane, wherein the optical element system (126) covers the light emitter structure (114). - At least one floodlight optical element (130) configured to defocus the light emitted by the light emitter (116) of the floodlight source (124) to form an overlapping light spot, wherein the floodlight optical element (130) is configured to ensure that the emitted infrared light pattern is unaffected.

2. The photoelectric device (112) according to claim 1, wherein, The first array (118) and the second array (122) are arranged side by side on the plane, or in which, The cavities of the light emitters (116) of the first array (118) and the light emitters (116) of the second array (122) form a combined pattern in which the cavities of the light emitters of the first array (118) and the light emitters of the second array (122) alternate, for example, row by row or row by row.

3. The photoelectric device (112) according to any one of the preceding claims, wherein, The floodlight optical element (130) includes at least one element selected from the group consisting of: at least one plate with a refractive index greater than 1.4, at least one diffuser plate, at least one lens, at least one microlens, at least one prism, at least one Fresnel lens, at least one diffractive optical element (DOE), at least one superlens, and / or wherein, The optical element system (126) includes one or more of the following: at least one refracting lens, multiple refracting lenses; at least one diffractive optical element (DOE), multiple DOEs, and multiple superlenses.

4. The photoelectric device (112) according to any one of the preceding claims, wherein, The optoelectronic device (112) includes a single floodlight optical element (130) covering all the light emitters (116) of the second array (122), or wherein, The optoelectronic device (112) includes a plurality of floodlight optical elements (130), wherein each light emitter (116) of the second array (122) includes at least one assigned floodlight optical element (130).

5. The photoelectric device (112) according to any one of the preceding claims, wherein, The light emitter structure (114) is located at the focal point of the optical element system (126).

6. The photoelectric device (112) according to any one of the preceding claims, wherein, The optoelectronic device (112) is included in the device (110), wherein the device (110) includes a display (132), and the infrared light pattern passes through the display (132) when illuminated from the pattern irradiation source (120), and / or the infrared floodlight passes through the display (132) when illuminated from the floodlight irradiation source (124), wherein the display (132) of the device (110) is at least partially transparent in at least one continuous area covering the pattern irradiation source (120) and the floodlight irradiation source (124).

7. The photoelectric device (112) according to the preceding claim, wherein, The device (110) is selected from the group consisting of: television equipment; game consoles; personal computers; mobile devices, especially mobile phones, and / or smartphones, and / or tablet computers, and / or laptop computers, and / or tablet computers, and / or virtual reality devices, and / or wearable devices such as smartwatches; or other types of portable computers.

8. The photoelectric device (112) according to any one of the preceding claims, wherein, Each of these optical emitters (116) includes at least one vertical cavity surface-emitting laser (VCSEL).

9. The photoelectric device (112) according to any one of the preceding claims, wherein, These light emitters (116) are configured to emit light in the near-infrared spectral range, preferably with a wavelength of 760 nm to 1.5 µm, preferably 940 nm, 1140 nm or > 1400 nm.

10. The photoelectric device (112) according to any one of the preceding claims is used for authenticating the user of the device (110), the device including the device (112).

11. A device (110) for authenticating a user of a device (110) to perform at least one operation requiring authentication on the device (110), the device (110) comprising: - The optoelectronic device (112) according to any one of the preceding claims relating to optoelectronic devices; - At least one image generation unit (134) is configured to generate at least one pattern image when the pattern illumination source (120) emits an infrared light pattern, and is configured to generate at least one floodlight image when the floodlight illumination source (124) emits infrared floodlight; - At least one display (132), wherein the infrared light pattern passes through the display (132) when illuminated from the pattern illumination source (120), and / or the infrared floodlight passes through the display (132) when illuminated from the floodlight illumination source (124), wherein the display (132) of the device (110) is at least partially transparent in at least one continuous area covering the pattern illumination source (120), the floodlight illumination source (124), and / or the image generation unit (134). - At least one authentication unit (136) configured to use the floodlight image and the pattern image to perform at least one authentication process for a user.

12. A method for a user of an authentication device (110) to perform at least one operation requiring authentication on the device (110), wherein, The device (110) is the device according to the preceding claim, and the method includes: - (138) Illuminate the user with an infrared light pattern from the pattern illumination source (120), - (140) Illuminate the user with infrared floodlight from the floodlight source (124), - (142) The image generation unit (134) generates at least one pattern image showing at least a portion of the user when the user is illuminated by the infrared light pattern, and the image generation unit (134) generates at least one image showing at least a portion of the user when the user is illuminated by the infrared floodlight. - (144) Identify the user based on the floodlight image, - (146) Extract material data from the at least one pattern image; and - (148) Based on the material data and the identification, allow the user to perform at least one operation on the device that requires authentication.

13. A computer program comprising instructions that, when executed by the device (110) according to claim 11, cause the device (110) to perform the method according to the preceding claim.

14. A computer-readable storage medium comprising instructions that, when executed by the device (110) according to claim 11, cause the device (110) to perform the method according to claim 12.

15. A non-transient computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 12.

Images