Facial authentication reference measurement timing

Abstract

A device (110) for authenticating a user (112) is proposed. The device (110) comprises:-at least one projector (116) configured for projecting a plurality of light beams onto the user (112) through at least one display (118),-at least one image generation unit (122) configured for generating a pattern image showing a projection of the plurality of light beams on the user (112); -at least one processor (124) configured for extracting living data from the pattern image and allowing the user (112) to perform an operation requiring authentication on the device (110) based on the living data; -at least one control unit (126) and at least one state query device (128), which is configured for retrieving at least one item of state information about a current environmental state of the device (110), the control unit (126) is configured to automatically trigger at least one calibration depending on the fulfillment of at least one predetermined environmental state condition.

Description

Technical Field

[0001] This invention relates to a device for authenticating users and a method for performing at least one reference measurement. The invention further relates to a computer program, a computer-readable storage medium, and a non-transient computer-readable medium. The device, method, and uses according to the invention are specifically applicable, for example, to various fields or sciences including daily life, security technology, gaming, transportation technology, production technology, photography (e.g., digital or video photography for artistic, documentation, or technical purposes), safety technology, information technology, agriculture, crop protection, maintenance, cosmetics, and medical technology. However, other applications are also possible. Background Technology

[0002] Available authentication systems in mobile devices (such as smartphones, tablets, etc.) include a receiver, such as at least one camera. The mobile device typically has a display, such as an organic light-emitting diode (OLED) area and / or a quantum dot light-emitting diode (QLED) area. The receiver can be positioned behind the display. Furthermore, such devices for authentication also use light emitters, such as a projector, or one or more light-emitting diodes and / or lasers, which can be positioned behind the display. Typically, the light emitter projects a pattern, such as a dot pattern, onto a target (e.g., a face), the receiver (e.g., a camera) captures the image projected onto the user, and a processor determines material information. If the material is classified as skin, it is identified as human; otherwise, it is identified as a spoofed target. The pattern generated by the light projector can be designed for 3D algorithms; that is, the pattern can be designed to easily solve the so-called correspondence problem. For smartphone applications, the resulting 3D depth map can be used for further facial authentication.

[0003] Tolerances are introduced during the manufacturing of the sensor head for such authentication systems and its integration into mobile devices. For distance sensing using mobile devices, calibration steps for the sensor head and / or the finished mobile device are necessary. This is particularly critical for integration behind OLED screens, as sensor head placement strongly influences diffraction characteristics. Typical calibration steps may involve capturing images of a white, flat surface at different distances. However, calibration processes are time-consuming and require machinery, making them costly. Furthermore, dimensional variations due to warping or other stresses on the mobile device and / or its components can degrade facial authentication or distance sensing performance throughout the product's lifespan. Additionally, it is desirable to avoid users performing calibration tasks at regular intervals upon initial product activation or throughout the product's lifespan. While this may sometimes be necessary for certain sensors (e.g., white balance in cameras, magnetometer calibration in compasses in smartphones), the task can be considered cumbersome and carries the risk of performance issues if performed improperly.

[0004] WO 2022 / 253777A1 describes a detector for determining the location of at least one object. The detector includes: - at least one projector for illuminating the object with at least one illumination pattern, wherein the illumination pattern includes a plurality of illumination features; - at least one sensor element having a matrix of optical sensors, each of which has a photosensitive region, wherein each optical sensor is designed to generate at least one sensor signal in response to illumination of a corresponding photosensitive region of the optical sensor by a reflected beam propagating from the object to the detector, wherein the sensor element is configured to determine at least one reflection image including a plurality of reflection features, wherein each of the reflection features includes a beam profile; - at least one evaluation device configured to determine initial distance information of the reflection features by analyzing the respective beam profiles of the reflection features, wherein analyzing the beam profile includes evaluating a combined signal Q from the respective sensor signals, wherein the evaluation device is configured to perform a calibration method including: a) matching the reflection features with reference features of a reference image, taking into account the initial distance information, thereby determining a matching pair of reflection features and reference features; b) For each of these matching reflection feature and reference feature pairs, determine the epipolar line of the matching reference feature in the reference image; c) determine the epipolar line distance d from the matching reflection feature to the epipolar line; d) evaluate these epipolar line distances d as a function of the image position (x,y) in the reference image to determine a geometric pattern; e) determine at least one correction for rotation and / or translation of the reflection image based on the geometric pattern.

[0005] Guoqiang Yang et al. described Under Screen Face Authentication (USFA) in “An Integrated Solution for Under Screen Face Authentication”, SID SYMPOSIUM DIGEST OFTECHNICAL PAPERS, WILEY-BLACKWELL Publishing, USA, Vol. 54, August 3, 2023, pp. 323-326, XP072509100, ISSN: 0097-966X, DOI: 10.1002 / SDTP.16294. The problem to be solved

[0006] Therefore, the object of the present invention is to provide devices and methods that address the aforementioned technical challenges of known devices and methods. Specifically, the object of the present invention is to provide devices and methods that allow for reduced costs of factory calibration and the calibration of 3D imagers behind OLED screens in fully assembled mobile devices. Summary of the Invention

[0007] This problem is solved by a device for authenticating a user, having the features of the independent claims, and a method for performing at least one reference measurement. Advantageous embodiments that can be implemented independently or in any arbitrary combination are set forth in the dependent claims and throughout the specification.

[0008] In a first aspect of the invention, a device for authenticating users is disclosed. The device includes:

[0009] - At least one projector configured to project multiple beams of light onto the user through at least one display.

[0010] - At least one image generation unit configured to generate a patterned image showing the projection of the plurality of light beams onto the user;

[0011] - At least one processor, configured to extract liveness data from the pattern image and, based on the liveness data, allow the user to perform an operation requiring authentication on the device;

[0012] - At least one control unit and at least one status query device, the at least one status query device being configured to retrieve at least one status information about the current environmental state of the device, wherein the control unit is configured to automatically trigger at least one calibration depending on the satisfaction of at least one predetermined environmental state condition.

[0013] The device can be selected from the group consisting of: television equipment; game consoles; personal computers; mobile devices, particularly mobile phones, and / or smartphones, and / or tablet computers, and / or laptop computers, and / or tablet computers, and / or virtual reality devices, and / or wearable devices such as smartwatches; or other types of portable computers. Specifically, the device can be a portable device. As used herein, the term "portable" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. The term can specifically refer to, but is not limited to, the characteristic that at least one object can be moved by human movement (e.g., by a single user). Specifically, the weight of an object characterized by the term "portable" may not exceed 10 kg, specifically not exceed 5 kg, more specifically not exceed 1 kg, or even not exceed 500 g. Additionally or alternatively, the size of an object characterized by the term "portable" may allow the object to extend no more than 0.3 m in any dimension, specifically not exceed 0.2 m in any dimension. Specifically, the volume of the object may not exceed 0.03 m³, specifically not exceed 0.01 m³, or even not exceed 0.001 m³.

[0014] As used herein, the term "user" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, a person who intends to use the device and / or uses the device.

[0015] As used herein, the term "authentication" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, verifying the identity of a user. Specifically, authentication may include distinguishing a user from other humans or objects, particularly distinguishing authorized access from unauthorized access. Authentication may include verifying the identity of the corresponding user and / or assigning an identity to the user. Authentication may include generating and / or providing identity information, for example, providing it to other devices or units (e.g., providing it to at least one authorized unit) for authorizing access to that device. Identity information can be proven through authentication. For example, identity information may be and / or may include at least one identity token. If authentication is successful, it can be verified that the facial image recorded by at least one image generation unit is the user's facial image, and / or the user's identity is verified. Authentication can be performed using at least one authentication process. The authentication process may include multiple steps, such as at least one face detection (e.g., on at least one floodlight image, as will be described in more detail below), and at least one recognition step, wherein an identity is assigned to the detected face and / or at least one identity check and / or verification of the user's identity is performed.

[0016] As used herein, the term "light" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, electromagnetic radiation in one or more of the infrared, visible, and ultraviolet spectral ranges. In this document, the term "ultraviolet spectral range" generally refers to electromagnetic radiation with wavelengths from 1 nm to 380 nm, preferably from 100 nm to 380 nm. Further, in part according to the standard ISO-21348, the effective version of this document as of the date of this document, the term "visible spectral range" generally refers to the spectral range from 380 nm to 760 nm. The term "infrared spectral range" (IR) generally refers to electromagnetic radiation from 760 nm to 1000 µm, wherein the range from 760 nm to 1.5 µm is generally referred to as the "near-infrared spectral range" (NIR), the range from 1.5 µm to 15 µm is referred to as the "mid-infrared spectral range" (MidIR), and the range from 15 µm to 1000 µm is referred to as the "far-infrared spectral range" (FIR). Preferably, the light used for the typical purposes of the present invention is light in the infrared (IR) spectral range, more preferably light in the near-infrared (NIR) and / or mid-infrared spectral range (MidIR). Specifically, the projector projects at least one infrared light pattern, wherein the projected beam has a wavelength preferably from 800 nm to 1300 nm, more preferably from 900 nm to 1000 nm, and most preferably from 1100 nm to 1200 nm in the infrared spectral range.

[0017] As used herein, the term "ray" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, a line perpendicular to the wavefront of light and pointing in the direction of energy flow. As used herein, the term "beam" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, a collection of rays. In the following text, the terms "ray" and "beam" will be used as synonyms. As used herein, the term "beam" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, a quantity of light, specifically a quantity of light traveling substantially in the same direction, including the possibility that the beam has an extension angle or widening angle. A beam may have spatial extension. Specifically, a beam may have a non-Gaussian beam profile. The beam profile may be selected from the group consisting of: trapezoidal beam profile; triangular beam profile; conical beam profile. The trapezoidal beam profile may have a raised platform region and at least one edge region. Specifically, the beam may be a Gaussian beam or a linear combination of Gaussian beams, as will be further detailed below. However, other embodiments are also possible.

[0018] As used herein, the term "projection" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, the process of providing at least one light beam (particularly a light pattern) onto at least one surface. As used herein, the term "projector" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, an optical device configured to project at least one light beam onto a surface. A projector is configured to project multiple light beams. These multiple light beams can form a light pattern. A projector can be configured to generate and / or provide at least one light pattern, particularly at least one infrared light pattern.

[0019] As used herein, the term "light pattern" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, at least one arbitrary pattern comprising multiple light spots. The light spots may extend at least partially in space. At least one light spot, or any light spot, may have any shape. In some cases, a circular shape of at least one light spot, or any light spot, may be preferred. The arrangement of the light spots can be considered in relation to the structure of the display. Typically, the arrangement of the OLED pixel structure of the display can be considered.

[0020] The light pattern can be an infrared light pattern. As used herein, the term "infrared light pattern" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term can refer to, but is not limited to, light patterns containing spots within the infrared spectral range. An infrared light pattern can also be a near-infrared light pattern.

[0021] The light projected by the projector can be coherent. The light pattern can be a coherent light pattern, particularly an infrared light pattern. The projector can be configured to emit light of a single wavelength, such as light in the near-infrared region. In other embodiments, the projector can be adapted to emit light with multiple wavelengths, for example, to allow for additional measurements in other wavelength channels.

[0022] The light pattern may include at least one regular and / or constant and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or a pattern including further embossed patterns. For example, the light pattern is a hexagonal pattern, preferably a hexagonal infrared pattern, and preferably a 2 / 5 hexagonal infrared pattern. Using a periodic 2 / 5 hexagonal pattern allows for the differentiation of artifacts from usable signals.

[0023] A light pattern may include at least one dot pattern.

[0024] At least one of the light spots can be associated with a beam divergence of 0.2° to 0.5°, preferably 0.1° to 0.3°. As used herein, the term "beam divergence" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, at least one measure in which at least one diameter and / or at least one diameter equivalent (e.g., radius) increases with distance from the optical aperture from which the beam exits. The measure may be an angle or an angle equivalent. In the context of this invention, typically, beam divergence can be determined in the range of 1 / e 2 .

[0025] A projector may include at least one emitter, and in particular multiple emitters. As used herein, the term "emitter" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, at least one arbitrary device configured to provide at least one beam. The emitter may be selected from the group consisting of: at least one laser source, such as at least one semiconductor laser, at least one dual heterostructure laser, at least one external cavity laser, at least one independently confined heterostructure laser, at least one quantum cascade laser, at least one distributed Bragg reflector laser, at least one polaron laser, at least one hybrid silicon laser, at least one extended cavity diode laser, at least one quantum dot laser, at least one bulk Bragg grating laser, at least one indium arsenide laser, at least one gallium arsenide laser, at least one transistor laser, at least one diode-pumped laser, at least one distributed feedback laser, at least one quantum well laser, at least one interband cascade laser, at least one semiconductor ring laser, at least one vertical-cavity surface-emitting laser (VCSEL); at least one non-laser source, such as at least one LED or at least one bulb; at least one edge-emitting laser.

[0026] For example, a projector includes at least one VCSEL, preferably multiple VCSELs. These multiple VCSELs can be arranged in at least one array, such as a VCSEL matrix. The VCSELs can be arranged on the same substrate or different substrates. As used herein, the term "vertical-cavity surface-emitting laser" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. The term can specifically refer to, but is not limited to, a semiconductor laser diode configured to emit a laser beam perpendicularly to its top surface. Examples of VCSELs can be found, for example, at en.wikipedia.org / wiki / Verticalcavity_surface-emitting_laser. VCSELs are generally known to those skilled in the art, for example, from WO 2017 / 222618 A. Each of the VCSELs is configured to generate at least one beam. A VCSEL or the multiple VCSELs can be configured to generate a desired number of light spots. A VCSEL can be configured to emit a beam with a wavelength range of 800 nm to 1000 nm. For example, a VCSEL can be configured to emit beams of 808 nm, 850 nm, 940 nm, and / or 980 nm. Preferably, the VCSEL emits light at 940 nm because ground solar radiation has a local minimum of irradiance at that wavelength, as described, for example, in CIE 085-1989 "Solar spectral irradiance".

[0027] The display can be configured to modify the light spot generated by the projector as it passes through the display, for example, by increasing the number of light spots. For example, the display can act as a diffractive optical element (DOE). Therefore, since the display acts as a DOE, eliminating the need for further DOEs, resources can be saved. Furthermore, since the beam can be multiplied by the display, fewer emitters, such as VCSEL cavities, are required. Alternatively or additionally, the projector may include at least one optical element, configured to modify the light spot, for example, by increasing the number of light spots, selected from the group consisting of at least one lens; at least one microlens array (MLA); at least one diffractive optical element (DOE); and at least one metasurface element. The DOE and / or metasurface element can be configured to generate multiple beams from a single incident beam. For example, a VCSEL projecting up to 2000 light spots and an optical element including multiple metasurface elements can be used to replicate the number of light spots. Further arrangements (particularly including different numbers of projecting VCSELs and / or at least one different optical element configured to increase the number of light spots) are possible. Other multiplication factors are also possible. For example, one or more VCSELs can be used, and the generated laser spot can be replicated by using at least one DOE.

[0028] The projector includes at least one transmission device. As used herein, the term "transmission device," also referred to as "transmission system," is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, one or more optical elements suitable for modifying a beam, particularly for generating at least a portion of an infrared light pattern, by means of modifying one or more of the beam parameters, beam width, or beam direction of the beam. The transmission device may include at least one imaging optical device. Specifically, the transmission device may include one or more of the following: at least one lens, for example, at least one lens selected from the group consisting of at least one focusing lens, at least one aspherical lens, at least one spherical lens, and at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflection element, preferably at least one reflector; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitting mirror; at least one multi-lens system; at least one holographic optical element; at least one meta-optical element. Specifically, the transmission device includes at least one stack of refractive optical lenses. The transmission device may include a multi-lens system with refractive properties.

[0029] The device may further include at least one floodlight source configured to emit floodlight. An image generation unit may be configured to generate at least one floodlight image when the floodlight source emits floodlight. As used herein, the term "floodlight source" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, at least one arbitrary device configured to provide substantially continuous spatial illumination. As used herein, the term "floodlight" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, substantially continuous spatial illumination, particularly diffuse and / or uniform illumination. Floodlight has wavelengths in the infrared range, particularly in the near-infrared range. The floodlight source may include at least one LED or at least one VCSEL, preferably multiple VCSELs. The multiple VCSELs may overlap on a uniform area. As used herein, the term "substantially continuous spatial illumination" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term can specifically refer to, but is not limited to, uniform spatial illumination, where non-uniform areas are possible. An area illuminated from a floodlight source, for example, covering a user, a portion of the user, and / or the user's face, can be continuous. Power can be distributed across the entire illumination field. In contrast, illumination provided by a light pattern can include at least two continuous areas, particularly multiple continuous areas, and / or power can be concentrated in a smaller area of ​​the illumination field (compared to the entire illumination field). Infrared floodlight illumination can be suitable for illuminating continuous areas, particularly a single continuous area. Infrared pattern illumination can be suitable for illuminating at least two continuous areas.

[0030] A floodlight source can illuminate a measurement area, such as a user, a portion of the user, and / or the user's face, with a substantially constant intensity. As used herein, the term "constant" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, the temporal aspect during the exposure time. Floodlight can vary over time and / or can be substantially constant over time. As used herein, the term "substantially constant" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, completely constant illumination, and embodiments that allow deviations from constant illumination of ≤ ± 10%, preferably ≤ ± 5%, more preferably ≤ ± 2%.

[0031] The emission of the floodlight and the illumination of the light pattern can be performed subsequently or at least at a time that overlaps. For example, the floodlight and the light pattern can be emitted simultaneously. For example, one of the floodlight or the light pattern can be emitted at a lower intensity compared to the other.

[0032] The projector and floodlight source may include at least one VCSEL, preferably multiple VCSELs. The projector may include multiple first VCSELs mounted on a first platform. The floodlight source may include multiple second VCSELs mounted on a second platform. The second platform may be adjacent to the first platform. The projector may include a heat sink. Above the heat sink, a first increment including the first platform may be attached. Above the heat sink, a second increment including the second platform may be attached. The second increment may be different from the first increment. Thus, the first platform may be further away from the optical element configured to increase (e.g., replicate) the number of light spots. The second platform may be closer to the optical element. The beam emitted from the second VCSELs may be defocused, and thus form overlapping light spots. This results in substantially continuous illumination, and therefore floodlight illumination.

[0033] As used herein, the term "display" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, a device of any shape configured for displaying information. This information can be any type of information, such as at least one image, at least one chart, at least one histogram, at least one graphic, text, numbers, at least one symbol, operation menus, etc. A display may be or may include at least one display panel. A display may have any shape, such as a rectangular shape. A display may be the front display of a device. A display may include at least one of a display panel (particularly comprising multiple pixels and / or multiple transistors) or glass (specifically, a covering glass, particularly configured to cover the display panel).

[0034] The display, specifically the display panel, may be or may include at least one organic light-emitting diode (OLED) display and / or at least one quantum dot light-emitting diode (QLED). As used herein, the term "organic light-emitting diode" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, a light-emitting diode (LED), wherein the emitting electroluminescent layer is an organic compound film configured to emit light in response to an electric current. OLED displays may be configured to emit visible light. As used herein, the term "organic light-emitting diode" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, display technologies that utilize semiconductor particles called quantum dots to generate colors on a display. These quantum dots, when excited by light, can emit a variety of different colors of light depending on their size. By using combinations of red, green, and / or blue quantum dots, QLED displays can display a variety of colors with high brightness and color accuracy.

[0035] The display may be at least partially transparent. The display may be at least partially transparent over at least one continuous area covering the projector, floodlight source, and / or image generation unit. The display may have a transmittance of less than or equal to 20%, preferably less than or equal to 15%, more preferably less than or equal to 10%. For example, the intensity of a light beam after it has been projected through the display may correspond to ≤ 10% of the intensity associated with the light beam at the time of emission.

[0036] The display may be at least partially transparent in at least one continuous area such that at least one of the following is true:

[0037] - The light pattern incident on a continuous area passes through the display while being illuminated from the projector;

[0038] - Floodlight incident on a continuous area passes through the display while being illuminated from the floodlight source;

[0039] User light, generated by the light pattern and / or floodlight incident on the user and incident on a continuous area, passes through the display and strikes the image generation unit.

[0040] As used herein, the term "at least partially transparent" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, the property of a display to allow light, particularly light of a specific wavelength range (e.g., light in the infrared spectral region, particularly light in the near-infrared spectral region), to pass through at least partially. For example, a display may be translucent in the near-infrared region. For example, a display may have 20% to 50% transparency in the near-infrared region. A display may have different transparency for different wavelength ranges. The present invention may propose an apparatus comprising an image generating unit and a projector that can be placed behind the display of the apparatus. The transparent areas(s) of the display may allow operation of the image generating unit and the projector behind the display.

[0041] As described above, the display can be at least partially transparent. A continuous, partially transparent area of ​​the display can be associated with a first pixel density value (pixels per inch (PPI)), and another area of ​​the display can be associated with a second pixel density value. The first pixel density value can be lower than the second pixel density value. Light transmission through the continuous area can be higher than transmission through the other area. The first pixel density value can be equal to or lower than 450 PPI, preferably between 300 and 440 PPI, more preferably between 350 and 450 PPI. The first pixel density value can remain constant over the entire continuous area, with a maximum deviation of 20% or preferably 10%. The second pixel density value can be between 400 and 500 PPI, preferably between 450 and 500 PPI.

[0042] A continuous, at least partially transparent region of the display may include a first region and a second region. The first region may be associated with a first number of transistors configured to control at least one pixel, and the second region may be associated with a second number of transistors configured to control at least one pixel, wherein the first number of transistors may be less than the second number of transistors. The first number of transistors and / or the second number of transistors may refer to, or be, the density of these transistors.

[0043] As used herein, the term "pixel" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, a picture unit representing an addressable element, particularly the smallest picture unit. The collective of these pixels may represent a display. Pixels can be manipulated by changing their color, brightness, and / or contrast, etc. In particular, for the purpose of manipulating a pixel, it may be driven by at least one transistor, exemplarily by a transistor that controls the current required to drive the pixel. Typically, thin-film transistors (TFTs) can be used to drive pixels. TFTs are preferably used in flat panel displays.

[0044] An image generation unit is configured to generate a patterned image showing the projections of the plurality of light beams onto the user. As used herein, the term "image generation unit" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, at least one unit of the device configured to generate at least one image. The image may be generated via a hardware and / or software interface, which may be considered the image generation unit. As used herein, the terms "image generation" or "imaging" are broad terms and will be given their common and conventional meaning to those skilled in the art and are not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, capturing and / or generating and / or determining and / or recording at least one image using an image generation unit. Image generation may include imaging and / or recording images. Image generation may include capturing a single image and / or multiple images, such as an image sequence. For the purpose of generating images via a hardware and / or software interface, the capture and / or generation and / or determination and / or recording of images may be caused and / or initiated by the hardware and / or software interface. For example, image generation may include continuously recorded image sequences, such as videos or movies. Image generation may be initiated by user action or may be initiated automatically, for example, when at least one object or user is automatically detected within and / or a predetermined area of ​​the field of view of the image generation unit. As used herein, the term "field of view" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a particular or custom meaning. The term may specifically refer to, but is not limited to, the angular range of the observable world and / or at least one scene that can be captured or viewed by an optical system (such as the image generation unit). The field of view is typically expressed in degrees and / or radians, and may exemplary represent the total angle spanned by the image and / or the visible area.

[0045] The image generation unit may include at least one optical sensor, particularly at least one pixelated optical sensor. The image generation unit may include at least one CMOS sensor or at least one CCD chip. For example, the image generation unit may include at least one CMOS sensor that may be sensitive in the infrared spectral range. As used herein, the term "image" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, data recorded using an optical sensor, such as multiple electronic readings from a CMOS or CCD chip. An image may include raw image data or may be a preprocessed image. For example, preprocessing may include applying at least one filter and / or at least one background correction and / or at least one background subtraction to the raw image data.

[0046] For example, the image generation unit may include one or more of the following: at least one monochrome camera (e.g., including monochrome pixels), at least one color (e.g., RGB) camera (e.g., including color pixels), and at least one IR camera. The camera may be a CMOS camera. The camera may include at least one monochrome camera chip, such as a CMOS chip. The camera may include at least one color camera chip, such as an RGB CMOS chip. The camera may include at least one IR camera chip, such as an IR CMOS chip. For example, the camera may include monochrome (e.g., black and white) pixels and color pixels. Color pixels and monochrome pixels may be combined internally within the camera. The camera typically includes a one-dimensional or two-dimensional array of image sensors (e.g., pixels).

[0047] As described above, the image generation unit can be at least one camera. For example, the camera can be an internal camera and / or an external camera of the device. As described above, the internal camera and / or external camera of the device can be accessed via a hardware and / or software interface used as the image generation unit. In the case where the device is or includes a smartphone, the image generation unit can be the smartphone's front-facing camera (e.g., a selfie camera) and / or rear-facing camera.

[0048] The image generation unit may have a field of view between 10° × 10° and 75° × 75°, preferably 55° × 65°. The image generation unit may have a resolution of less than 2 MP, preferably between 0.3 MP and 1.5 MP.

[0049] The image generation unit may include additional components, such as one or more optical elements, like one or more lenses. As an example, the optical sensor may be a fixed-focus camera, where at least one lens is fixed relative to the camera's adjustment. Alternatively, the camera may include one or more variable lenses that can be adjusted automatically or manually. The camera may include at least one optical filter, such as at least one bandpass filter. The bandpass filter may be matched to the spectrum of the light emitter. However, other cameras are also feasible.

[0050] As used herein, the term "pattern image" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, an image generated by an image generation unit while a light pattern is applied (e.g., to an object and / or a user). A pattern image may include an image showing at least a portion of a user, particularly the user's face, when the user is illuminated by the light pattern, particularly over a corresponding region of interest included in the image. A pattern image can be generated by imaging and / or recording light reflected from an object and / or user illuminated by the light pattern. A pattern image showing a user may include at least a portion of the illuminated light pattern on at least a portion of the user. For example, projection by a projector and imaging using an image generation unit can be synchronized, for example, by using at least one control unit of the device.

[0051] As described above, a floodlight source can be configured to emit floodlight, and an image generation unit can be configured to generate at least one floodlight image when the floodlight source emits floodlight. As used herein, the term "floodlight image" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, an image generated by the image generation unit when the source emits infrared floodlight (e.g., on an object and / or a user). A floodlight image may include an image showing a user, particularly the user's face, when the user is illuminated with floodlight. A floodlight image can be generated by imaging and / or recording light reflected from the illuminated object and / or user. A floodlight image showing the user may include at least a portion of the floodlight on at least a portion of the user. For example, the illumination from the floodlight source and the imaging using the image generation unit can be synchronized, for example, by using at least one control unit of the device.

[0052] The image generation unit can be configured to image and / or record patterned images and floodlight images simultaneously or at different times. The image generation unit can also be configured to image and / or record patterned images and floodlight images at at least partially overlapping measurement areas or equivalents of these measurement areas.

[0053] The device includes at least one processor configured to extract liveness data from the pattern image and, based on the liveness data, allow the user to perform an authentication-required operation on the device. Specifically, the device can be configured to authenticate the user of the device to perform at least one authentication-required operation on the device. Authentication can be performed using the processor. The processor can be part of or be the at least one authentication unit configured to perform at least one authentication process for the user. The authentication unit can be configured to allow the user to perform an authentication-required operation on the device based on liveness data. Specifically, the authentication unit can be configured for a facial recognition authentication process operating on the floodlight image, the pattern image, and / or the extracted liveness data (particularly obtained from the pattern image).

[0054] As used herein, the term "processor" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, any logic circuit configured to perform basic operations of a computer or system, and / or generally to a device configured to perform computations or logical operations. In particular, a processor may be configured to process the basic instructions that drive a computer or system. As an example, a processor may include at least one arithmetic logic unit (ALU), at least one floating-point unit (FPU), such as a math coprocessor or a digital coprocessor, multiple registers, specifically configured to provide operands to the ALU and store the results of operations, and memories such as L1 and L2 cache memories. In particular, a processor may be a multi-core processor. Specifically, a processor may be or may include a central processing unit (CPU). Additionally or alternatively, a processor may be or may include a microprocessor; therefore, specifically, the elements of a processor may be contained within a single integrated circuit (IC) chip. Alternatively or additionally, the processor may be or may include one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs) and / or one or more tensor processing units (TPUs) and / or one or more chips, such as dedicated machine learning optimization chips. Specifically, the processor may be configured, for example, by software programming, to perform one or more operations. At least one or any component of the computer program configured to perform the authentication process may be executed by the processor. Alternatively or additionally, the authentication unit may be or may include a connectivity interface. The connectivity interface may be configured to transfer data from one device to a remote device; or vice versa. At least one or any component of the computer program configured to perform the authentication process may be executed by a remote device.

[0055] As used herein, the term "authentication unit" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, at least one unit configured to perform at least one authentication process for a user. An authentication unit may be or may include at least one processor. An authentication unit may be designed as software or an application.

[0056] For example, the authentication unit can perform at least one face detection using a floodlight image. Face detection can be performed locally on the device. However, face recognition (i.e., assigning identity to detected faces) can be performed remotely, for example, in the cloud, especially when identification rather than just verification is required. User templates can be stored at a remote device, such as in the cloud, and do not need to be stored locally. This can be advantageous from a storage and security perspective.

[0057] The authentication unit can be configured to identify a user based on a floodlight image. Therefore, in particular, the authentication unit can forward data to a remote device. Alternatively or additionally, the authentication unit can perform user identification based on the floodlight image, particularly by running an appropriate computer program with corresponding functionality. As used herein, the term "identification" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. The term can specifically refer to, but is not limited to, assigning identity to a detected face and / or at least one identity check and / or verification of the user's identity.

[0058] The authentication process may include multiple steps. For example, the authentication process may include performing at least one face detection. The face detection step may include analyzing a floodlight image. Additionally, for example, the authentication process may include identification. Identification may include assigning an identity to a detected face and / or at least one identity check and / or verification of the user's identity. Identification may include performing face verification on the imaged face to confirm whether it is the user's face. Identifying the user may include matching the floodlight image (e.g., showing the outline of parts of the user, particularly parts of the user's face) with a template. Identifying the user may include determining whether the imaged face is the user's face, and in particular determining whether the imaged face corresponds to at least one image of the user's face stored in at least one memory of a device, for example. If the floodlight image cannot match the image template, authentication may be unsuccessful.

[0059] The authentication process may include analyzing a flood image by, for example, one or more of the following: filtering; selecting at least one region of interest; forming a difference image between the flood image and at least one offset; inverting the flood image; background correction; decomposing into color channels; decomposing into hue, saturation, and luminance channels; frequency decomposition; singular value decomposition; applying a Canny edge detector; applying a Laplacian Gaussian filter; applying a difference Gaussian filter; applying the Sobel operator; applying the Laplacian operator; applying the Scharr operator; applying the Prewitt operator; applying the Roberts operator; applying the Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying the Radon transform; applying the Hough transform; applying a wavelet transform; thresholding; and creating a binary image. The region of interest may be manually determined by the user or may be automatically determined, for example, by identifying the user within the image. In particular, the analysis of the flood image may include using at least one image recognition technique, especially facial recognition technology. Image recognition technology includes at least one process of identifying a user in an image. Image recognition may include at least one technique using a combination of the following: color-based image recognition, for example using features such as template matching; segmentation and / or connected component (blob) analysis, for example using size or shape; machine learning and / or deep learning, for example using at least one convolutional neural network.

[0060] Analysis of the floodlight image may include determining multiple facial features. The analysis may include comparing the determined facial features with template features, specifically performing a matching process. Template features may be features extracted from at least one template. The template may be or may include at least one image generated during registration (e.g., when initializing the device). The template may be an image of an authorized user. Template features and / or facial features may include vectors. Feature matching may include determining the distance between vectors. User identification may include comparing the distance between vectors to at least one predefined limit, wherein if the distance is at least within tolerance ≤ the predefined limit, the user is successfully identified. Otherwise, the user is rejected and / or dismissed.

[0061] For example, image recognition may include a trained model using at least one model, particularly one including at least one face recognition model. Analysis of floodlight images can be performed using a face recognition system such as FaceNet, as described, for example, in Florian Schroff, Dmitry Kalenichenko, and James Philbin, “FaceNet: A Unified Embedding for face Recognition and Clustering,” arXiv:1503.03832. The trained model may include at least one convolutional neural network. For example, a convolutional neural network may be designed as described in the following literature: MD Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” CoRR, abs / 1311.2901, 2013; or C. Szegedy et al., “Going deeper with convolutions,” CoRR, abs / 1409.4842, 2014. For more details on convolutional neural networks for face recognition systems, please refer to: Florian Schroff, Dmitry Kalenichenko, and James Philbin, “FaceNet: A Unified Embedding for face Recognition and Clustering”, arXiv:1503.03832. Labeled image data from image databases can be used as training data.Specifically, labeled faces can be used from one or more of the following sources: GB Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” Technical Report 07-49, University of Massachusetts Amherst, October 2007; the YouTube® Faces database as described in L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrained videos with matched background similarity,” IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2011; or the Google® Facial Expression Comparison Dataset. The training of convolutional neural networks can be described as in "FaceNet: A Unified Embedding for face Recognition and Clustering" by Florian Schroff, Dmitry Kalenichenko, and James Philbin, arXiv:1503.03832.

[0062] As described above, the processor is configured to extract liveness data from the pattern image. To extract the liveness data, the authentication unit can forward the data to a remote device. Alternatively or additionally, the authentication unit can perform the extraction of liveness data based on the pattern image, particularly by running an appropriate computer program with corresponding functionality. In particular, by treating the liveness data as a parameter for verifying the authentication process, the authentication process can be robust to prevent spoofing using a recorded user image.

[0063] The authentication unit can be configured to outsource at least one step of the authentication process (such as user identification) and / or at least one step of the verification process (such as consideration of material data) to a remote device, specifically a server and / or a cloud server. This device and the remote device can be part of a computer network, particularly the Internet. Thus, the device can function as a field device used by the user to generate data required in the authentication process and / or its verification. The device can transmit the generated data and / or data associated with intermediate steps of the authentication process and / or its verification to the remote device. In this scenario, the authentication unit can be and / or may include a connection interface configured to transmit information to the remote device. Data generated by the remote device used in the authentication process and / or its verification can be further transmitted to the device. This data can be received by the connection interface included in the device. The connection interface can be specifically configured to transmit or exchange information. In particular, the connection interface can provide a data transmission connection. As an example, the connection interface can be or may include at least one port, including one or more of a network or Internet port, a USB port, and a disk drive.

[0064] It is important to emphasize that data from a device can be transferred to a specific remote device based on at least one circumstance (e.g., date, day, load of a particular remote device, etc.). A field device may not be able to select a specific remote device. Conversely, another device may choose which specific remote device the data can be transferred to. The authentication process and / or the generation of verification data may involve several different entities using the remote device. At least one entity may generate intermediate data and transfer that intermediate data to at least one other entity.

[0065] As used herein, the term "living data" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, data that allows for the distinction between living humans (particularly users) and inanimate objects (e.g., paper, 3D masks, etc.). Living data may include blood perfusion data and / or material data. Extracting living data may include extracting material data and / or extracting blood perfusion data. Living data may include information about the material on the user's surface to which the light spot is projected. Living data may include information about at least one vital sign.

[0066] Extracting liveness data (e.g., by using an authentication unit) may include extracting material data from a patterned image by beam profile analysis of the light spots. For information on beam profile analysis, see WO 2018 / 091649 A1, WO 2018 / 091638 A1, and WO 2018 / 091640 A1, the entire contents of which are incorporated herein by reference. Beam profile analysis can allow for reliable classification of a scene based on several light spots. Each light spot in a patterned image may include a beam profile. As used herein, the term "beam profile" generally refers to at least one intensity distribution of a light spot on an optical sensor as a function of pixels. Beam profiles may be selected from the group consisting of: trapezoidal beam profiles; triangular beam profiles; conical beam profiles; and linear combinations of Gaussian beam profiles.

[0067] Extracting material data from a pattern image may include generating material type and / or data derived from the material type. Preferably, the material data extraction may be based on the pattern image. Material data can be extracted by using at least one model. Extracting material data may include providing the pattern image to the model and / or receiving material data from the model. Providing the pattern image to the model may include, and subsequently may be, receiving the pattern image at the model's input layer or via a model loss function.

[0068] The model can be a data-driven model. A data-driven model can include convolutional neural networks and / or encoder-decoder structures, such as autoencoders. Other examples for generating representations can be FFT, wavelets, deep learning (such as CNNs), energy models, normalizing flows, GANs, visual transformers or transformers for natural language processing, autoregressive image modeling, normalizing flows, deep autoencoders, and deep energy-based models. Supervised or unsupervised schemes can be applied to generating representations and also to generating embeddings in ML languages, such as cosine or Euclidean metrics. The data-driven model can be parameterized based on a training dataset comprising at least one image and material data, preferably at least one pattern image and material data. In another embodiment, extracting material data can include providing a pattern image to the model and / or receiving material data from the model. In another embodiment, the data-driven model can be trained based on a training dataset comprising at least one image and material data. In another embodiment, the data-driven model can be parameterized based on a training dataset comprising at least one image and material data. The data-driven model can be parameterized based on the training dataset to receive images and provide material data based on the received images. A data-driven model can be trained on a training dataset to receive images and provide material data as output based on the received images. The training dataset may include at least one image and material data (preferably material data associated with at least one image).

[0069] An image can be or may include a representation of an image. A representation can be a low-dimensional representation of the image. A representation can include at least a portion of the data or information associated with the image. An image representation can include feature vectors. In embodiments, determining a representation, particularly a low-dimensional representation, can be based on principal component analysis (PCA) mapping or radial basis function (RBF) mapping. Determining a representation can also be referred to as generating a representation. Generating a representation based on a PCA mapping can include clustering based on features in the patterned image and / or a portion of the image. Alternatively or additionally, the generating representation can be based on a neural network architecture suitable for dimensionality reduction. A neural network architecture suitable for dimensionality reduction can include an encoder and / or a decoder. In an example, the neural network architecture can be an autoencoder. In an example, the neural network architecture can include a convolutional neural network (CNN). A CNN can include at least one convolutional layer and / or at least one pooling layer. A CNN can reduce the dimensionality of a portion of the image and / or the image by applying convolutions (e.g., based on convolutional layers) and / or by pooling. Applying convolutions can be suitable for selecting features related to material information of the patterned image.

[0070] The model can be adapted to determine the output based on the input. Specifically, the model can be adapted to determine material data based on an image as input. The model can be a deterministic model, a data-driven model, or a hybrid model. Preferably, the deterministic model reflects the physical phenomenon in a mathematical form, for example, including first-principles models. The deterministic model can include a set of equations describing the interaction between the material and patterned electromagnetic radiation, thereby producing measures of condition, vital signs, etc. The data-driven model can be a classification model. The hybrid model can be a classification model that includes at least one machine learning architecture and model parameters with deterministic or statistical adjustments. Statistical or deterministic adjustments can be introduced to improve the quality of the results because these adjustments provide a systematic relationship between empiricism and theory. In an embodiment, the data-driven model can be a classification model. The classification model can include at least one machine learning architecture and model parameters. For example, the machine learning architecture can be or can include one or more of the following: linear regression, logistic regression, random forest, piecewise linear, nonlinear classifier, support vector machine, Naive Bayes classification, nearest neighbor, neural network, convolutional neural network, generative adversarial network, support vector machine, or gradient boosting algorithm, etc. In the case of neural networks, the model can be a multi-scale neural network or a recurrent neural network (RNN), such as, but not limited to, a gated recurrent unit (GRU) recurrent neural network or a long short-term memory (LSTM) recurrent neural network. The data-driven model can be parameterized based on a training dataset. The data-driven model can be trained based on the training dataset. Training the model can include parameterizing the model. The term "training" can also refer to learning. Specifically, the term can refer to, but is not limited to, the process of constructing a classification model, and particularly the process of determining and / or updating the parameters of a classification model. Updating the parameters of a classification model can also be referred to as retraining. Training as discussed herein can include retraining. In embodiments, the training dataset can include at least one image and material information.

[0071] Extracting material data from an image using a data-driven model can include feeding the image to the data-driven model. Alternatively or additionally, extracting material data from an image using a data-driven model can include generating an embedding associated with the image based on the data-driven model. The embedding can refer to a low-dimensional representation associated with the image, such as a feature vector. The feature vector can be adapted to suppress the background while preserving the material signature indicating the material data. In this context, the background can refer to information independent of the material signature and / or the material data. Further, the background can refer to information related to biometric features, such as facial features. Based on the embedding associated with the image, the material data can be determined using the data-driven model. Alternatively or additionally, extracting material data from an image by feeding the image to the data-driven model can include transforming the image into material data, particularly material feature vectors indicating the material data. Therefore, the material data can further include material feature vectors and / or the material feature vectors can be used to determine the material data.

[0072] The authentication process can be verified based on extracted material data. In an embodiment, verification based on the extracted material data may include determining whether the extracted material data corresponds to expected material data. Determining whether the extracted material data matches the expected material data may be termed verification. Allowing or denying a user and / or object from performing at least one operation requiring authentication on the device based on material data may include verifying the authentication or authentication process. Verification may be based on material data and / or images. Determining whether the extracted material data corresponds to expected material data may include determining the similarity between the extracted material data and the expected material data. Determining the similarity between the extracted material data and the expected material data may include comparing the extracted material data with the expected material data. The expected material data may refer to predetermined material data. In an example, the expected material data may be skin. It may be determined whether the material data corresponds to the expected material data. In an example, the material data may be a non-skin material or silicon. Determining whether the material data corresponds to the expected material data may include comparing the material data with the expected material data. The comparison of the material data with the expected material data may result in allowing and / or denying a user and / or object from performing at least one operation requiring authentication. In the example, skin as the desired material data can be compared with non-skin materials or silicon as material data, and the result can be negative because silicon or non-skin materials may differ from skin.

[0073] The certification process or its verification may include generating at least one feature vector from material data and matching the material feature vector with an associated material reference template vector.

[0074] In addition to using material data or as an alternative, extracting live data can include extracting blood perfusion data. For example, the beam projected by a projector can be coherently patterned infrared illumination. Extracting blood perfusion data can include determining the speckle contrast of the patterned image and determining a blood perfusion metric based on the determined speckle contrast. The speckle contrast can be a measure of the average contrast of the intensity distribution within a region of the speckle pattern. Specifically, the speckle contrast K over the speckle pattern region can be expressed as the standard deviation σ versus the average speckle intensity. The ratio, that is,

[0075]

[0076] Speckle contrast can include speckle contrast values. Speckle contrast values ​​can range from 0 to 1. Blood perfusion can be determined based on speckle contrast.

[0077] Blood perfusion measurements can depend on the determined speckle contrast. If the speckle contrast changes, the blood perfusion measurements obtained from the speckle contrast will also change accordingly. Blood perfusion measurements can be a single number or value that represents the likelihood that the object is a living organism.

[0078] For example, to determine speckle contrast, a complete pattern image can be used. Alternatively, a portion of the pattern image can be used to determine speckle contrast. Preferably, a portion of the pattern image represents a smaller area of ​​the pattern image than the area of ​​the complete pattern image. A portion of the pattern image can be obtained by cropping the pattern image.

[0079] In this embodiment, a data-driven model can be used to determine blood perfusion measurements. The data-driven model is parameterized and / or trained based on a training dataset. The training dataset may include pattern images and blood perfusion measurements. The data-driven model can be parameterized and / or trained based on the training dataset to output blood perfusion measurements based on received pattern images.

[0080] The authentication process can be verified based on blood perfusion measurements. In an embodiment, verification based on blood perfusion measurements may include determining whether the blood perfusion measurement corresponds to a human blood perfusion measurement. Determining whether a blood perfusion measurement corresponds to a human can be termed verification. Allowing or denying a user and / or object to perform at least one operation requiring authentication on the device based on blood perfusion measurements may include verifying the authentication or authentication process. Verification may be based on blood perfusion measurements. Determining whether a blood perfusion measurement corresponds to a human may include comparing the blood perfusion measurement to at least one predefined or predetermined range of blood perfusion measurement values, for example, stored in at least one database. If the extracted blood perfusion measurement is at least within the tolerance range of the redefined or predetermined range of blood perfusion measurement values, then the authentication passes; otherwise, the authentication fails. If the authentication passes, the method may include allowing the user to perform at least one operation requiring authentication. Otherwise, if the authentication fails, the authentication unit may deny the user from performing at least one operation requiring authentication.

[0081] At least one authorization unit can be used to allow and / or deny a user from performing authentication-required operations on the device based on liveness data. Specifically, the authorization unit can be configured to perform at least one authorization step, for example, by using at least one authorization unit. As used herein, the term "authorization step" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or customized meaning. The term can specifically refer to, but is not limited to, the step of assigning access rights to a user (particularly selective permission or selective restriction of access to the device and / or at least one resource of the device). The authorization unit can be configured for access control. As used herein, the term "authorization unit" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or customized meaning. The term can specifically refer to, but is not limited to, a unit configured to authorize a user, such as a processor. The authorization unit may include at least one processor or may be designed as software or an application. The authorization unit and the authentication unit may be integrated, for example, by using the same processor. The authorization unit can be configured to allow the user to perform at least one operation on the device, such as unlocking the device, if the user is successfully authenticated, or to deny the user from performing at least one operation on the device if authentication is unsuccessful. Thus, the user can be aware of the authentication result. This method may include displaying the authentication result on a monitor.

[0082] At least one operation requiring authentication on the device may be accessing the device (e.g., unlocking the device) and / or accessing an application preferably associated with the device and / or accessing a portion of an application preferably associated with the device. In embodiments, allowing a user to access a resource may include allowing the user to perform at least one operation with the device and / or system. The resource may be a device, system, a function of the device, a function of the system, and / or an entity. Additionally and / or alternatively, allowing a user to access a resource may include allowing the user to access an entity. This entity may be a physical entity and / or a virtual entity. A virtual entity may be, for example, a database. A physical entity may be an access-restricted area. An access-restricted area may be one of the following: a secure area, a room, an apartment, a vehicle, a portion of the examples mentioned above, etc. The device and / or system may be locked. The device and / or system may be unlockable only by an authorized user.

[0083] The device includes at least one control unit and at least one status query device configured to retrieve at least one piece of status information regarding the current environmental state of the device. The control unit is configured to automatically trigger at least one calibration depending on the satisfaction of at least one predetermined environmental state condition.

[0084] As used herein, the term "control unit" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, a device or combination of devices capable of and / or configured to perform at least one computational operation and / or to control at least one other device (such as at least one other component of the device for authentication). Specifically, the at least one control unit may be embodied in at least one processor and / or may include at least one processor, wherein the processor may be specifically configured, through software programming, to perform one or more operations.

[0085] As used herein, the term "status query device" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, a device or combination of devices capable of or configured to retrieve at least one type of status information as described above. Specifically, a status query device may include at least one interface (e.g., a wireless or wired interface) for retrieving status information in an electronic format (e.g., a data format), and / or at least one device (e.g., at least one sensor device) configured to generate status information, as will be further detailed below. A status query device may also be wholly or partially integrated into a control unit. Additionally or alternatively, a status query device may include one or more devices integrated into the authentication device, such as one or more integrated sensors.

[0086] As used herein, the term "retrieval" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, the process of providing at least one object or item to at least one entity. This process may include generating at least one item or object, or obtaining at least one item or object from another source, and providing at least one item or object to at least one entity. Thus, reference to retrieving at least one piece of status information about the current environmental state of the device used for authentication may, as an example, include obtaining status information from at least one source (e.g., from at least one external source) via at least one interface (e.g., a wireless and / or wired interface), such as the Internet. Additionally or alternatively, in order to retrieve at least one piece of status information, the device may also include one or more sensors or sensing devices for measuring at least one measurable value from which at least one piece of status information about the current environmental state of the device can be obtained directly or indirectly.

[0087] As used herein, the term "current environmental state of the device" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, the situation and / or conditions in which the device used for authentication is located at any given moment, i.e., at the time of the query and / or within a predetermined time span before and after the time of the query. The term "environmental state" may relate to at least one condition of the environment (specifically, the surrounding environment) in which the device used for authentication is located and / or at least one relationship between the device and its surrounding environment, such as location and / or orientation.

[0088] As an example, state information regarding the current environmental state refers to one or more of the following: ambient lighting conditions, such as ambient light level and / or spatial location and / or orientation relative to external light sources; weather conditions, temperature information indicating the current temperature; location information of the device; orientation information of the device; relative orientation and / or relative position of the device to at least one object in the environment; operational information indicating the user's current operating mode of the device; at least one piece of information available via at least one network; and the current environmental state approximated based on analysis of previously recorded environmental state information. For example, the current environmental state can be approximated based on analysis of previously recorded environmental state information. To determine the current environmental state, previously obtained measurements can be used and / or considered. Actual measurements and previously obtained measurements can be combined.

[0089] As described above, the control unit is configured to automatically trigger at least one calibration depending on the satisfaction of at least one predetermined environmental condition. As used herein, the term "automatically triggered" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, a process that initiates action without human interaction (specifically, without any human interaction or only with optional human interaction).

[0090] After retrieving status information about the current environmental state of the device, the control unit automatically triggers calibration. Calibration of the device may include calibrating at least one hardware component that affects operational performance, particularly certification. Calibration may include calibrating one or more of the following: projector position, image generation unit, calibration reference pattern, and other hardware used for certification (particularly those that have or are suspected of having temperature dependence).

[0091] Calibration can be automatically triggered by the control unit. Calibration can be automatically triggered by the control unit if the retrieved status information regarding the current environmental state meets predetermined environmental state conditions.

[0092] As used herein, the term "predetermined environmental state condition" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, the conditions of the environment in which the device for certification is located and / or the conditions of the relationship between the device for certification and the environment, which are predetermined to be sufficient to perform calibration. As an example, the at least one environmental state condition may be at least one condition that must be met for certification to be performed based on at least one piece of status information regarding the current environmental state of the device. As an example, the status information may be compared to at least one maximum or minimum threshold, and the condition may be satisfied when the status information is above the minimum threshold or below the maximum threshold, respectively, and as an example, calibration may be automatically triggered. Additionally or alternatively, the environmental state condition may be satisfied when at least one piece of status information regarding the current environmental state of the device for certification and / or at least one minor value obtained from the at least one piece of status information using a predetermined relation or function is within at least one predetermined range. Therefore, as an example, the control unit can obtain at least one piece of status information about the current environmental state from a status query device, optionally transform the at least one piece of status information into at least one secondary value, and then check whether the at least one piece of status information and / or the at least one secondary value meets environmental state conditions, such as having a predetermined target value or being within a predetermined range. If this environmental state condition is met, the control unit can automatically trigger at least one calibration with or without delay.

[0093] The predetermined environmental conditions may include at least one condition selected from the group consisting of: ambient light level within a predetermined suitable range for performing calibration measurements; the device being in a suitable location for performing calibration measurements; the device being in a suitable orientation for performing calibration measurements; the device not facing an external light source; the device not being near or pointing at an object; the device not being in a pocket; the device being at a temperature within a predetermined temperature range suitable for calibration measurements; time-temperature variation within a predetermined range suitable for calibration measurements; weather conditions within a predetermined range suitable for calibration measurements; the device not currently being used for another function; and the device being available via at least one network.

[0094] Therefore, when predetermined environmental conditions indicate that the environmental conditions are suitable for at least one calibration, using at least one piece of status information regarding the current environmental state of the device for certification allows the device for certification to repeatedly perform calibration. Specifically, the at least one piece of status information can be retrieved using the integrated device for certification.

[0095] The device can be configured to use at least one integrated sensor device as at least part of the at least one state query device. Therefore, one or more integrated sensor devices that are present in the device in any other way can be used, for example, for one or more other purposes.

[0096] Typically, and specifically, in the case of using a mobile communication device, the status query device may also include at least one device selected from the group consisting of: a front-facing camera positioned on the same side as the display; a rear-facing camera positioned on the opposite side of the display; a position sensor; an illumination sensor configured to determine at least one illumination state in the environment of the device; a temperature sensor; a motion sensor; a gyroscope sensor; a magnetic sensor; a material sensor configured to determine at least one material property of at least one object near the device; a spectrometer device configured to acquire at least one spectral information; and at least one software sensor configured to generate information about the status of the device by processing inputs from multiple physical sensors. The software sensor may be configured to determine the status of the device indirectly or in a learning mode. This may also include user behavior, time of day, or other relevant information fragments. Many, or even all, of these devices, specifically sensor devices, are typically integrated into mobile communication devices such as smartphones. Each of these devices is typically configured to provide at least one status information about the current environmental state of the device. Specifically, the device, or one or more of the devices, or even all of the devices, can provide information that can be used to determine whether environmental conditions allow calibration to be triggered. Signals from one or more devices can be used individually or in combination to determine whether conditions for calibration are provided.

[0097] The status query device can be configured to determine whether the device is placed on a table in a room by using at least one gyroscope sensor, and / or wherein the status query device is configured to determine the ambient light level by determining at least one dark image without having a projector projecting multiple beams, and / or wherein the status query device is configured to determine whether the ceiling is visible by capturing at least one image with a projector projecting multiple beams, and / or wherein the status query device is configured to determine whether there is an obstacle near the device by using at least one proximity sensor.

[0098] For example, a device can attempt to detect whether it is on a table in a room. A gyroscope sensor can support detecting whether a device (e.g., a smartphone) is lying flat on a table. For example, a device can capture image frames without any floodlight or light from a projector to determine if the room is dark, i.e., with little sunlight. For example, a proximity sensor can check for obstacles on a device (e.g., a smartphone). For example, a device can capture image frames with a projector on. If there is a ceiling, this will be visible on the laser frame. The device can detect the light pattern on that frame for calibration.

[0099] As used herein, the term "calibration" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, the process of determining the deviation between the actual performance of at least one hardware component (e.g., an image generation unit) of the device that affects certification and the target performance of said hardware component. Calibration may further include determining a correction for the determined deviation. For example, calibration includes one or more of calibrating a reference pattern, calibrating the position of the projector, at least one temperature calibration, or at least one diffraction calibration.

[0100] This calibration includes at least one user-guided calibration and / or automatic calibration. As used herein, the term "user-guided calibration" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, calibration that includes at least one user action and / or interaction in at least one step. As used herein, the term "automatic calibration" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to a specific or custom meaning. Specifically, the term may refer to, but is not limited to, calibration performed entirely automatically, such as calibration without any user action or user interaction.

[0101] Calibration may include procedures as described in WO 2022 / 253777, the contents of which are included herein by reference.

[0102] Calibration may include calibrating a reference pattern, and / or particularly calibrating the position of the projector.

[0103] Calibration may include calibrating a reference pattern by performing the following steps:

[0104] i) While the projector projects the multiple beams, at least one calibration image is captured at the relative distance between the device and at least one reference object using an image generation unit;

[0105] ii) Determine the reflection pattern by identifying the light spots generated on the captured calibration image by the reference object in response to the illumination of the plurality of beams;

[0106] iii) Considering the different relative distances between the device and the reference object when capturing the calibration image, match the light spot of the reflection pattern with the features of the reference pattern to determine the matching reflection feature pairs;

[0107] iv) Recalculate the reference pattern based on the image coordinates of these corresponding matching spots.

[0108] In step i), multiple calibration images may be captured at different relative distances, for example, at least one calibration image may be captured at each relative distance. Determining the calibration images may include imaging at least one two-dimensional image of a white surface at different relative distances using an image generation unit with multiple beams projected by a projector. For example, calibration may include identifying at least one flat white surface (e.g., a wall), for example, by using a selfie camera while carrying a smartphone. For example, the user may be asked to capture at least two calibration images of a wall at a coarse distance.

[0109] At least one evaluation device (such as a processor) can be configured to perform image analysis and identify light spots in the calibrated image.

[0110] As used herein, the term "calibration image" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to any particular or custom meaning. The term may specifically refer to, but is not limited to, an image used for calibration. The reference object can be any object, such as a wall. The calibration image can be a far-field image.

[0111] Triangulation can be used to perform calibration. Calibration can utilize information about the relative positions of a reference pattern and the projector. The reference pattern can be the projector's pattern observed from the image generation unit at an assumed infinity distance. The reflection pattern can be captured by the image generation unit at a known distance z0. If reflection patterns at different distances (e.g., 20 cm, 1 m, 2 m, or infinity) are known, the reference pattern can be recalculated.

[0112] Many applications involve the near field (e.g., 0.15 m to 0.6 m). However, a rough estimate of distances outside the near field (> 1.5 m) can provide a good approximation of the near field. This is because triangulation error has a quadratic relationship with distance. For example, a target at a distance of 2 m is estimated to be at a rough distance of 1.5 m. The distance error can be 0.5 m. A triangulation error of 0.2 m can be simply given by quadratic error propagation, i.e., 0.5 m / (2 m). 2 m) (0.2 m 0.2 m) = 5 mm.

[0113] The evaluation device can be configured to select spots and match them with corresponding features of a reference pattern. As used herein, the term "match" refers to determining and / or evaluating corresponding features of the reference pattern and spots of the calibration image. Matching can be performed as follows. The evaluation device can be configured to identify at least one feature in the reference pattern that has substantially the same ordinate as the selected spots. The term "substantially the same" means that the similarity is within 10%, preferably within 5%, and most preferably within 1%. Epipolar geometry can be used to determine the features of the reference pattern corresponding to the spots. For a description of epipolar geometry, please refer, for example, Chapter 2 of "DreidimensionalesComputersehen" edited by X. Jiang and H. Bunke, Springer, Heidelberg, Berlin, 1997. Epipolar geometry can assume that the reference pattern and the calibration image can be images determined at different spatial locations and / or spatial orientations with fixed distances. The evaluation device can be configured to match corresponding spots in the spots of the calibration image with corresponding reference features in the reference features within the displacement region by using at least one linear scaling algorithm, taking distance information into account.

[0114] Specifically, step iv) may include recalculating the reference pattern by taking into account the estimated relative distance to the reference object. The distance estimate may be assumed, or a distance sensor may be used. The distance information may be an estimate of the different relative distances between the device and the reference object when capturing the calibration image.

[0115] The evaluation device can be configured to determine the epipolar line in a reference pattern. The evaluation device can be configured to determine a straight line extending from a feature of the reference pattern. This straight line may include possible features corresponding to the selected spot. This straight line and the baseline span the epipolar plane. Since the reference pattern is determined at a different relative position than the calibration image, the corresponding possible features may be imaged on a straight line in the reference pattern, called the epipolar line. Therefore, it is assumed that the feature of the reference pattern corresponding to the selected spot lies on the epipolar line. As described above, the evaluation device can be configured to pre-classify the selected spot using information about distance, particularly estimates of the different relative distances between the device and the reference object. This allows for explicit assignment to a feature of the reference pattern. Furthermore, in particular, the features of the projected pattern can be arranged such that corresponding features of the reference pattern have the largest possible relative distance to each other on the epipolar line. The features of the projected pattern can be arranged such that only a few features of the reference pattern are located on the epipolar line.

[0116] After matching the features of the spot and the reference pattern, the reference pattern is recalculated. This recalculation can be performed using the image coordinates of the corresponding matched spot. The image coordinates of the spot can be mapped to projector coordinates. This will produce a calibrated reference pattern.

[0117] For example, the calibration of the reference pattern can be performed as follows. In step i), the reflected pattern can be captured at a distance (e.g., 1.5 m to 3 m), and the spot can be identified. Calibration may include checking whether the pattern is complete. Next, in step iii), the reflected pattern can be matched with the reference pattern. As described above, step iv) may include recalculating the reference pattern by taking into account the estimated relative distance to the reference object. For example, the projection target may be assumed to be at a nominal distance, or an estimate of the distance may be used (e.g., by illumination from the spot). In step iv), the image coordinates of the spot are mapped to projector coordinates. This produces the reference pattern.

[0118] Calibration may include calibrating the projector's position. Projector position calibration can be performed after calibrating the reference pattern.

[0119] The calibration of the projector's position may include the following steps:

[0120] I) While the projector projects the multiple beams, at least one calibrated facial image is captured at the relative distance between the device and the user's face using the image generation unit;

[0121] II) Estimate the relative distance by analyzing the calibrated facial image, wherein the analysis includes extracting the key points (landmarks) of the face by using two-dimensional face detection.

[0122] III) Identify light spots on the user's face in the calibrated facial image and match these identified light spots with features of a reference pattern, taking into account the estimated relative distance, to determine matching pairs of reflection features;

[0123] IV) The translation vector describing the position of the projector is determined by determining the epipolar distance of each of these matched pairs of reflective features.

[0124] As used herein, the term "calibrated facial image" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, an image of at least a portion of a user's face used for calibration. Determining the calibrated facial image may include imaging at least one two-dimensional image of the user's face at a relative distance between the projector and the user's face, using an image generation unit with multiple beams projected by the projector. The calibrated facial image may be captured, for example, during registration or unlocking. For example, a 2D image generated using a selfie camera may be used.

[0125] Facial images can be calibrated and analyzed by extracting facial keypoints through 2D face detection. Keypoint extraction can be performed, for example, as described in en.wikipedia.org / wiki / Landmark_detection or as described in OpenCV: Facelandmark detection in an image. This allows for the estimation of facial distances. Alternatively, a distance sensor can be used.

[0126] The light pattern of the calibrated facial image can be analyzed. For example, in step III), only the light spot on the face can be used. The light spot can be matched with a reference pattern, and the matching can be performed as described above. This is possible because the facial distance is known by using key points.

[0127] After matching the light spot with a reference pattern, the epipolar distance d can be determined. Calibration may include determining the epipolar line of the matching feature of the reference pattern for each pair of matching light spots and feature pairs for the reference pattern. In particular, the epipolar line used for matching can be used as the epipolar line of the pair. As used herein, the term "epidial distance" can refer to the distance between a feature of the reference pattern and the epipolar line used for matching (denoted as the corresponding epipolar line). This distance can be determined by determining the image coordinates of the calibrated face image and the image coordinates of the corresponding epipolar line and comparing these image coordinates. The minimum distance to the corresponding epipolar line can be used as the epipolar distance. Under good external calibration, the epipolar distance is close to zero.

[0128] The epipolar distance can be defined as a function d(x, y) of the position (x, y) of a reference pattern. The epipolar distance function d can be analyzed to calculate corrections for rotation and / or translation. In the case of a miscalibrated system, function d can generate a geometric pattern. The shape of the geometric pattern uniquely indicates the degree of miscalibration. The geometric pattern can be, or can include, one or more of the repetition, steepness, discontinuity, and curvature in the function d(x, y), which can be used for calibration. If the rotation and / or translation of the projector and / or image generation unit changes, this result can be observed as a geometric pattern in function d. The evaluation device can be configured to execute algorithms designed to analyze d(x, y) and to calculate corrections for rotation and / or translation. The evaluation device can be configured to determine corrections to the reflected image by evaluating one or more of the shape, repetition, steepness, discontinuity, and curvature of the geometric pattern.

[0129] This generates a translation vector to position the projector. The length of the translation vector can be the baseline length. This value may already be known in the hardware design. With this step, calibration is complete. The reference pattern and projector position (rotation and / or translation) are calibrated.

[0130] Alternatively or additionally, calibration may include obtaining reference data for the hardware used for certification (such as a projector and / or image generation unit) at different temperatures. This allows for the correction of temperature-related drift. Therefore, calibration may include at least one temperature calibration. For example, temperature calibration includes the following steps:

[0131] a) While the projector projects the plurality of beams, it captures at least one temperature-calibrated image of at least one reference object at at least a plurality of temperatures;

[0132] b) Identify the light spots on these temperature calibration images and, taking into account the estimated relative distance between the reference object and the projector, match the identified light spots with features of the reference pattern to determine a matching pair of reflection features for each temperature;

[0133] c) Determine the temperature-dependent correction of the reference pattern by determining the deviation between the positions of these matched reflection features for each temperature, and store the temperature-dependent correction of the reference pattern in at least one database.

[0134] Alternatively or concurrently, calibration may include at least one diffraction calibration. For example, a selfie camera (potentially located behind a perforation, meaning it is unaffected by diffraction effects in the receiving path) can be used to obtain diffraction pattern correction data. Projection and imaging through a display can introduce diffraction effects on both TX and RX. By providing a selfie camera image of the projected pattern on a surface and comparing the RX image with the selfie camera image, the effects of RX and TX diffraction can be categorized. This can support simplification and / or greater robustness of the underlying algorithms and provides the potential to correct for variations over the product's lifespan.

[0135] For example, diffraction calibration includes the following steps:

[0136] A) While the projector projects the multiple beams, at least one diffraction calibration image of at least one reference object is captured using a camera behind the perforation;

[0137] B) Identify the light spots on the diffraction calibration image and, taking into account the estimated relative distance between the reference object and the projector, match the identified light spots with features of the reference pattern to determine the matching reflection feature pairs;

[0138] C) The diffraction correction of the reference pattern is determined by determining the deviation between the positions of these matched spots and features, and the diffraction correction of the reference pattern is stored in at least one database.

[0139] The control unit can be configured to repeatedly and automatically trigger calibration, particularly calibration measurements. The control unit can be configured to perform status checks at predetermined time points. For example, these status checks each include receiving at least one status information about the current environmental state and checking whether at least one predetermined environmental state condition is met. The control unit can further be configured to trigger calibration measurements based on whether at least one environmental state condition is met.

[0140] As used herein, the term "satisfy" is a broad term and will be given its common and conventional meaning to those skilled in the art, and is not limited to a specific or custom meaning. The term may specifically refer to, but is not limited to, at least one condition that must be satisfied. For example, at least one environmental state condition may be at least one condition that at least one piece of state information regarding the current environmental state must satisfy. For example, state information may be compared to at least one maximum or minimum threshold, and the condition may be satisfied when the state information is above the minimum threshold or below the maximum threshold, respectively, and, for example, calibration is automatically triggered. Additionally or alternatively, an environmental state condition may be satisfied when at least one piece of state information regarding the current environmental state and / or at least one minor value obtained from the at least one piece of state information using a predetermined relation or function is within at least one predetermined range (e.g., above the minimum threshold or below the maximum threshold, respectively). As an example, the control unit can obtain at least one piece of status information about the current environmental state from a status query device. It can optionally transform this at least one piece of status information into at least one secondary value, and then check whether the at least one piece of status information and / or the at least one secondary value meets environmental state conditions, such as having a predetermined target value or being within a predetermined range. If the environmental state conditions are met, the control unit can automatically trigger at least one reference measurement, with or without delay, such as by triggering a device to capture an image and / or issuing an instruction to the user to capture an image.

[0141] In summary, it was unexpectedly discovered that automatically determining the correct timing for performing calibration can be advantageously used to calibrate hardware built into smartphones for facial authentication. This could allow for reduced costs associated with factory calibration. Furthermore, calibration of the 3D imager behind the OLED screen can be performed within a fully assembled smartphone.

[0142] In another aspect of the invention, a method for performing at least one reference measurement is disclosed using an apparatus according to the invention (such as any of the embodiments described above and / or any of the embodiments described in further detail below).

[0143] The method includes:

[0144] i. Retrieve at least one piece of status information regarding the current environmental status of the device;

[0145] ii. Check that at least one environmental condition is met for performing the calibration, particularly at least one calibration measurement; and

[0146] iii. If the environmental conditions in step ii. are met, calibration, in particular at least one calibration measurement, is automatically triggered.

[0147] These method steps can be performed in a given order or in a different order. Furthermore, there may be one or more additional method steps not listed. Furthermore, one, more than one, or even all of the method steps may be performed repeatedly. Therefore, specifically, method steps i through iii may be performed repeatedly (e.g., in a given order). Moreover, as an example and as described above, repetition may occur at regular time intervals and / or at determinable or predetermined time points. As an example, a predetermined time point may include the time when the device is started, or it may include the time when the app on the device (e.g., a mobile communication device, such as a smartphone) is started.

[0148] The method may include

[0149] - Project multiple beams of light onto the user through at least one display using at least one projector;

[0150] - By using at least one image generation unit, a pattern image showing the projection of the plurality of light beams onto the user is generated;

[0151] - By using at least one processor to extract liveness data from the pattern image, and based on the liveness data, allow the user to perform authentication-required operations on the device;

[0152] - By using at least one control unit and at least one status query device to retrieve at least one status information about the current environmental status of the device, and by automatically triggering at least one calibration depending on the satisfaction of at least one predetermined environmental status condition.

[0153] For definitions and embodiments, refer to the definitions and embodiments described above for the device or in more detail below.

[0154] This method can be computer-implemented. As used herein, the term "computer-implemented" is a broad term and will be given its common and conventional meaning to those skilled in the art and is not limited to any particular or custom meaning. Specifically, the term may refer to, but is not limited to, methods involving at least one computer and / or at least one computer network. The computer and / or computer network may include at least one processor configured to perform at least one method step of the method according to the invention. Specifically, each of these method steps is performed via a computer and / or computer network. The method can be performed entirely automatically, specifically without user interaction.

[0155] All described method steps can be performed using the device. Therefore, a single processing device can be configured to exclusively execute at least one computer program, particularly at least one line of computer program code configured to execute at least one algorithm, as used in at least one embodiment of the method according to the invention. Hereinafter, the computer program executed on the single processing device may include all instructions that cause the computer to perform the described method. Alternatively or additionally, at least one method step can be performed using at least one remote device, particularly selected from at least one of a server or a cloud server, especially when the device and the remote device are part of a computer network. In this case, the computer program may include at least one remote component to be executed by at least one remote processing device to perform at least one method step. The remote component may have the capability to perform user identification and / or material data extraction. Further, the computer program may include at least one interface configured to forward data to and / or receive data from at least one remote component of the computer program.

[0156] This document further discloses and proposes a computer program comprising computer-executable instructions for performing the methods according to the invention in one or more embodiments included herein, when the program is executed on a computer or computer network. Specifically, the computer program may be stored on a computer-readable data carrier and / or a computer-readable storage medium.

[0157] As used herein, the terms "computer-readable data carrier" and "computer-readable storage medium" specifically refer to non-transitory data storage devices, such as hardware storage media on which computer-executable instructions are stored. Computer-readable data carriers or storage media can specifically be or may include storage media such as random access memory (RAM) and / or read-only memory (ROM).

[0158] Therefore, specifically, one, more, or even all of the method steps i to iii as indicated above can be performed by using a computer or computer network, preferably by using a computer program.

[0159] This document further discloses and proposes a computer program product having program code means for performing the method according to the invention in one or more embodiments included herein when the program is executed on a computer or computer network. Specifically, the program code means may be stored on a computer-readable data carrier and / or a computer-readable storage medium.

[0160] This document further discloses and proposes a data carrier having a data structure stored thereon, which, after being loaded into a computer or computer network (e.g., into the working memory or main memory of the computer or computer network), can perform methods according to one or more embodiments disclosed herein.

[0161] This document further discloses and proposes a computer program product having program code means stored on a machine-readable medium to perform methods according to one or more embodiments disclosed herein when the program is executed on a computer or computer network. As used herein, a computer program product refers to a program that is a tradable product. The product can generally exist in any format, such as in paper format, or on a computer-readable data carrier and / or computer-readable storage medium. Specifically, the computer program product can be distributed via a data network.

[0162] Finally, this document discloses and proposes a modulated data signal containing computer system or computer network readable instructions for performing methods according to one or more embodiments disclosed herein.

[0163] Referring to the computer implementation aspects of the present invention, one or more, or even all, of the method steps in one or more of the methods disclosed in the embodiments herein can be performed using a computer or computer network. Therefore, typically, any of the method steps involving the provision and / or manipulation of data can be performed using a computer or computer network. Generally, these method steps can include any method steps, except for those that typically require manual work, such as providing samples and / or performing certain aspects of actual measurements.

[0164] Specifically, this article further discloses:

[0165] - A computer or computer network including at least one processor, wherein the processor is adapted to perform a method according to one of the embodiments described in this specification.

[0166] - A computer-loadable data structure adapted to perform a method according to one of the embodiments described in this specification when the data structure is executed on a computer.

[0167] - A computer program, wherein the computer program is adapted, when executed on a computer, to perform a method according to one of the embodiments described in this specification.

[0168] A computer program comprising program means for performing a method according to one of the embodiments described herein when the computer program is executed on a computer or a computer network.

[0169] - A computer program comprising program means according to a previous embodiment, wherein the program means is stored on a computer-readable storage medium.

[0170] - A storage medium wherein a data structure is stored on the storage medium, and wherein the data structure is adapted to perform a method according to one of the embodiments described herein after being loaded into the main storage device and / or working storage device of a computer or computer network.

[0171] - A computer program product having program code means, wherein the program code means may be stored or stored on a storage medium for performing a method according to one of the embodiments described herein when the program code means is executed on a computer or a computer network.

[0172] As used herein, the terms “have,” “include,” or “contain,” or any of their grammatical variations, are used in a non-exclusive manner. Thus, these terms can refer either to a situation where no other features exist in the entity described in the context besides those introduced by these terms, or to a situation where one or more other features exist. For example, the statements “A has B,” “A includes B,” and “A contains B” can refer either to a situation where no other elements exist in A besides B (i.e., A consists solely of B), or to a situation where entity A contains one or more other elements besides B (such as element C, elements C and D, or even other elements).

[0173] Furthermore, it should be noted that the terms "at least one," "one or more," or similar expressions indicating a feature or element may appear once or more, but are typically used only once when describing the corresponding feature or element. In most cases, the expressions "at least one" or "one or more" are not repeated when referring to the corresponding feature or element, but in fact, the corresponding feature or element may appear once or more.

[0174] Furthermore, as used herein, the terms “preferredly,” “more preferably,” “particularly,” “more particularly,” “specifically,” “more specifically,” or similar terms are used in combination with optional features without limiting the possibility of alternatives. Therefore, the features introduced by these terms are optional features and are not intended to limit the scope of the claims in any way. As those skilled in the art will recognize, the invention can be practiced by using alternative features. Similarly, features introduced by phrases such as “in embodiments of the invention” are intended to be optional features and do not limit any alternative embodiments of the invention, the scope of the invention, or the possibility of combining features introduced in this way with other optional or non-optional features of the invention.

[0175] In summary, and without excluding other possible embodiments, the following embodiments are conceivable:

[0176] Example 1. A device for authenticating users, the device comprising:

[0177] - At least one projector configured to project multiple beams of light onto the user through at least one display.

[0178] - At least one image generation unit configured to generate a patterned image showing the projection of the plurality of light beams onto the user;

[0179] - At least one processor, configured to extract liveness data from the pattern image and, based on the liveness data, allow the user to perform an operation requiring authentication on the device;

[0180] - At least one control unit and at least one status query device, the at least one status query device being configured to retrieve at least one status information about the current environmental state of the device, wherein the control unit is configured to automatically trigger at least one calibration depending on the satisfaction of at least one predetermined environmental state condition.

[0181] Example 2. According to the device described in the previous embodiment, the state information regarding the current environmental state refers to one or more of the following: ambient lighting conditions, such as ambient light level and / or spatial position and / or orientation relative to an external light source; weather conditions, temperature information indicating the current temperature; the device's location information; the device's orientation information; the relative orientation and / or relative position of the device to at least one object in the environment; operation information indicating the user's current operating mode of the device; at least one piece of information available via at least one network; and the current environmental state approximated based on the analysis of previously recorded environmental state information.

[0182] Example 3. The device according to any one of the foregoing embodiments, wherein the predetermined environmental conditions include at least one condition selected from the group consisting of: the ambient light level is within a predetermined suitable level range for performing calibration measurements; the device is in a suitable position for performing calibration measurements; the device is in a suitable orientation for performing calibration measurements; the device is not facing an external light source; the device is not near or pointing at an object; the device is not in a pocket; the device is at a temperature within a predetermined temperature range suitable for calibration measurements; the time-temperature variation is within a predetermined range suitable for calibration measurements; the weather conditions are within a predetermined range suitable for calibration measurements; the device is not currently used for another function; the device is available via at least one network.

[0183] Example 4. The device according to any one of the foregoing embodiments, wherein the device is configured to use at least one integrated sensor device of the device as at least part of the at least one state query device.

[0184] Example 5. The device according to any one of the preceding embodiments, wherein the status query device includes at least one device selected from the group consisting of: a front-facing camera positioned on the same side as the display; a rear-facing camera positioned on the opposite side of the display; a position sensor; an illumination sensor configured to determine at least one illumination state in the environment of the device; a temperature sensor; a motion sensor; a gyroscope sensor; a magnetic sensor; a material sensor configured to determine at least one material property of at least one object near the device; a spectrometer device configured to acquire at least one spectral information; and at least one software sensor configured to generate information about the status of the device by processing inputs from a plurality of physical sensors.

[0185] Example 6. The device according to any one of the preceding embodiments, wherein the status query device is configured to determine whether the device is placed on a table in a room by using at least one gyroscope sensor, and / or wherein the status query device is configured to determine the ambient light level by determining at least one dark image without a projector projecting multiple beams, and / or wherein the status query device is configured to determine whether the ceiling is visible by capturing at least one image with a projector projecting multiple beams, and / or wherein the status query device is configured to determine whether there is an obstacle near the device by using at least one proximity sensor.

[0186] Example 7. The device according to any one of the foregoing embodiments, wherein the control unit is configured to repeatedly and automatically trigger calibration, wherein the control unit is configured to perform status checks at predetermined time points, wherein each of these status checks includes receiving at least one status information about the current environmental state and checking whether the at least one predetermined environmental state condition is met, and wherein the control unit is further configured to trigger the calibration depending on whether the at least one environmental state condition is met.

[0187] Example 8. The device according to any one of the foregoing embodiments, wherein the calibration includes at least one user-guided calibration and / or automatic calibration, wherein the calibration includes one or more of calibration reference pattern, calibration of the position of the projector, at least one temperature calibration or at least one diffraction calibration.

[0188] Example 9. The device according to any one of the foregoing embodiments, wherein the calibration includes calibrating the reference pattern by performing the following steps:

[0189] i) While the projector projects the multiple beams, at least one calibration image is captured at the relative distance between the device and at least one reference object using an image generation unit;

[0190] ii) Determine the reflection pattern by identifying the light spots generated on the captured calibration image by the reference object in response to the illumination of the plurality of beams;

[0191] iii) Considering the different relative distances between the device and the reference object when capturing the calibration image, match the light spot of the reflection pattern with the features of the reference pattern to determine the matching reflection feature pairs;

[0192] iv) Recalculate the reference pattern based on the image coordinates of these corresponding matching spots.

[0193] Example 10. The device according to the previous embodiment, wherein determining the calibration image includes imaging at least one two-dimensional image of a white surface for different relative distances by using an image generation unit to project multiple light beams onto a projector.

[0194] Example 11. The device according to any one of the foregoing embodiments, wherein the calibration includes calibrating the position of the projector by performing the following steps:

[0195] I) While the projector projects the multiple beams, at least one calibrated facial image is captured at the relative distance between the device and the user's face using the image generation unit;

[0196] II) Estimate the relative distance by analyzing the calibrated facial image, wherein the analysis includes extracting key points of the face by using two-dimensional face detection;

[0197] III) Identify light spots on the user's face in the calibrated facial image and match these identified light spots with features of a reference pattern, taking into account the estimated relative distance, to determine matching pairs of reflection features;

[0198] IV) The translation vector describing the position of the projector is determined by determining the epipolar distance of each of these matched pairs of reflective features.

[0199] Example 12. The device according to the previous embodiment, wherein determining the calibrated facial image includes using the image generation unit to image at least one two-dimensional image of the user's face at the relative distance between the projector and the user's face when the projector projects the plurality of light beams.

[0200] Example 13. The device according to any one of the foregoing embodiments, wherein the calibration includes at least one temperature calibration, wherein the temperature calibration includes the following steps:

[0201] a) While the projector projects the plurality of beams, it captures at least one temperature-calibrated image of at least one reference object at at least a plurality of temperatures;

[0202] b) Identify the light spots on these temperature calibration images and, taking into account the estimated relative distance between the reference object and the projector, match the identified light spots with features of the reference pattern to determine a matching pair of reflection features for each temperature;

[0203] c) Determine the temperature-dependent correction of the reference pattern by determining the deviation between the positions of these matched reflection features for each temperature, and store the temperature-dependent correction of the reference pattern in at least one database.

[0204] Example 14. The device according to any one of the foregoing embodiments, wherein the calibration includes at least one diffraction calibration, wherein the diffraction calibration includes the following steps:

[0205] A) While the projector projects the multiple beams, at least one diffraction calibration image of at least one reference object is captured using a camera behind the perforation;

[0206] B) Identify the light spots on the diffraction calibration image and, taking into account the estimated relative distance between the reference object and the projector, match the identified light spots with features of the reference pattern to determine the matching reflection feature pairs;

[0207] C) The diffraction correction of the reference pattern is determined by determining the deviation between the positions of these matched spots and features, and the diffraction correction of the reference pattern is stored in at least one database.

[0208] Example 15. The device according to any one of the foregoing embodiments, wherein the device is selected from the group consisting of: television equipment; game consoles; personal computers; mobile devices, especially mobile communication devices, such as mobile phones, smartphones, tablet computers, laptop computers, tablet computers, virtual reality devices, or wearable devices such as smartwatches; or other types of portable computers.

[0209] Example 16. The device according to any one of the preceding embodiments, wherein the display is or includes at least one organic light-emitting diode (OLED) display and / or at least one quantum dot light-emitting diode (QLED) display.

[0210] Example 17. The device according to any one of the preceding embodiments, wherein extracting live data includes extracting material data and / or extracting blood perfusion data, wherein extracting material data includes providing the pattern image to a model and / or receiving material data from the model, wherein extracting blood perfusion data includes determining the speckle contrast of the pattern image and determining a blood perfusion measure based on the determined speckle contrast, wherein the speckle contrast represents a measure of the average contrast of the intensity distribution within the region of the speckle pattern.

[0211] Example 18. The device according to any one of the foregoing embodiments, wherein the automatic triggering of at least one calibration includes starting the calibration process without human-computer interaction, specifically without any human-computer interaction or with only optional human-computer interaction.

[0212] Example 19. A method for performing at least one reference measurement using the device according to any one of the foregoing embodiments, the method comprising:

[0213] i. Retrieve at least one piece of status information regarding the current environmental status of the device;

[0214] ii. Check whether at least one environmental condition for performing at least one calibration is met; and

[0215] iii. If the environmental conditions are met in step ii., at least one calibration is automatically triggered.

[0216] Example 20. The method according to the previous example, wherein steps i. to iii. of the method are repeated.

[0217] Example 21. A method for performing at least one reference measurement using the device according to any one of the foregoing embodiments of the method, the method comprising:

[0218] - Project multiple beams of light onto the user through at least one display using at least one projector;

[0219] - By using at least one image generation unit, a pattern image showing the projection of the plurality of light beams onto the user is generated;

[0220] - By using at least one processor to extract liveness data from the pattern image, and based on the liveness data, allow the user to perform authentication-required operations on the device;

[0221] - By using at least one control unit and at least one status query device to retrieve at least one status information about the current environmental status of the device, and by automatically triggering at least one calibration depending on the satisfaction of at least one predetermined environmental status condition.

[0222] Example 22. A computer program comprising instructions that, when executed by a control unit of a device according to any one of the foregoing embodiments relating to the device, cause the control unit to perform a method according to any one of the foregoing embodiments relating to the method.

[0223] Example 23. A computer-readable storage medium comprising instructions that, when executed by a control unit of the device according to any one of the foregoing embodiments of the related device, cause the control unit to perform a method according to any one of the foregoing embodiments of the related method. Attached Figure Description

[0224] Further optional features and embodiments will be disclosed in more detail, preferably in conjunction with the dependent claims, in the following embodiments. As those skilled in the art will recognize, the corresponding optional features can be implemented independently and in any feasible combination. The scope of the invention is not limited to the preferred embodiments. Embodiments are schematically depicted in the accompanying drawings. The same reference numerals in these drawings denote the same or functionally equivalent elements.

[0225] In the attached diagram:

[0226] Figure 1 An embodiment of the device according to the invention is shown;

[0227] Figure 2 A flowchart illustrating an embodiment of the method according to the present invention is shown;

[0228] Figures 3A to 3D An embodiment of the calibration reference pattern is shown; and

[0229] Figures 4A to 4D An example of calibrating the position of the projector is shown. Detailed Implementation

[0230] Figure 1 An embodiment of a device 110 for authenticating user 112 according to the present invention is shown in a highly illustrative manner.

[0231] Device 110 can be selected from the group consisting of: television equipment; game consoles; personal computers; mobile devices, especially mobile phones, and / or smartphones, and / or tablet computers, and / or laptop computers, and / or tablet computers, and / or virtual reality devices, and / or wearable devices such as smartwatches; or other types of portable computers. Specifically, the device can be a portable device.

[0232] Authentication may include verifying the identity of user 112. Specifically, authentication may include distinguishing user 112 from other humans or objects, particularly distinguishing authorized access from unauthorized access. Authentication may include verifying the identity of the corresponding user and / or assigning an identity to user 112. Authentication may include generating and / or providing identity information, such as providing it to other devices or units (e.g., providing it to at least one authorizing unit 114) for authorizing access to that device. The identity information can be proven through authentication. For example, the identity information may be and / or may include at least one identity token. If authentication is successful, it may be verified that the facial image recorded by at least one image generation unit is the user's facial image, and / or the user's identity is verified. Authentication may be performed using at least one authentication process. The authentication process may include multiple steps, such as at least one face detection (e.g., on at least one floodlight image, as will be described in more detail below), and at least one recognition step, wherein an identity is assigned to the detected face and / or at least one identity check and / or verification of the user's identity is performed.

[0233] Device 110 includes at least one projector 116 configured to project multiple light beams through at least one display 118 onto a user 112. Projection may include providing at least one light beam, particularly a light pattern, onto at least one surface. Projector 116 may be an optical device configured to project at least one light beam onto a surface. Projector 116 is configured to project multiple light beams. The multiple light beams may form a light pattern. Projector 116 may be configured to generate and / or provide at least one light pattern, particularly at least one infrared light pattern. The infrared light pattern may be a near-infrared light pattern. The light pattern may include at least one regular and / or constant and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or a pattern including further embossed patterns. For example, the light pattern is a hexagonal pattern, preferably a hexagonal infrared light pattern, preferably a 2 / 5 hexagonal infrared light pattern. Using a periodic 2 / 5 hexagonal pattern allows for the differentiation of artifacts and available signals. The light pattern may include at least one dot pattern.

[0234] Projector 116 may include at least one emitter, and in particular multiple emitters. The emitter may be selected from the group consisting of: at least one laser source, such as at least one semiconductor laser, at least one dual heterostructure laser, at least one external cavity laser, at least one independently confined heterostructure laser, at least one quantum cascade laser, at least one distributed Bragg reflector laser, at least one polaron laser, at least one hybrid silicon laser, at least one extended cavity diode laser, at least one quantum dot laser, at least one bulk Bragg grating laser, at least one indium arsenide laser, at least one gallium arsenide laser, at least one transistor laser, at least one diode-pumped laser, at least one distributed feedback laser, at least one quantum well laser, at least one interband cascade laser, at least one semiconductor ring laser, at least one vertical-cavity surface-emitting laser (VCSEL); at least one non-laser source, such as at least one LED or at least one bulb; at least one edge-emitting laser.

[0235] Display 118 may be or may include at least one organic light-emitting diode (OLED) display and / or at least one quantum dot light-emitting diode (QLED). The display may be at least partially transparent. The display may be at least partially transparent over at least one continuous area covering the projector 116, the floodlight source 120, and / or the image generation unit 122. Display 118 may have a transmittance of less than or equal to 20%, preferably less than or equal to 15%, more preferably less than or equal to 10%. For example, the intensity of a light beam after being projected through display 118 may correspond to ≤10% of the intensity associated with the light beam at the time of emission.

[0236] Device 110 includes at least one image generation unit 122 configured to generate a patterned image showing the projection of multiple light beams onto user 112. Image generation unit 122 can be configured to generate at least one image. The image can be generated via a hardware and / or software interface, which can be considered image generation unit 122. Image generation unit 122 may include at least one optical sensor, particularly at least one pixelated optical sensor. Image generation unit 122 may include at least one CMOS sensor or at least one CCD chip. For example, image generation unit 122 may include at least one CMOS sensor that is sensitive in the infrared spectral range. For example, image generation unit 122 may include one or more of the following: at least one monochrome camera (e.g., including monochrome pixels), at least one color (e.g., RGB) camera (e.g., including color pixels), at least one IR camera. The camera may be a CMOS camera. The camera may include at least one monochrome camera chip, such as a CMOS chip. The camera may include at least one color camera chip, such as an RGB CMOS chip. The camera may include at least one IR camera chip, such as an IR CMOS chip. For example, the camera may include monochrome (e.g., black and white) pixels and color pixels. Color pixels and monochrome pixels can be combined internally within the camera. A camera typically includes a one-dimensional or two-dimensional array of image sensors (such as pixels). For example, the camera can be an internal and / or external camera of a device. As described above, the internal and / or external cameras of a device can be accessed via hardware and / or software interfaces used as image generation units. In the case where device 110 is or includes a smartphone, image generation unit 122 can be the smartphone's front-facing camera (such as a selfie camera) and / or rear-facing camera.

[0237] The pattern image can be an image generated by the image generation unit while a light pattern is illuminating, for example, an object and / or a user. The pattern image can include an image showing at least a portion of the user, particularly the user's face, when illuminated by the light pattern, especially within a corresponding region of interest included in the image. The pattern image can be generated by imaging and / or recording light reflected from the object and / or user illuminated by the light pattern. The pattern image showing the user can include at least a portion of the illuminated light pattern on at least a part of the user. For example, the projection of the projector and the imaging using the image generation unit 122 can be synchronized, for example, by using at least one control unit of the device 110.

[0238] The device 110 may further include at least one floodlight source 120 configured to emit floodlight. The image generation unit 122 may be configured to generate at least one floodlight image when the floodlight source 120 emits floodlight. The emission of the floodlight and the illumination of the light pattern may be performed subsequently or at least partially overlapping times. For example, the floodlight and the light pattern may be emitted simultaneously. For example, one of the floodlight or the light pattern may be emitted at a lower intensity compared to the other.

[0239] Floodlight source 120 can be configured to emit floodlight, and image generation unit 122 can be configured to generate at least one floodlight image when floodlight source 120 emits floodlight. The floodlight image may include an image showing a user, particularly the user's face, when illuminated by the floodlight. The floodlight image can be generated by imaging and / or recording light reflected from the illuminated object and / or user. The floodlight image showing the user may include at least a portion of the floodlight on at least a part of the user. For example, illumination by floodlight source 120 and imaging using image generation unit 122 can be synchronized, for example, by using at least one control unit of device 110.

[0240] Device 110 includes at least one processor 124 configured to extract liveness data from the pattern image and, based on the liveness data, allow the user to perform an authentication-required operation on device 110. Specifically, device 110 may be configured to authenticate the user of device 110 to enable the performance of at least one authentication-required operation on the device. Authentication can be performed using processor 124. Processor 124 may be part of or be the at least one authentication unit configured to perform at least one authentication process for the user. The authentication unit may be configured to allow the user to perform an authentication-required operation on the device based on liveness data. Specifically, the authentication unit may be configured for a facial recognition authentication process operating on the floodlight image, the pattern image, and / or the extracted liveness data (particularly derived from the pattern image).

[0241] For example, the authentication unit can perform at least one face detection using a floodlight image. Face detection can be performed locally on device 110. However, face recognition (i.e., assigning identity to detected faces) can be performed remotely, for example, in the cloud, especially when identification rather than just verification is required. User templates can be stored at a remote device, for example, in the cloud, and do not need to be stored locally. This can be advantageous from a storage and security perspective. The authentication unit can be configured to identify users based on a floodlight image. Therefore, in particular, the authentication unit can forward data to a remote device. Alternatively or additionally, the authentication unit can perform user identification based on a floodlight image, particularly by running an appropriate computer program with corresponding functionality.

[0242] The authentication process may include multiple steps. For example, the authentication process may include performing at least one face detection. The face detection step may include analyzing a floodlight image. Additionally, for example, the authentication process may include identification. Identification may include assigning an identity to a detected face and / or at least one identity check and / or verification of the user's identity. Identification may include performing face verification on the imaged face to confirm whether it is the user's face. Identifying the user may include matching the floodlight image (e.g., showing the outline of parts of the user, particularly parts of the user's face) with a template. Identifying the user may include determining whether the imaged face is the user's face, and in particular determining whether the imaged face corresponds to at least one image of the user's face stored in at least one memory of a device, for example. If the floodlight image cannot match the image template, authentication may be unsuccessful.

[0243] The authentication process may include analyzing a flood image by, for example, one or more of the following: filtering; selecting at least one region of interest; forming a difference image between the flood image and at least one offset; inverting the flood image; background correction; decomposing into color channels; decomposing into hue, saturation, and luminance channels; frequency decomposition; singular value decomposition; applying a Canny edge detector; applying a Laplacian Gaussian filter; applying a difference Gaussian filter; applying the Sobel operator; applying the Laplacian operator; applying the Scharr operator; applying the Prewitt operator; applying the Roberts operator; applying the Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying the Radon transform; applying the Hough transform; applying a wavelet transform; thresholding; and creating a binary image. The region of interest may be manually determined by the user or may be automatically determined, for example, by identifying the user within the image. In particular, the analysis of the flood image may include using at least one image recognition technique, especially facial recognition technology. Image recognition technology includes at least one process of identifying a user in an image. Image recognition may include at least one technique using a combination of the following: color-based image recognition, for example using features such as template matching; segmentation and / or connected component (blob) analysis, for example using size or shape; machine learning and / or deep learning, for example using at least one convolutional neural network.

[0244] The authentication process may include extracting liveness data. Liveness data may include blood perfusion data and / or material data. Extracting liveness data may include extracting material data and / or extracting blood perfusion data. Liveness data may include information about the material on the user's surface to which the light spot is projected. Liveness data may include information about at least one vital sign. Extracting liveness data (e.g., by using an authentication unit) may include extracting material data from a pattern image through beam profile analysis of the light spot. For beam profile analysis, see WO 2018 / 091649 A1, WO 2018 / 091638 A1 and WO2018 / 091640 A1, the entire contents of which are incorporated herein by reference. Extracting material data from a pattern image may include generating material type and / or data derived from the material type. In addition to or as an alternative to using material data, extracting liveness data may include extracting blood perfusion data.

[0245] At least one authentication-required operation on device 110 may be accessing device 110 (e.g., unlocking device 110) and / or accessing an application preferably associated with device 110 and / or accessing a part of an application preferably associated with device 110.

[0246] The device 110 includes at least one control unit 126 and at least one status query device 128, which is configured to retrieve at least one piece of status information regarding the current environmental status of the device 110. The control unit 126 is configured to automatically trigger at least one calibration depending on the satisfaction of at least one predetermined environmental condition.

[0247] The status query device 128 may be a device or combination of devices configured to retrieve the at least one piece of status information. Specifically, the status query device 128 may include at least one interface (e.g., a wireless or wired interface) for retrieving status information in an electronic format (e.g., a data format), and / or at least one device (e.g., at least one sensor device) configured to generate status information, as will be further detailed below. The status query device 128 may also be wholly or partially integrated into the control unit 126. Alternatively or alternatively, the status query device 128 may include one or more devices integrated into a device for authentication, such as one or more integrated sensors.

[0248] As an example, state information regarding the current environmental state refers to one or more of the following: ambient lighting conditions, such as ambient light level and / or spatial location and / or orientation relative to external light sources; weather conditions, temperature information indicating the current temperature; location information of the device; orientation information of the device; relative orientation and / or relative position of the device to at least one object in the environment; operational information indicating the user's current operating mode of the device; at least one piece of information available via at least one network; and the current environmental state approximated based on analysis of previously recorded environmental state information. For example, the current environmental state can be approximated based on analysis of previously recorded environmental state information. To determine the current environmental state, previously obtained measurements can be used and / or considered. Actual measurements and previously obtained measurements can be combined.

[0249] The predetermined environmental state condition can be the condition of the environment in which the device for certification is located and / or the condition of the relationship between the device for certification and the environment, which is predetermined to be sufficient to perform calibration. As an example, the at least one environmental state condition can be at least one condition that must be met for certification to be performed based on at least one piece of status information regarding the current environmental state of the device. As an example, the status information can be compared with at least one maximum or minimum threshold, and the condition can be met when the status information is above the minimum threshold or below the maximum threshold, respectively, and as an example, calibration can be automatically triggered. Alternatively or additionally, the environmental state condition can be met when at least one piece of status information regarding the current environmental state of the device for certification and / or at least one minor value obtained from the at least one piece of status information using a predetermined relationship or function is within at least one predetermined range. Therefore, as an example, the control unit 126 can obtain at least one piece of status information regarding the current environmental state from a status query device, optionally transform the at least one piece of status information and convert it into at least one minor value, and then check whether the at least one piece of status information and / or the at least one minor value meets the environmental state condition, such as having a predetermined target value or being within a predetermined range. If these environmental conditions are met, the control unit can automatically trigger at least one calibration, with or without delay.

[0250] The predetermined environmental conditions may include at least one condition selected from the group consisting of: ambient light level within a predetermined suitable range for performing calibration measurements; the device being in a suitable location for performing calibration measurements; the device being in a suitable orientation for performing calibration measurements; the device not facing an external light source; the device not being near or pointing at an object; the device not being in a pocket; the device being at a temperature within a predetermined temperature range suitable for calibration measurements; time-temperature variation within a predetermined range suitable for calibration measurements; weather conditions within a predetermined range suitable for calibration measurements; the device not currently being used for another function; and the device being available via at least one network.

[0251] Therefore, when predetermined environmental conditions indicate that the environmental conditions are suitable for at least one calibration, using at least one piece of status information regarding the current environmental state of the device for certification allows the device 110 for certification to repeatedly perform calibration. Specifically, the at least one piece of status information can be retrieved using the integrated device for certification.

[0252] Device 110 may be configured to use at least one integrated sensor device as at least part of the at least one status query device. Therefore, one or more integrated sensor devices that are present in the device anyway may be used, for one or more other purposes. Status query device 128 may include at least one device selected from the group consisting of: a front-facing camera positioned on the same side as the display; a rear-facing camera positioned on the opposite side of the display; a position sensor; an illumination sensor configured to determine at least one illumination state in the environment of the device; a temperature sensor; a motion sensor; a gyroscope sensor; a magnetic sensor; a material sensor configured to determine at least one material property of at least one object near the device; a spectrometer device configured to acquire at least one spectral information; and at least one software sensor configured to generate information about the status of the device by processing inputs from multiple physical sensors.

[0253] The status query device 128 can be configured to determine whether the device is placed on a table in a room by using at least one gyroscope sensor, and / or wherein the status query device 128 is configured to determine the ambient light level by determining at least one dark image without projecting multiple beams of light from the projector 116, and / or wherein the status query device 128 is configured to determine whether the ceiling is visible by capturing at least one image with projecting multiple beams of light from the projector 116, and / or wherein the status query device 128 is configured to determine whether there is an obstacle near the device 110 by using at least one proximity sensor.

[0254] For example, device 110 can attempt to detect whether it is on a table in the room. A gyroscope sensor can support the detection of whether the device (e.g., a smartphone) is placed flat on the table. For example, the device can capture image frames without any floodlight or light from the projector to determine if the room is dark, i.e., with little sunlight. For example, a proximity sensor can check for obstacles on the device (e.g., a smartphone). For example, device 110 can capture image frames with the projector on. If there is a ceiling, this will be visible on the laser frame. Device 110 can detect the light pattern on the frame for calibration.

[0255] For example, calibration includes one or more of the following: calibrating a reference pattern, calibrating the position of the projector, at least one temperature calibration, or at least one diffraction calibration.

[0256] Figure 2A flowchart illustrating an embodiment of a method for performing at least one reference measurement using a device according to the invention is shown. In this method, a device according to the invention, such as any of the embodiments described above and / or any of the embodiments further described in detail below, is used.

[0257] The method includes:

[0258] i. (130) Retrieve at least one status information regarding the current environmental state of the device 110;

[0259] ii. (132) Check that at least one environmental condition is met for performing the calibration, particularly at least one calibration measurement; and

[0260] iii. (134) If the environmental conditions in step ii. are met, calibration, in particular at least one calibration measurement, is automatically triggered.

[0261] These method steps can be executed in a given order or in a different order. Furthermore, there may be one or more additional method steps not listed. Furthermore, one, more than one, or even all of the method steps may be executed repeatedly.

[0262] The calibration includes at least one user-guided calibration and / or automatic calibration.

[0263] Calibration may include procedures as described in WO 2022 / 253777, the contents of which are included herein by reference.

[0264] Figures 3A to 3D and Figures 4A to 4D An example of calibration is shown. In Figures 3A to 3D An embodiment of the calibration reference pattern is shown in the figure, and in Figure 4A and Figure 4D The document describes an embodiment for calibrating the position of the projector.

[0265] Calibration may include calibrating a reference pattern, and / or particularly calibrating the position of the projector.

[0266] Calibration may include calibrating a reference pattern by performing the following steps:

[0267] i) While the projector 116 projects the plurality of beams, at least one calibration image is captured at the relative distance between the device 110 and at least one reference object 136 by using the image generation unit 122;

[0268] ii) Determine the reflection pattern by identifying the light spots generated by the reference object 136 in response to the illumination of the plurality of beams on the captured calibration image;

[0269] iii) Taking into account the different relative distances between the device 110 and the reference object 136 when capturing the calibration image, the spot of the reflection pattern is matched with the features of the reference pattern to determine the matching reflection feature pair;

[0270] iv) Recalculate the reference pattern based on the image coordinates of these corresponding matching spots.

[0271] In step i), as Figure 3A and Figure 3B As shown, multiple calibration images can be captured at different relative distances, for example, at least one calibration image can be captured at each relative distance. Determining the calibration images can include imaging at least one two-dimensional image of a white surface at different relative distances using an image generation unit while multiple beams are projected by a projector. For example, calibration can include identifying at least one flat white surface (e.g., a wall), for example, by using a selfie camera while carrying a smartphone. For example, as... Figure 3A and Figure 3B As shown, user 112 is requested to capture at least two calibration images of the wall at a coarse distance. The calibration images can be captured at a distance of 1.5 m to 3 m. Figure 3C The image shows the imaging of a light spot grid using device 110.

[0272] At least one evaluation device (e.g., processor 124) can be configured to perform image analysis and identify spots in the calibration image. The evaluation device can check whether the imaging grid of the calibration image is complete. Then, the spots of the grid are matched with a reference pattern (e.g., a nominally known grid). In step iv), the estimated relative distance from the reference object 136 to the device 110 can be used. The estimated relative distance can be assumed to be at a nominal distance, or it can be determined, for example, by illumination by a laser spot. Next, the reference pattern is recalculated. This recalculation can be performed using the image coordinates of the corresponding matched spots. The image coordinates of the spots can be mapped to projector coordinates. This will produce a calibrated reference grid. Figure 3D A comparison of the spot grid between the initial reference pattern and the calibration image is shown. A miscalibration can be observed, which can be corrected by recalculating the reference pattern.

[0273] For example, a complete automatic calibration can be performed as follows, and may include the following steps:

[0274] The smartphone attempts to detect if it is on a table in the room. A gyroscope sensor can help detect whether the smartphone is lying flat on the table.

[0275] - In the absence of any floodlights and projectors, image frames are captured using an image generation unit to determine whether the room is dark, i.e., there is little sunlight.

[0276] - Proximity sensors can detect obstacles on a smartphone.

[0277] - With the projector on, the image generation unit captures an image frame. If there is a ceiling on the image frame, a calibration image is projected onto the frame for calibration, and calibration steps i) to iv) are performed to calibrate the reference pattern.

[0278] Calibration may include calibrating the position of projector 116. Calibration of the position of projector 116 may be performed after calibration of the reference pattern. Calibration of the position of projector 116 may include the following steps:

[0279] I) While the projector 116 projects the multiple beams, at least one calibrated facial image is captured at the relative distance between the face of the device 110 and the user 112 by using the image generation unit 122.

[0280] II) Estimate the relative distance by analyzing the calibrated facial image, wherein the analysis includes extracting key points of the face by using two-dimensional face detection;

[0281] III) Identify light spots on the face of user 112 in the calibrated facial image, and match these identified light spots with features of a reference pattern, taking into account the estimated relative distance, to determine the matching reflection feature pairs;

[0282] IV) The translation vector describing the position of the projector is determined by determining the epipolar distance of each of these matched pairs of reflective features.

[0283] The calibration facial image can be an image of at least a portion of the user 112's face used for calibration. Determining the calibration facial image may include imaging at least one two-dimensional image of the user's face at the relative distance between the projector 116 and the user 112's face using the image generation unit 122, with multiple beams projected by the projector. The calibration facial image may be captured, for example, during registration or unlocking. For example, a 2D image generated using a selfie camera may be used. Figure 4A and Figure 4B The image shows two calibrated facial images of a user's face at different distances between the user 112 and the device 110, obtained by using a selfie camera.

[0284] Calibrate facial images (e.g., such as...) Figure 4CAs shown, facial keypoints can be extracted using 2D face detection for analysis. Keypoint extraction can be performed, for example, as described in en.wikipedia.org / wiki / Landmark_detection. This allows for the estimation of facial distances. Alternatively, a distance sensor can be used.

[0285] The light pattern of the calibrated facial image can be analyzed. For example, in step III), only the light spot on the face can be used. The light spot can be matched with a reference pattern, and the matching can be performed as described above. This is possible because the facial distance is known by using key points. The face can be in the near field.

[0286] By matching the light spot with a reference pattern, the epipolar direction can be extracted. Figure 4D The matching of the light spot on the face with a reference pattern is shown, which produces the epipolar direction. This produces the relative position of projector 116, specifically the translation vector that positions the projector. The length of the translation vector is the baseline length. This value is known in the hardware design. Calibration is complete. The reference grid and projector position (translation) are estimated.

[0287] List of reference numerals

[0288] 110 equipment

[0289] 112 users

[0290] 114 Authorization Unit

[0291] 116 Projector

[0292] 118 monitor

[0293] 120 floodlight source

[0294] 122 Image Generation Unit

[0295] 124 processor

[0296] 126 control unit

[0297] 128 Status Query Device

[0298] 130 Search

[0299] 132 Inspection

[0300] 134 automatically triggers calibration

[0301] 136 Reference objects.

Claims

1. A device (110) for authenticating a user (112), the device (110) comprising: - at least one projector (116) configured for projecting a plurality of light beams through at least one display (118) onto the user (112), - at least one image generation unit (122) configured for generating a pattern image showing the projection of the plurality of light beams onto the user (112); - at least one processor (124) configured for extracting vital data from the pattern image and allowing the user (112) to perform an operation on the device (110) requiring authentication based on the vital data; - at least one control unit (126) and at least one state query device (128) configured for retrieving at least one item of state information about a current environmental state of the device (110), wherein the control unit (126) is configured for automatically triggering at least one calibration depending on the state information about the current environmental state satisfying at least one predetermined environmental state condition.

2. The device (110) according to the preceding claim, wherein, The state information about a current environmental state refers to one or more of the following: an environmental lighting condition, such as an ambient light level and / or a spatial position and / or orientation relative to an external light source; a weather condition, temperature information indicative of a current temperature; position information of the device (110); orientation information of the device (110); a relative orientation and / or relative position between the device (110) and at least one item in the environment; operation information indicative of a current mode of operation of the device (110) by the user (112); at least one item of information available via at least one network; a current environmental state approximated from an analysis of previously recorded environmental state information.

3. The device (110) according to any of the preceding claims, wherein, The predetermined environmental state condition comprises at least one condition selected from the group consisting of: an ambient light level being within a predetermined suitable level range for performing a calibration measurement; the device (110) being in a suitable position for performing a calibration measurement; the device (110) being in a suitable orientation for performing a calibration measurement; the device (110) not facing an external light source; the device (110) not being in the vicinity of or not being directed towards an object; the device (110) not being located within a pocket; the device (110) being at a temperature within a predetermined temperature range suitable for a calibration measurement; a temporal temperature change being within a predetermined range suitable for a calibration measurement; a weather condition being within a predetermined range suitable for a calibration measurement; the device (110) not currently being used for another function; the device (110) being available via at least one network.

4. The device (110) according to any of the preceding claims, wherein, The state querying device (128) comprises at least one device selected from the group consisting of: a front-facing camera positioned on the same side as the display (118); a back-facing camera positioned on the opposite side of the display (118); a position sensor; an illumination sensor configured for determining at least one illumination state in the environment of the device; a temperature sensor; a motion sensor; a gyroscope sensor; a magnetic sensor; a material sensor configured for determining at least one material property of at least one object in the vicinity of the device (110); a spectrometer device configured for acquiring at least one spectral information; at least one software sensor configured for generating information about the state of the device (110) by processing input from a plurality of physical sensors.

5. The device (110) according to any of the preceding claims, wherein, The calibration comprises at least one user-guided calibration and / or an automatic calibration, wherein the calibration comprises one or more of a calibration reference pattern, a calibration of the position of the projector, at least one temperature calibration, or at least one diffraction calibration.

6. The device (110) according to any of the preceding claims, wherein, The calibration comprises calibrating a reference pattern by performing the following steps: i) capturing at least one calibration image at a relative distance between the device (110) and at least one reference object (136) by using the image generation unit (122) while the projector (116) projects the plurality of light beams; ii) determining a reflection pattern by identifying spots generated on the captured calibration image by the reference object (136) in response to an illumination by the plurality of light beams; iii) matching the spots of the reflection pattern with features of a reference pattern, taking into account an estimation of different relative distances between the device (110) and the reference object (136) at the time of capturing the calibration image, thereby determining matched reflection feature pairs; iv) recomputing the reference pattern from the image coordinates of the corresponding matched spots.

7. The device (110) according to any of the preceding claims, wherein, The calibration comprises calibrating the position of the projector by performing the following steps: I) capturing at least one calibration face image at a relative distance between the device (110) and a face of a user (112) by using the image generation unit (122) while the projector (116) projects the plurality of light beams; II) estimating the relative distance by analyzing the calibration face image, wherein the analysis comprises extracting key points of the face by using two-dimensional face detection; III) identifying spots on the calibration face image that lie on the face of the user (112) and matching the identified spots with features of a reference pattern, taking into account the estimated relative distance, thereby determining matched reflection feature pairs; IV) determining a translation vector describing the position of the projector (116) by determining epipolar line distances of each of the matched reflection feature pairs.

8. The device (110) according to any of the preceding claims, wherein The calibration comprises at least one temperature calibration, wherein the temperature calibration comprises the following steps: a) capturing at least one temperature calibration image of at least one reference object at at least multiple temperatures while the projector (116) projects the multiple light beams; b) identifying light spots on the temperature calibration images and matching the identified light spots to features of a reference pattern, taking into account an estimated relative distance between the reference object and the projector (116), thereby determining for each temperature a matched pair of reflected features; c) determining a temperature-dependent correction of the reference pattern by determining deviations between positions of the matched reflected features for each temperature, and storing the temperature-dependent correction of the reference pattern in at least one database.

9. The device (110) according to any of the preceding claims, wherein, The calibration comprises at least one diffraction calibration, wherein the diffraction calibration comprises the following steps: A) capturing at least one diffraction calibration image of at least one reference object by using a camera behind a pinhole while the projector (116) projects the multiple light beams; B) identifying light spots on the diffraction calibration image and matching the identified light spots to features of a reference pattern, thereby determining a matched pair of reflected features, taking into account an estimated relative distance between the reference object and the projector (116); C) determining a diffraction correction of the reference pattern by determining deviations between positions of the matched light spots and features, and storing the diffraction correction of the reference pattern in at least one database.

10. The device (110) according to any of the preceding claims, wherein, The device (110) is selected from the group consisting of: a television device; a game console; a personal computer; a mobile device, in particular a mobile communication device, such as a mobile phone, a smartphone, a tablet computer, a laptop computer, a tablet computer, a virtual reality device, or a wearable device, such as a smartwatch; or other types of portable computers.

11. The device (110) according to any of the preceding claims, wherein The display (118) is or comprises at least one organic light emitting diode (OLED) display and / or at least one quantum dot light emitting diode (QLED) display.

12. The device (110) according to any of the preceding claims, wherein, Extracting vital data comprises extracting material data and / or extracting blood perfusion data, wherein extracting material data comprises providing the pattern image to a model and / or receiving material data from the model, and wherein extracting blood perfusion data comprises determining a speckle contrast of the pattern image and determining a blood perfusion measure based on the determined speckle contrast, wherein the speckle contrast represents a measure of the average contrast of the intensity distribution within a region of a speckle pattern.

13. A method of performing at least one reference measurement with a device (110) according to any one of the preceding claims, the method comprising: i. (130) retrieving at least one item of status information about a current environmental state of the device (110); ii. (132) checking whether at least one environmental state condition for performing at least one calibration measurement is fulfilled; and iii. (134) if in step ii. the status information about the current environmental state fulfils the environmental state condition, automatically triggering at least one calibration measurement.

14. A computer program comprising instructions which, when the program is executed by a control unit of a device according to any of the preceding claims relating to a device, cause the control unit to carry out the method according to any of the preceding claims relating to a method.

15. A computer-readable storage medium comprising instructions which, when executed by a control unit of a device according to any of the preceding claims relating to a device, cause the control unit to carry out the method according to any of the preceding claims relating to a method.

Images