Face authentication reference measurement timing

Abstract

A device (110) for authenticating a user (112) is proposed. The device (110) comprising: - at least one projector (116) configured for projecting a plurality of light beams through at least one display (118) onto the user (112), - at least one image generation unit (122) configured for generating a pattern image showing the projecting of the plurality of light beams onto the user (112); - at least one processor (124) configured for extracting liveness data from the pattern image and allowing the user (112) to perform an operation on the device (110) that requires authentication based on the liveness data; - at least one control unit (126) and at least one status inquiry device (128) configured for retrieving at least one item of status information on a current environmental status of the device (110), wherein the control unit (126) is configured for automatically triggering at least one calibration depending on the fulfillment of at least one predetermined environmental status condition.

Description

[0001] Face authentication reference measurement timing

[0002] Technical Field

[0003] The invention relates to a device for authenticating a user and a method of performing at least one reference measurement. The present invention further relates to a computer program, a computer-readable storage medium and a non-transient computer-readable medium. The devices, methods and uses according to the present invention specifically may be employed for example in various areas of daily life, security technology, gaming, traffic technology, production technology, photography such as digital photography or video photography for arts, documentation or technical purposes, safety technology, information technology, agriculture, crop protection, maintenance, cosmetics, medical technology or in the sciences. However, other applications are also possible.

[0004] Background art

[0005] Available authentication systems in mobile devices such as in smartphones, tablets and the like, include receivers such as at least one camera. Said mobile devices usually have a display such as an organic light-emitting diode (OLED) area and / or a quantum-dot light emitting diode (QLED) area. The receiver may be positioned behind said display. Moreover, in such devices for authentication a light emitter such as a projector is used, such as one or more light emitting diodes and / or laser, may be positioned behind the display. Usually, the light emitter projects a pattern such as a point pattern onto a target, e.g. a face, the receiver, e.g. the camera, captures an image of the projection onto the user and material information is determined by a processor. If the material is classified as skin, it is identified as human and, otherwise, as a spoof target. The pattern generated by the light projector can be designed for a 3D algorithm, i.e. the pattern may be designed to allow easily solving the so-called correspondence problem. For smartphone application, a resulting 3D depth map can be used for further face authentication.

[0006] During manufacturing of a sensor head for such an authentication system and built in of the sensor head into a mobile device tolerances are introduced. In order to use the mobile device for distance sensing a calibration step of the sensor head and / or the finished mobile device is necessary. This is particularly critical for behind OLED integration due to the strong influence of sensor head placement on diffraction characteristics. A typical calibration step may comprise the capture of images of a white, flat surface at different distances. However, calibration processes take time and require machines which makes them costly. Furthermore changes in dimensions due to warp or other stress on the mobile device and / or its components can degrade the performance of face authentication or distance sensing during the lifetime of the product. Moreover, it would be desired to prevent to perform the calibration task by users, both at product bring-up by the user or in regular intervals during the lifetime. While this may be sometimes required for certain sensors (e.g. white balancing of cameras, magnetometer calibration of the compass in smartphones), the task can be considered a nuisance and also holds the risk of improper execution resulting in performance issues.

[0007] WO 2022 / 253777A1 describes a detector for determining a position of at least one object. The detector comprises: - at least one projector for illuminating the object with at least one illumination pattern, wherein the illumination pattern comprises a plurality of illumination features; - at least one sensor element having a matrix of optical sensors, the optical sensors each having a light-sensitive area, wherein each optical sensor is designed to generate at least one sensor signal in response to an illumination of its respective light-sensitive area by a reflection light beam propagating from the object to the detector, wherein the sensor element is configured to determine at least one reflection image comprising a plurality of reflection features, wherein each of the reflection features comprises a beam profile; - at least one evaluation device configured for determining initial distance information of the reflection features by analysis of their respective beam profiles, wherein the analysis of a beam profile comprises evaluating a combined signal Q from the respective sensor signals, wherein the evaluation device is configured for performing a calibration method comprising: a) matching the reflection features to reference features of a reference image considering the initial distance information thereby determining pairs of matched reflection and reference features; b) Determining an epipolar line of the matched reference feature in the reference image for each of the pairs of matched reflection and reference features; c) determining an epipolar line distance d of the matched reflection feature to said epipolar line; d) valuating the epipolar line distances d as a function of an image position (x,y) in the reference image thereby determining a geometric pattern; e) determining at least one correction for rotation and / or translation of the reflection image depending on the geometric pattern.

[0008] Guoqiang Yang et al. "An Integrated Solution for Under Screen Face Authentication", SID SYMPOSIUM DIGEST OF TECHNICAL PAPERS, WILEY-BLACKWELL PUBLISHING, INC, US, vol. 54, 3 August 2023, pages 323-326, XP072509100, ISSN: 0097-966X, DOI: 10.1002 / SDTP.16294 describes under Screen Face Authentication (USFA).

[0009] Problem to be solved

[0010] It is therefore an object of the present invention to provide devices and methods facing the abovementioned technical challenges of known devices and methods. Specifically, it is an object of the present invention to provide devices and methods, which allow for reducing costs for a factory calibration and for performing calibration of a 3D imager behind OLED in a complete assembled mobile device.

[0011] Summary

[0012] This problem is addressed by a device for authenticating a user and a method of performing at least one reference measurement with the features of the independent claims. Advantageous embodiments which might be realized in an isolated fashion or in any arbitrary combinations are listed in the dependent claims as well as throughout the specification.

[0013] In a first aspect of the present invention, a device for authenticating a user, is disclosed. The device comprises: at least one projector configured for projecting a plurality of light beams through at least one display onto the user, at least one image generation unit configured for generating a pattern image showing the projecting of the plurality of light beams onto the user; at least one processor configured for extracting liveness data from the pattern image and allowing the user to perform an operation on the device that requires authentication based on the liveness data; at least one control unit and at least one status inquiry device configured for retrieving at least one item of status information on a current environmental status of the device, wherein the control unit is configured for automatically triggering at least one calibration depending on the fulfillment of at least one predetermined environmental status condition.

[0014] The device may be selected from the group consisting of: a television device; a game console; a personal computer; a mobile device, particularly a cell phone, and / or a smart phone, and / or, and / or a tablet computer, and / or a laptop, and / or a tablet, and / or a virtual reality device, and / or a wearable, such as a smart watch; or another type of portable computer. The device, in particular, may be a portable device. The term “portable” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to the property of at least one object of being moved by human force, such as by a single user. Specifically, the object characterized by the term “portable” may have a weight not exceeding 10 kg, specifically not exceeding 5 kg, more specifically not exceeding 1 kg or even not exceeding 500 g. Additionally or alternatively, the dimensions of the object characterized by the term “portable” may be such that the object extends by no more than 0.3 m into any dimension, specifically by no more than 0.2 m into any dimension. The object, specifically, may have a volume of no more than 0.03 m3, specifically of no more than 0.01 m3or even no more than 0.001 m3.

[0015] The term “user” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a person intended to and / or using the device.

[0016] The term “authenticating” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to verifying an identity of a user. Specifically, the authentication may comprise distinguishing between the user from other humans or objects, in particular between an authorized access from a non-authorized access. The authentication may comprise verifying identity of a respective user and / or assigning identity to a user. The authentication may comprise generating and / or providing identity information, e.g. to other devices or units such as to at least one authorization unit for authorization for providing access to the device. The identify information may be proofed by the authentication. For example, the identity information may be and / or may comprise at least one identity token. In case of successful authentication an image of a face recorded by at least one image generation unit may be verified to be an image of the user’s face and / or the identity of the user is verified. The authenticating may be performed using at least one authentication process. The authentication process may comprise a plurality of steps such as at least one face detection, e.g. on at least one flood image as will be described in more detail below, and at least one identification step in which an identity is assigned to the detected face and / or at least one identity check and / or verifying an identity of the user is performed.

[0017] The term “light” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to electromagnetic radiation in one or more of the infrared, the visible and the ultraviolet spectral range. Herein, the term “ultraviolet spectral range”, generally, refers to electromagnetic radiation having a wavelength of 1 nm to 380 nm, preferably of 100 nm to 380 nm. Further, in partial accordance with standard ISO- 21348 in a valid version at the date of this document, the term “visible spectral range”, generally, refers to a spectral range of 380 nm to 760 nm. The term “infrared spectral range” (IR) generally refers to electromagnetic radiation of 760 nm to 1000 pm, wherein the range of 760 nm to 1 .5 pm is usually denominated as “near infrared spectral range” (NIR) while the range from 1 .5 p to 15 pm is denoted as “mid infrared spectral range” (MidlR) and the range from 15 pm to 1000 pm as “far infrared spectral range” (FIR). Preferably, light used for the typical purposes of the present invention is light in the infrared (IR) spectral range, more preferred, in the near infrared (NIR) and / or the mid infrared spectral range (MidlR). Specifically, the projector projects at least one infrared light pattern, wherein the projected light beams have a wavelength in the infrared spectral range, preferably from 800 nm to 1300 nm, more preferably from 900 nm to 1000 nm, most preferably from 1100 nm to 1200 nm.

[0018] The term “ray” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a line that is perpendicular to wavefronts of light which points in a direction of energy flow. The term “light beam” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a collection of rays. In the following, the terms “ray” and “beam” will be used as synonyms. The term “light beam” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to an amount of light, specifically an amount of light traveling essentially in the same direction, including the possibility of the light beam having a spreading angle or widening angle. The light beam may have a spatial extension. Specifically, the light beam may have a non-Gaussian beam profile. The beam profile may be selected from the group consisting of a trapezoid beam profile; a triangle beam profile; a conical beam profile. The trapezoid beam profile may have a plateau region and at least one edge region. The light beam specifically may be a Gaussian light beam or a linear combination of Gaussian light beams, as will be outlined in further detail below. Other embodiments are feasible, however.

[0019] The term “projecting”, as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to the process of providing at least one light beam, in particular a light pattern onto at least one surface. The term “projector”, as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to an optical device configured for projecting at least one light beam onto a surface. The projector is configured for projecting a plurality of light beams. The plurality of light beams may form a light pattern. The projector may be configured for generating and / or for providing at least one light pattern, in particular at least one infrared light pattern.

[0020] The term “light pattern” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to at least one arbitrary pattern comprising a plurality of light spots. The light spot may be at least partially spatially extended. At least one spot or any spot may have an arbitrary shape. In some cases a circular shape of at least one spot or any spot may be preferred. The spots may be arranged by considering a structure of the display. Typically, an arrangement of an OLED-pixel-structure of the display may be considered.

[0021] The light pattern may be an infrared light pattern. The term “infrared light pattern” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a light pattern comprising spots in the infrared spectral range. The infrared light pattern may be a near infrared light pattern.

[0022] The light projected by the projector may be coherent. The light pattern may be a coherent, in particular infrared, light pattern. The projector may be configured for emitting light at a single wavelength, e.g. in the near infrared region. In other embodiments, the projector may be adapted to emit light with a plurality of wavelengths, e.g. for allowing additional measurements in other wavelengths channels. The light pattern may comprise at least one regular and / or constant and / or periodic pattern such as a triangular pattern, a rectangular pattern, a hexagonal pattern or a pattern comprising further convex tilings. For example, the light pattern is a hexagonal pattern, preferably a hexagonal light pattern, preferably a 2 / 5 hexagonal infrared light pattern. Using a periodical 2 / 5 hexagonal pattern can allow distinguishing between artefacts and usable signal.

[0023] The light pattern may comprise at least one point pattern.

[0024] At least one of the light spots may be associated with a beam divergence of 0.2° to 0.5°, preferably 0.1 ° to 0.3°. The term “beam divergence” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to at least one measure of an increase in at least one diameter and / or at least one diameter equivalent, such as a radius, with a distance from an optical aperture from which the beam emerges. The measure may be an angle or an angle equivalent. In the context of the present invention, typically, a beam divergence may be determined at 1 / e2.

[0025] The projector may comprise at least one least one emitter, in particular a plurality of emitters. The term “emitter” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to at least one arbitrary device configured for providing at least one light beam. The emitter may be selected from the group consisting of: at least one laser source such as at least one semi-conductor laser, at least one double heterostructure laser, at least one external cavity laser, at least one separate confinement heterostructure laser, at least one quantum cascade laser, at least one distributed Bragg reflector laser, at least one polariton laser, at least one hybrid silicon laser, at least one extended cavity diode laser, at least one quantum dot laser, at least one volume Bragg grating laser, at least one Indium Arsenide laser, at least one Gallium Arsenide laser, at least one transistor laser, at least one diode pumped laser, at least one distributed feedback lasers, at least one quantum well laser, at least one interband cascade laser, at least one semiconductor ring laser, at least one vertical cavity surface emitting laser (VCSEL); at least one non-laser light source such as at least one LED or at least one light bulb; at least one edge-emitting laser.

[0026] For example, the projector comprises at least one VCSEL, preferably a plurality of VCSELs. The plurality of VCSELs may be arranged in at least one array, e.g. comprising a matrix of VCSELs. The VCSELs may be arranged on the same substrate, or on different substrates. The term “vertical-cavity surface-emitting laser” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a semiconductor laser diode configured for laser beam emission perpendicular with respect to a top surface. Examples for VCSELs can be found e.g. in en.wikipedia.org / wiki / Verticalcavity_sur- face-emittingjaser. VCSELs are generally known to the skilled person such as from WO 2017 / 222618 A. Each of the VCSELs is configured for generating at least one light beam. The VCSEL or the plurality of VCSELs may be configured for generating the desired spot number. The VCSELs may be configured for emitting light beams at a wavelength range from 800 to 1000 nm. For example, the VCSELs may be configured for emitting light beams at 808 nm, 850 nm, 940 nm, and / or 980 nm. Preferably the VCSELs emit light at 940 nm, since terrestrial sun radiation has a local minimum in irradiance at this wavelength, e.g. as described in CIE 085- 1989 „Solar spectral Irradiance”.

[0027] The display may be configured for modifying the spots, e.g. by increasing a number of spots, generated by the projector when traversing the display. For example, the display may act as diffractive optical element (DOE). Thus, resources can be saved since the display functions as DOE and such that no further DOE is necessary. Moreover, less emitter, e.g. VCSEL cavities, are necessary since the light beams can be multiplicated by the display. Additionally or alterma- tively, the projector may comprise at least one optical element, e.g. configured for modifying the spots, e.g. by increasing a number of spots, selected from the group consisting of: at least one lens; at least one Micro-lens-array (MLA); at least one diffractive optical element (DOE); and at least one meta surface element. The DOE and / or the metasurface element may be configured for generating multiple light beams from a single incoming light beam. For example, a VCSEL projecting up to 2000 spots and an optical element comprising a plurality of metasurface elements may be used to duplicate the number of spots. Further arrangements, particularly comprising a different number of projecting VCSEL and / or at least one different optical element configured for increasing the number of spots may be possible. Other multiplication factors are possible. For example, a VCSEL or a plurality of VCSELs may be used and the generated laser spots may be duplicated by using at least one DOE.

[0028] The projector comprise at least one transfer device. The term “transfer device”, also denoted as “transfer system”, as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to one or more optical elements which are adapted to modify the light beam, particularly the light beam used for generating at least a portion of the infrared light pattern, such as by modifying one or more of a beam parameter of the light beam, a width of the light beam or a direction of the light beam. The transfer device may comprise at least one imaging optical device .The transfer device specifically may comprise one or more of: at least one lens, for example at least one lens selected from the group consisting of at least one focus-tunable lens, at least one aspheric lens, at least one spherical lens, at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflection element, preferably at least one mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitting mirror; at least one multi-lens system; at least one holographic optical element; at least one meta optical element. Specifically, the transfer device comprises at least one refractive optical lens stack. The transfer device may comprise a multi-lens system having refractive properties.

[0029] The device may further comprise at least one flood illumination source configured for emitting flood light. The image generation unit may be configured for generating at least one flood image while the flood illumination source is emitting flood light. The term “flood illumination source” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to at least one arbitrary device configured for providing substantially continuous spatial illumination. The term “flood light” as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to substantially continuous spatial illumination, in particular diffuse and / or uniform illumination. The flood light has a wavelength in the infrared range, in particular in the near infrared range. The flood illumination source may comprise at least one LED or at least one VCSEL, preferably a plurality of VCSELs. The plurality of VCSELs may overlap to a uniform area. The term “substantially continuous spatial illumination” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to uniform spatial illumination, wherein areas of non-uniform are possible. The area, e.g. covering a user, a portion of the user and / or a face of the user, illuminated from the flood illumination source, may be contiguous. Power may be spread over a whole field of illumination. In contrast, illumination provided by the light pattern may comprise at least two contiguous areas, in particular a plurality of contiguous areas, and / or power may be concentrated in small (compared to the whole field of illumination) areas of the field of illumination. The infrared flood illumination may be suitable for illuminating a contiguous area, in particular one contiguous area. The infrared pattern illumination may be suitable for illuminating at least two contiguous areas.

[0030] The flood illumination source may illuminate a measurement area, such as a user, a portion of the user and / or a face of the user, with a substantially constant illumination intensity. The term “constant” as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a time aspect during an exposure time. Flood light may vary temporally and / or may be substantially constant over time. The term “substantially constant” as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a completely constant illumination and embodiments in which deviations from a constant illumination of < ± 10 %, preferably < ± 5 %, more preferably < ± 2 % are possible.

[0031] The emitting of the flood light and the illumination of the light pattern may be performed subsequently or at at least partially overlapping times. For example, the flood light and the light pattern may be emitted at the same time. For example, one of the flood light or the light pattern may be emitted with a lower intensity compared to the other one.

[0032] The projector and the flood illumination source may comprise at least one VCSEL, preferably a plurality of VCSELs. The projector may comprise a plurality of first VCSELs mounted on a first platform. The flood illumination source may comprise a plurality of second VCSELs mounted on a second platform. The second platform may be beside the first platform. The projector may comprise a heat sink. Above the heat sink a first increment comprising the first platform may be attached. Above the heat sink a second increment comprising the second platform may be attached. The second increment may be different from the first increment. Thus, the first platform may be more distant to the optical element configured for increasing, e.g. duplicating, the number of spots. The second platform may be closer to the optical element. The beam emitted from the second VCSEL may be defocused and thus, form overlapping spots. This leads to a substantially continuous illumination and, thus, to flood illumination.

[0033] The term “display” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to an arbitrary shaped device configured for displaying an item of information. The item of information may be arbitrary information such as at least one image, at least one diagram, at least one histogram, at least one graphic, text, numbers, at least one sign, an operating menu, and the like. The display may be or may comprise at least one display panel. The display may have an arbitrary shape, e.g. a rectangular shape. The display may be a front display of the device. The display may comprise at least one of a display panel, particularly comprising a plurality of pixels and / or a plurality of transistors, or a glass, specifically a cover glass, particularly configured for covering the display panel.

[0034] The display, specifically the display panel, may be or may comprise at least one organic lightemitting diode (OLED) display and / or at least one quantum-dot light emitting diode (QLED). As used herein, the term “organic light emitting diode” is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a lightemitting diode (LED) in which an emissive electroluminescent layer is a film of organic compound configured for emitting light in response to an electric current. The OLED display may be configured for emitting visible light. As used herein, the term “organic light emitting diode” is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a display technology that utilizes semiconductor particles called quantum dots in order to produce colors on a display. These quantum dots may emit a plurality of different colors of light depending on their size when excited by light. By using a combination of red, green and / or blue quantum dots, a QLED display may display a wide range of colors with high brightness and color accuracy.

[0035] The display may be at least partially transparent. The display may be at least partially transparent in at least one continuous areas covering the projector, the flood illumination source and / or the image generation unit. The display may have a transmission below or equal to 20 %, preferably below or equal to 15 %, more preferably below or equal to 10 %. For example, an intensity of a light beam after being projected through the display may correspond to < 10 % of the intensity associated with the light beam when being emitted.

[0036] The display may be at least partially transparent in at least one continuous areas in a manner that at least one of:

[0037] - the light pattern incident on the continuous areas traverses the display while being illuminated from the projector;

[0038] - the flood light incident on the continuous areas traverses the display while being illuminated from the flood illumination source; user light, generated by the light pattern and / or the flood light incident on a user, incident on the continuous areas traverses the display for impinging on the image generation unit.

[0039] The term “at least partially transparent” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a property of the display to allow light, in particular of a certain wavelength range, e.g. in the infrared spectral region, in particular in the near infrared spectral region, to pass at least partially through. For example, the display may be semitransparent in the near infrared region. For example, the display may have a transparency of 20 % to 50 % in the near infrared region. The display may have a different transparency for differing wavelength ranges. The present invention may propose a device comprising the image generation unit and the projector that can be placed behind the display of a device. The transparent area(s) of the display can allow for operation of the image generation unit and the projector behind the display.

[0040] The display can be an at least partially transparent display, as described above. The partially transparent contiguous area of the display may be associated with a first pixel density value (Pixels per inch (PPI)), and a further area of the display may be associated with a second pixel density value. The first pixel density value may be lower than the second pixel density value. The transmission of light through the contiguous area may be higher compared to the transmission through the further area. The first pixel density value may be equal or below 450 PPI, preferably between 300 to 440 PPI, more preferably between 350 to 450 PPI. The first pixel density value may be constant over the entire contiguous area with a maximum deviation thereof of 20 %, or preferably 10 %. The second pixel density value may be between 400 to 500 PPI, preferably between 450 to 500 PPI.

[0041] The at least partially transparent continuous area of the display may comprise a first area and a second area. The first area may be associated with a first number of transistors configured for controlling at least one pixel and the second area may be associated with a second number of transistors configured for controlling at least one pixel, and wherein the first number of transistors may be smaller than the second number of transistors. The first number of transistors and / or the second number of transistors may refer to or be a density of the transistors. The term “pixel” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a picture unit, particularly the smallest picture unit, that represents an addressable element. The entirety of the pixels may represent the display. A pixel may be manipulated by changing its color, brightness and / or contrast or the like. Particularly for manipulating the pixel, the pixel may be driven by at least one transistor, exemplarily a transistor the controls a current required for driving the pixel. Typically, a thin-film transistor may be used for driving the pixel. TFTs may preferably be used in a flatpanel display.

[0042] The image generation unit is configured for generating a pattern image showing the projecting of the plurality of light beams onto the user. The term “image generation unit” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to at least one unit of the device configured for generating at least one image. The image may be generated via a hardware and / or a software interface, which may be considered as the image generation unit. The term “image generation”, or “imaging” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to capturing and / or generating and / or determining and / or recording at least one image by using the image generation unit. The image generation may comprise imaging and / or recording the image. The image generation may comprise capturing a single image and / or a plurality of images such as a sequence of images. For generating an image via a hardware and / or a software interface, the capturing and / or generating and / or determining and / or recording of the image may be caused and / or initiated by the hardware and / or the software interface. For example, the image generation may comprise recording continuously a sequence of images such as a video or a movie. The image generation may be initiated by a user action or may automatically be initiated, e.g. once the presence of at least one object or user within a field of view and / or within a predetermined sector of the field of view of the image generation unit is automatically detected. The term “field of view” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to an angular extent of the observable world and / or at least one scene that may be captured or viewed by an optical system, such as the image generation unit. The field of view may, typically, be expressed in degrees and / or radians, and, exemplarily, may represent the total angle spanned by the image and / or viewable area.

[0043] The image generation unit may comprise at least one optical sensor, in particular at least one pixelated optical sensor. The image generation unit may comprise at least one CMOS sensor or at least one CCD chip. For example, the image generation unit may comprise at least one CMOS sensor, which may be sensitive in the infrared spectral range. The term “image” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of or- dinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to data recorded by using the optical sensor, such as a plurality of electronic readings from the CMOS or CCD chip. The image may comprise raw image data or may be a pre-processed image. For example, the pre-processing may comprise applying at least one filter to the raw image data and / or at least one background correction and / or at least one background subtraction.

[0044] For example, the image generation unit may comprise one or more of at least one monochrome camera e.g. comprising monochrome pixels, at least one color (e.g. RGB) camera e.g. comprising color pixels, at least one IR camera. The camera may be a CMOS camera. The camera may comprise at least one monochrome camera chip, e.g. a CMOS chip. The camera may comprise at least one color camera chip, e.g. an RGB CMOS chip. The camera may comprise at least one IR camera chip, e.g. an IR CMOS chip. For example, the camera may comprise monochrome, e.g. black and white, pixels and color pixels. The color pixels and the monochrome pixels may be combined internally in the camera. The camera generally may comprise a one-dimensional or two-dimensional array of image sensors, such as pixels.

[0045] As outlined, above, the image generation unit may be at least one camera. For example, the camera may be an internal and / or external camera of the device. As described in the above, the internal and / or external camera of the device may be accessed via a hardware and / or a software interface, which is used as the image generation unit. In case, the device is or comprises a smartphone the image generating unit may be a front camera, such as a selfie camera, and / or back camera of the smartphone.

[0046] The image generation unit may have a field of view between 10°x10° and 75°x75°, preferably 55°x65°. The image generation unit may have a resolution below 2 MP, preferably between 0.3 MP and 1.5 MP.

[0047] The image generation unit may comprise further elements, such as one or more optical elements, e.g. one or more lenses. As an example, the optical sensor may be a fix-focus camera, having at least one lens which is fixedly adjusted with respect to the camera. Alternatively, however, the camera may also comprise one or more variable lenses which may be adjusted, automatically or manually. The camera may comprise at least one optical filter, e.g. at least one bandpass filter. The bandpass filter may be matched to the spectrum of the light emitters. Other cameras, however, are feasible.

[0048] The term “pattern image” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to an image generated by the image generation unit while illuminating with the light pattern, e.g. on an object and / or a user. The pattern image may comprise an image showing a user, in particular at least parts of the face of the user, while the user is being illuminated with the light pattern, particularly on a respective area of interest comprised by the image. The pattern image may be generated by imaging and / or recording light reflected by an object and / or user which is illuminated by the light pattern. The pattern image showing the user may comprise at least a portion of the illuminated light pattern on at least a portion the user. For example, the projection by the projector and the imaging by using the image generation unit may be synchronized, e.g. by using at least one control unit of the device.

[0049] As outlined above, the flood illumination source may be configured for emitting flood light by and the image generation unit may be configured for generating at least one flood image while the flood illumination source is emitting flood light. The term “flood image” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to an image generated by the image generation unit while illumination source is emitting infrared flood light, e.g. on an object and / or a user. The flood image may comprise an image showing a user, in particular the face of the user, while the user is being illuminated with the flood light. The flood image may be generated by imaging and / or recording light reflected by an object and / or user which is illuminated by the flood light. The flood image showing the user may comprise at least a portion of the flood light on at least a portion the user. For example, the illumination by the flood illumination source and the imaging by using the image generation unit may be synchronized, e.g. by using at least one control unit of the device.

[0050] The image generation unit may be configured for imaging and / or recording the pattern image and the flood image at the same time or at different times. The image generation unit may be configured for imaging and / or recording the pattern image and the flood image at at least partially overlapping measurement areas or equivalents of the measurement areas.

[0051] The device comprises the at least one processor configured for extracting liveness data from the pattern image and allowing the user to perform an operation on the device that requires authentication based on the liveness data. In particular, the device may be configured for authenticating a user of the device to perform at least one operation on the device that requires authentication. The authenticating may be performed by using the processor. The processor may be or may be a part of at least one authentication unit configured for performing at least one authentication process of a user. The authentication unit may be configured for allowing the user to perform an operation on the device that requires authentication based on the liveness data. Specifically, the authentication unit may be configured for using a facial recognition authentication process operating on the flood image, the pattern image and / or extracted liveness data, particularly derived from the pattern image.

[0052] The term “processor” as generally used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to an arbitrary logic circuitry configured for performing basic operations of a computer or system, and / or, gen- erally, to a device which is configured for performing calculations or logic operations. In particular, the processor may be configured for processing basic instructions that drive the computer or system. As an example, the processor may comprise at least one arithmetic logic unit (ALU), at least one floating-point unit (FPU), such as a math co-processor or a numeric co-processor, a plurality of registers, specifically registers configured for supplying operands to the ALU and storing results of operations, and a memory, such as an L1 and L2 cache memory. In particular, the processor may be a multi-core processor. Specifically, the processor may be or may comprise a central processing unit (CPU). Additionally or alternatively, the processor may be or may comprise a microprocessor, thus specifically the processor’s elements may be contained in one single integrated circuitry (IC) chip. Additionally or alternatively, the processor may be or may comprise one or more application-specific integrated circuits (ASICs) and / or one or more field- programmable gate arrays (FPGAs) and / or one or more tensor processing unit (TPU) and / or one or more chip, such as a dedicated machine learning optimized chip, or the like. The processor specifically may be configured, such as by software programming, for performing one or more operations. At least one or any component of a computer program configured for performing the authentication process may be executed by the processor. Alternatively or in addition, the authentication unit may be or may comprise a connection interface. The connection interface may be configured to transfer data from the device to a remote device; or vice versa. At least one or any component of a computer program configured for performing the authentication process may be executed by the remote device.

[0053] The term “authentication unit” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to at least one unit configured for performing at least one authentication process of a user. The authentication unit may be or may comprise the at least one processor. The authentication unit may be designed as software or application.

[0054] For example, the authentication unit may perform at least one face detection using the flood image. The face detection may be performed locally on the device. Face identification, i.e. assigning an identity to the detected face, however, may be performed remotely, e.g. in the cloud, e.g. especially when identification needs to be done and not only verification. User templates can be stored at the remote device, e.g. in the cloud, and would not need to be stored locally. This can be an advantage in view of storage space and security.

[0055] The authentication unit may be configured for identifying the user based on the flood image. Particularly therefore, the authentication unit may forward data to a remote device. Alternatively or in addition, the authentication unit may perform the identification of the user based on the flood image, particularly by running an appropriate computer program having a respective functionality. The term “identifying” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to assigning an identity to a detected face and / or at least one identity check and / or verifying an identity of the user. The authentication process may comprise a plurality of steps. For example, the authentication process may comprise performing at least one face detection. The face detection step may comprise analyzing the flood image. In addition, for example, the authentication process may comprise identifying. The identifying may comprise assigning an identity to a detected face and / or at least one identity check and / or verifying an identity of the user. The identifying may comprise performing a face verification of the imaged face to be the user’s face. The identifying the user may comprise matching the flood image, e.g. showing a contour of parts of the user, in particular parts of the user’s face, with a template. The identifying of the user may comprise determining if the imaged face is the face of the user, in particular if the imaged face corresponds to at least one image of the user’s face stored in at least one memory, e.g. of the device. Authentication may be unsuccessful if the flood image cannot be matched with an image template.

[0056] The authentication process may comprise analyzing of the flood image, e.g. by one or more of the following: a filtering; a selection of at least one region of interest; a formation of a difference image between the flood image and at least one offset; an inversion of flood image; a background correction; a decomposition into color channels; a decomposition into hue; saturation; and brightness channels; a frequency decomposition; a singular value decomposition; applying a Canny edge detector; applying a Laplacian of Gaussian filter; applying a Difference of Gaussian filter; applying a Sobel operator; applying a Laplace operator; applying a Scharr operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transformation; applying a Radon- transformation; applying a Hough-transformation; applying a wavelet-transformation; a thresholding; creating a binary image. The region of interest may be determined manually by a user or may be determined automatically, such as by recognizing the user within the image. In particular, the analyzing of the flood image may comprise using at least one image recognition technique, in particular a face recognition technique. An image recognition technique comprises at least one process of identifying the user in an image. The image recognition may comprise using at least one technique selected from the technique consisting of: color-based image recognition, e.g. using features such as template matching; segmentation and / or blob analysis e.g. using size, or shape; machine learning and / or deep learning e.g. using at least one convolutional neural network.

[0057] The analyzing of the flood image may comprise determining a plurality of facial features. The analyzing may comprise comparing, in particular matching, the determined facial features with template features. The template features may be features extracted from at least one template. The template may be or may comprise at least one image generated in an enrollment process, e.g. when initializing the device. Template may be an image of an authorized user. The template features and / or the facial feature may comprise a vector. Matching of the features may comprise determining a distance between the vectors. The identifying of the user may comprise comparing the distance of the vectors to a least one predefined limit, wherein the user is successfully identified in case the distance is < the predefined limit at least within tolerances. The user declining and / or rejected otherwise. For example, the image recognition may comprise using at least one model, in particular a trained model comprising at least one face recognition model. The analyzing of the flood image may be performed by using a face recognition system, such as FaceNet, e.g. as described in Florian Schroff, Dmitry Kalenichenko, James Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering”, arXiv: 1503.03832. The trained model may comprises at least one convolutional neural network. For example, the convolutional neural network may be designed as described in M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks”, CoRR, abs / 1311.2901 , 2013, or C. Szegedy et aL, “Going deeper with convolutions”, CoRR, abs / 1409.4842, 2014. For more details with respect to convolutional neural network for the face recognition system reference is made to Florian Schroff, Dmitry Kalenichenko, James Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering”, arXiv: 1503.03832. As training data labelled image data from an image database may be used. Specifically, labeled faces may be used from one or more of G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments”, Technical Report 07-49, University of Massachusetts, Amherst, October 2007, the Youtube® Faces Database as described in L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrained videos with matched background similarity”, in IEEE Conf, on CVPR, 2011 , or Google® Facial Expression Comparison dataset. The training of the convolutional neural network may be performed as described in Florian Schroff, Dmitry Kalenichenko, James Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering”, arXiv: 1503.03832.

[0058] As outlined above, the processor is configured for extracting liveness data from the pattern image. For extracting the liveness data, the authentication unit may forward data to a remote device. Alternatively or in addition, the authentication unit may perform the extraction of the liveness data based on the pattern image, particularly by running an appropriate computer program having a respective functionality. Particularly by considering the liveness data as a parameter for validating the authentication process, the authentication process may be robust against being outwitted by using a recorded image of the user.

[0059] The authentication unit may be configured for outsourcing at least one step of the authentication process, such as the identifying of the user, and / or at least one step of the validation of the authentication process, such as the consideration of the material data, to a remote device, specifically a server and / or a cloud server. The device and the remote device may be part of a computer network, particularly the internet. Thereby, the device may be used as a field device that is used by the user for generating data required in the authentication process and / or its validation. The device may transmit the generated data and / or data associated to an intermediate step of the authentication process and / or its validation to the remote device. In such a scenario, the authentication unit may be and / or may comprise a connection interface configured for transmitting information to the remote device. Data generated by the remote device used in the authentication process and / or its validation may further be transmitted to the device. This data may be re- ceived by the connection interface comprised by the device. The connection interface may specifically be configured for transmitting or exchanging information. In particular, the connection interface may provide a data transfer connection. As an example, the connection interface may be or may comprise at least one port comprising one or more of a network or internet port, a USB-port, and a disk drive.

[0060] It is emphasized that data from the device may be transmitted to a specific remote device depending on at least one circumstance, such as a date, a day, a load of the specific remote device, and so on. The specific remote device may not be selected by the field device. Rather a further device may select to which specific remote device the data may be transmitted. The authentication process and and / or the generation of validation data may involve a use of several different entities of the remote device. At least one entity may generate intermediate data and transmit the intermediate data to at least one further entity.

[0061] The term “liveness data” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to data allowing for distinguishing between a living human, in particular a user, and a non-living object such as a paper, 3D facial masks and the like. The liveness data may comprise blood perfusion data and / or material data. Extracting liveness data may comprises extracting material data and / or extracting blood perfusion data. The liveness data may comprise information about a material of the surface of the user on which the spots are projected. The liveness data may comprise information about at least one vital sign.

[0062] The extracting of liveness data, e.g. by using the authentication unit, may comprise extracting the material data from the pattern image by beam profile analysis of the light spots. With respect to beam profile analysis reference is made to WO 2018 / 091649 A1 , WO 2018 / 091638 A1 and WO 2018 / 091640 A1 , the full content of which is included by reference. Beam profile analysis can allow for providing a reliable classification of scenes based on a few light spots. Each of the light spots of the pattern image may comprise a beam profile. As used herein, the term “beam profile” may generally refer to at least one intensity distribution of the light spot on the optical sensor as a function of the pixel. The beam profile may be selected from the group consisting of a trapezoid beam profile; a triangle beam profile; a conical beam profile and a linear combination of Gaussian beam profiles.

[0063] Extracting material data from the pattern image may comprise generating the material type and / or data derived from the material type. Preferably, extracting material data may be based on the pattern image. Material data may be extracted by using at least one model. Extracting material data may comprise providing the pattern image to a model and / or receiving material data from the model. Providing the pattern image to a model may comprise and may be followed by receiving the pattern image at an input layer of the model or via a model loss function. The model may be a data-driven model. Data-driven model may comprise a convolutional neural network and / or an encoder decoder structure such as an autoencoder. Other examples for generating a representation may be FFT, wavelets, deep learning, like CNNs, energy models, normalizing flows, GANs, vision transformers, or transformers used for natural language processing, Autoregressive Image Modeling, Normalizing Flows, Deep Autoencoders, Deep Energy-Based Models. Supervised or unsupervised schemes may be applicable to generate a representation, also embedding in e.g. cosine or Euclidian metric in ML language. The data-driven model may be parametrized according to a training data set including at least one image and material data, preferably at least one pattern image and material data. In another embodiment, extracting material data may include providing the pattern image to a model and / or receiving material data from the model. In another embodiment, the data-driven model may be trained according to a training data set including at least one image and material data. In another embodiment, the data-driven model may be parametrized according to a training data set including at least one image and material data. The data-driven model may be parametrized according to a training data set to receive the image and provide material data based on the received image. The data-driven model may be trained according to a training data set to receive the image and provide material data as output based on the received image. The training data set may comprise at least one image and material data, preferably material data associated with the at least one image.

[0064] The image may be or may comprise a representation of the image. The representation may be a lower dimensional representation of the image. The representation may comprise at least a part of the data or the information associated with the image. The representation of an image may comprise a feature vector. In an embodiment, determining a representation, in particular a lower-dimensional representation may be based on principal component analysis (PCA) mapping or radial basis function (RBF) mapping. Determining a representation may also be referred to as generating a representation. Generating a representation based on PCA mapping may include clustering based on features in the pattern image and / or partial image. Additionally or alternatively, generating a representation may be based on neural network structures suitable for reducing dimensionality. Neural network structures suitable for reducing dimensionality may comprise encoder and / or decoder. In an example, neural network structure may be an autoencoder. In an example, neural network structure may comprise a convolutional neural network (CNN). The CNN may comprise at least one convolutional layer and / or at least one pooling layer. CNNs may reduce the dimensionality of a partial image and / or an image by applying a convolution, e.g. based on a convolutional layer, and / or by pooling. Applying a convolution may be suitable for selecting feature related to material information of the pattern image.

[0065] A model may be suitable for determining an output based on an input. In particular, model may be suitable for determining material data based on an image as input. A model may be a deterministic model, a data-driven model or a hybrid model. The deterministic model, preferably, reflects physical phenomena in mathematical form, e.g., including first-principles models. A deterministic model may comprise a set of equations that describe an interaction between the material and the patterned electromagnetic radiation thereby resulting in a condition measure, a vital sign measure or the like. A data-driven model may be a classification model. A hybrid model may be a classification model comprising at least one machine-learning architecture with deterministic or statistical adaptations and model parameters. Statistical or deterministic adaptations may be introduced to improve the quality of the results since those provide a systematic relation between empiricism and theory. In an embodiment, the data-driven model may be a classification model. The classification model may comprise at least one machine-learning architecture and model parameters. For example, the machine-learning architecture may be or may comprise one or more of: linear regression, logistic regression, random forest, piecewise linear, nonlinear classifiers, support vector machines, naive Bayes classifications, nearest neighbors, neural networks, convolutional neural networks, generative adversarial networks, support vector machines, or gradient boosting algorithms or the like. In the case of a neural network, the model can be a multi-scale neural network or a recurrent neural network (RNN) such as, but not limited to, a gated recurrent unit (GRU) recurrent neural network or a long short-term memory (LSTM) recurrent neural network. The data-driven model may be parametrized according to a training data set. The data-driven model may be trained based on the training data set. Training the model may include parametrizing the model. The term training may also be denoted as learning. The term specifically may refer, without limitation, to a process of building the classification model, in particular determining and / or updating parameters of the classification model. Updating parameters of the classification model may also be referred to as retraining. Retraining may be included when referring to training herein. In an embodiment, the training data set may include at least one image and material information.

[0066] Extracting material data from the image with a data-driven model may comprise providing the image to a data-driven model. Additionally or alternatively, extracting material data from the image with a data-driven model may comprise may comprise generating an embedding associated with the image based on the data-driven model. An embedding may refer to a lower dimensional representation associated with the image such as a feature vector. Feature vector may be suitable for suppressing the background while maintaining the material signature indicating the material data. In this context, background may refer to information independent of the material signature and / or the material data. Further, background may refer to information related to biometric features such as facial features. Material data may be determined with the data-driven model based on the embedding associated with the image. Additionally or alternatively, extracting material data from the image by providing the image to a data-driven model may comprise transforming the image into material data, in particular a material feature vector indicating the material data. Hence, material data may comprise further the material feature vector and / or material feature vector may be used for determining material data.

[0067] The authentication process may be validated based on the extracted material data. In an embodiment, the validating based on the extracted material data may comprise determining if the extracted material data corresponds a desired material data. Determining if extracted material data matches the desired material data may be referred to as validating. Allowing or declining the user and / or object to perform at least one operation on the device that requires authentication based on the material data may comprise validating the authentication or authentication process. Validating may be based on material data and / or image. Determining if the extracted material data corresponds a desired material data may comprise determining a similarity of the extracted material data and the desired material data. Determining a similarity of the extracted material data and the desired material data may comprise comparing the extracted material data with the desired material data. Desired material data may refer to predetermined material data. In an example, desired material data may be skin. It may be determined if material data may correspond to the desired material data. In the example, material data may be non-skin material or silicon. Determining if material data corresponds to a desired material data may comprise comparing material data with desired material data. A comparison of material data with desired material data may result in a allowing and / or declining the user and / or object to perform at least one operation that requires authentication. In the example, skin as desired material data may be compared with non-skin material or silicon as material data and the result may be declination since silicon or non-skin material may be different from skin.

[0068] The authentication process or its validation may include generating at least one feature vector from the material data and matching the material feature vector with associate reference template vector for material.

[0069] Additionally or alternatively of using material data, the extracting of liveness data may comprise extracting blood perfusion data. For example, the light beams projected by the projector may be coherent patterned infrared illumination. Extracting blood perfusion data may comprise determining a speckle contrast of the pattern image and determining a blood perfusion measure based on the determined speckle contrast. A speckle contrast may represents a measure for a mean contrast of an intensity distribution within an area of a speckle pattern. In particular, a speckle contrast K over an area of the speckle pattern may be expressed as a ratio of standard deviation o to the mean speckle intensity <l>, i.e. ,

[0070] Speckle contrast may comprise a speckle contrast value. Speckle contrast values may be distributed between 0 and 1. The blood perfusion measure may be determined based on the speckle contrast.

[0071] The blood perfusion measure may depend on the determined speckle contrast. If the speckle contrast changes, the blood perfusion measure derived from the speckle contrast may change accordingly. A blood perfusion measure may be a single number or value that may represent a likelihood that the object is a living subject.

[0072] For example, for determining the speckle contrast, the complete pattern image may be used. Alternatively, for determining the speckle contrast, a section of the pattern image may be used. The section of the pattern image, preferably, represents a smaller area of the pattern image than an area of the complete pattern image. The section of the pattern image may be obtained by cropping the pattern image. In an embodiment, a data-driven model may be used for determining a blood perfusion measure. Data-driven model be parametrized and / or trained based on a training data set. The training data set may comprise a pattern image and a blood perfusion measure. The data-driven model may be parametrized and / or trained based on the training data set to output a blood perfusion measure based on receiving a pattern image.

[0073] The authentication process may be validated based on the blood perfusion measure. In an embodiment, the validating based on the blood perfusion measure may comprise determining if the blood perfusion measure corresponds to blood perfusion measure of a human being. Determining if the blood perfusion measure corresponds a human being may be referred to as validating. Allowing or declining the user and / or object to perform at least one operation on the device that requires authentication based on the blood perfusion measure may comprise validating the authentication or authentication process. Validating may be based on the blood perfusion measure. Determining if the blood perfusion measure corresponds a human being may comprise comparing the blood perfusion measure to at least one pre-defined or pre-determined range of values of blood perfusion measure, e.g. stored in at least one database. In case the extracted blood perfusion measure is at least within tolerances within the re-defined or pre-determined range of values of blood perfusion measure, the authentication is validated, otherwise not. In case the authentication is validated the method may comprise allowing the user to perform at least one operation that requires authentication. Otherwise, in case the authentication is not validated, the authentication unit may decline the user to perform at least one operation that requires authentication.

[0074] The allowing and / or declining the user to perform an operation on the device that requires authentication based on the liveness data may be performed by using at least one authorization unit. In particular, the authorization unit may be configured for performing at least authorization step, e.g. by using at least one authorization unit. The term “authorization step” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a step of assigning access rights to the user, in particular a selective permission or selective restriction of access to the device and / or at least one resource of the device. The authorization unit may be configured for access control. The term “authorization unit” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a unit such as a processor configured for authorization of a user. The authorization unit may comprise at least one processor or may be designed as software or application. The authorization unit and the authentication unit may be embodied integral, e.g. by using the same processor. The authorization unit may be configured for allowing the user to perform at least one operation on the device, e.g. unlocking the device, in case of successful authentication of the user or declining the user to perform at least one operation on the device in case of non-successful authentication. Thereby, the user may become aware of the result of the authentication. The method may comprise displaying the result of the authentication on the display.

[0075] At least one operation on the device that requires authentication may be access to the device, e.g. unlocking the device, and / or access to an application, preferably associated with the device and / or access to a part of an application, preferably associated with the device. In an embodiment, allowing the user to access a resource may include allowing the user to perform at least one operation with a device and / or system. The resource may be a device, a system, a function of a device, a function of a system and / or an entity. Additionally and / or alternatively, allowing the user to access a resource may include allowing the user to access an entity. The entity may be a physical entity and / or a virtual entity. The virtual entity may be a database for example. The physical entity may be an area with restricted access. The area with restricted access may be one of the following: security areas, rooms, apartments, vehicles, parts of the before mentioned examples, or the like. Device and / or system may be locked. The device and / or the system may only be unlocked by an authorized user.

[0076] The device comprises the at least one control unit and the at least one status inquiry device configured for retrieving at least one item of status information on a current environmental status of the device. The control unit is configured for automatically triggering at least one calibration depending on the fulfillment of at least one predetermined environmental status condition

[0077] The term “control unit” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a device or combination of devices capable and / or configured for performing at least one computing operation and / or for controlling at least one function of at least one other device, such as of at least one other component of the device for authenticating. Specifically, the at least one control unit may be embodied as at least one processor and / or may comprise at least one processor, wherein the processor may be configured, specifically by software programming, for performing one or more operations.

[0078] The term “status inquiry device” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a device or a combination of devices capable of our configured for retrieving the at least one item of status information as outlined above. Specifically, the status inquiry device may comprise at least one of an interface for retrieving the item of status information in an electronic format, such as a data format, such as a wireless or wire bound interface, and / or at least one device configured for generating the item of status information, such as at least one sensor device, as will be outlined in further detail below. The status inquiry device may also fully or partially be integrated into the control unit. Additionally or alternatively, the status inquiry device may comprise one or more devices integrated into the device for authenticating, such as one or more integrated sensors. The term “retrieving” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a process of providing at least one object, or item to at least one entity. The process may comprise generating the at least one item or object or obtaining the at least one item or object from another source and providing the at least one item or object to the at least one entity. Thus, referring to the retrieving of the at least one item of status information on a current environmental status of the device for authenticating, the retrieving may, as an example, comprise obtaining the item of status information from at least one source, such as from at least one external source, e.g. via at least one interface, such as a wireless and / or a wire bond interface, the external source, as an example, may be the Internet. Additionally or alternatively, for retrieving the at least one item of status information, the device may also comprise one or more sensors or sensing devices for measuring at least one measurable value from which the at least one item of status information on the current environmental status of the device may be derived, directly or indirectly.

[0079] The term “current environmental status of the device” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a situation and / or condition in which the device for authenticating momentarily is in, i.e. at the moment of inquiry and / or within a predetermined time span around the moment of inquiry. Therein, the term “environmental status” may relate to at least one condition of an environment, specifically a surrounding, in which the device for authenticating is in and / or to at least one relationship between the device and the surrounding, such as a location and / or an orientation.

[0080] As an example, the item of status information on a current environmental status refers to one or more of: an environmental lighting condition such a level of ambient light and / or spatial location and / or orientation with respect to an external light source; a weather condition, an item of temperature information representing a current temperature; an item of position information of the device; an item of orientation information of the device; a relative orientation and / or a relative location between the device and at least one item in the environment; an item of handling information representing a current mode of handling of the device by a user; at least one item of information available via at least one network; a current environmental status approximated from the analysis of previously recorded environmental status information. For example, the current environmental status may be approximated from the analysis of previously recorded environmental status information. For determining the current environmental status measurements from the past may be used and / or considered. Actual measurements and measurements from the past can be combined.

[0081] The control unit, as outlined above, is configured for automatically triggering at least one calibration, depending on the fulfillment of at least one predetermined environmental status condition. The term “automatically triggering” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to the process of starting an action without the necessity of human interaction, specifically without any human interaction or with only optional human interaction.

[0082] Subsequent to retrieving the item of status information on the current environmental status of the device, the control unit automatically triggers the calibration. The calibration of the device may comprise calibrating at least one hardware component of the device having an influence on the operation performance, in particular the authentication. The calibration may comprise calibrating one or more of a position of the projector, the image generation unit, calibrating a reference pattern, other hardware used for authentication, in particular having or suspected to have a temperature dependency.

[0083] The calibration may be automatically trigged by the control unit. The calibration may be automatically triggered by the control unit in case the predetermined environmental status condition is fulfilled by the retrieved item of status information on a current environmental status.

[0084] The term “predetermined environmental status condition” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a condition of the environment the device for authenticating is in and / or a condition of a relationship between the device for authenticating and the environment which has been predetermined to be sufficient for performing the calibration As an example, the at least one environmental status condition may be at least one condition that must be fulfilled by the at least one item of status information on the current environmental status of the device for authenticating. As an example, the item of status information may be compared with at least one maximum or minimum threshold value, and the condition may be fulfilled and the calibration may, as an example, be automatically triggered when the item of status information is above or below the minimum or maximum threshold value, respectively. Additionally or alternatively, the environmental status condition may be fulfilled when the at least one item of status information on the current environmental status of the device for authenticating and / or at least one secondary value derived from the at least one item of status information by using a predetermined relationship or function are within at least one predetermined range. Thus, as an example, the control unit may obtain the at least one item of status information on a current environmental status from the status inquiry device, may optionally transform the at least one item of status information and to at least one secondary value and may then check whether the at least one item of status information and / or the at least one secondary value fulfills the environmental status condition, such as has a predetermined target value or is within a predetermined range. If this environmental status condition is fulfilled, the control unit may automatically trigger, with or without delay, the at least one calibration.

[0085] The predetermined environmental status condition may comprise at least one condition selected from the group consisting of: a level of ambient light is within a predetermined suitable level range for performing a calibration measurement; the device is in a suitable location for performing a calibration measurement; the device is in a suitable orientation for performing a calibration measurement; the device is not faced towards an external light source; the device is not in proximity to an object or pointed to an object; the device is not located within a pocket; the device is at a temperature within a predetermined temperature range suited for a calibration measurement; a temporal temperature change is within a predetermined range suited for a calibration measurement; weather conditions are within a predetermined range suited for a calibration measurement; the device presently is not used for another function; the device is available via at least one network.

[0086] The use of the at least one item of status information on a current environmental status of the device for authenticating may, thus, enable the device for authenticating to perform calibrations repeatedly, when the predetermined environmental status condition indicates that the environmental conditions are suited for the at least one calibration. The at least one item of status information specifically may be retrieved by using integrated means of the device for authenticating.

[0087] The device may be configured for using at least one integrated sensor device of the device as at least a part of the at least one status inquiry device. Thus, one or more integrated sensor devices may be used which are present in the device anyway, such as for one or more other purposes.

[0088] Generally, and specifically also in the case of the use of a mobile communication device, the status inquiry device may comprise at least one device selected from the group consisting of: a front camera being positioned on the same side as the display; a rear camera being positioned on the opposing side as the display; a location sensor; an illumination sensor configured for determining at least one state of illumination in an environment of the device; a temperature sensor; a motion sensor; a gyroscopic sensor; a magnetic sensor; a material sensor configured for determining at least one material property of at least one object in proximity of the device; a spectrometer device configured for acquiring at least one item of spectral information; at least one software sensor configured for generating information about the status of the device by processing input from a plurality of physical sensors. The software sensor may be configured for determining the status of the device by indirect means or learning patterns. This can also include user behavior, time of day or other correlatable pieces of information. Many or even all of these devices, specifically sensor devices, are typically integrated in mobile communication devices such as smart phones. Each of said devices is generally configured for providing at least one item of status information on a current environmental status of the device. Specifically, said devices or one or more or even all of said devices may provide information which may be used for determining whether the environmental status is such that the calibration may be triggered. Signals from one or more of the devices may be used in isolation or in combination to determine whether the conditions for the calibration are given. The status inquiry device may be configured for determining if the device is placed on a table in a room by using at least one gyroscopic sensor, and / or wherein the status inquiry device is configured for determining an ambient light level by determining at least one dark image without the projector projecting the plurality of light beams, and / or wherein the status inquiry device is configured for determining if a ceiling is visible by capturing at least one image with the projector projecting the plurality of light beams, and / or wherein the status inquiry device is configured for determining presence of an obstacle in proximity of the device by using at least one proximity sensor.

[0089] For example, the device may try to detect if it is on a table in a room. A gyroscopic sensor may support to detect if the device, e.g. a smartphone, lies flat on a table. For example, the device may capture a frame of images without any flood light and light from the projector in order to determine if it is dark in the room, i.e. less sunlight. For example, proximity sensors can check that there lies no obstacle on the device, e.g. a smartphone. For example, the device may capture a frame of images with projector on. If there is a ceiling this would be visible on the laser frame. The device may detect the light pattern on that frame for calibration.

[0090] The term “calibration” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a process of determining a deviation of an actual performance of at least one hardware component of the device having an influence on the authentication, e.g. the image generation unit, from a target performance of said hardware component. The calibration may further comprise determining a correction of the determined deviation. For example, the calibration comprises one or more of calibrating a reference pattern, calibrating a position of the projector, at least one temperature calibration, or at least one diffraction calibration.

[0091] The calibration comprises at least one user guided calibration and / or an automated calibration. The term “user-guided calibration” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a calibration comprising in at least one step at least one user action and / or interaction. The term “automated calibration” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a completely automatically performed calibration such as without any user action or user interaction.

[0092] The calibration may comprise a process as described in WO 2022 / 253777, the content of which is included by reference.

[0093] The calibration may comprise calibrating a reference pattern and / or, in particular subsequently, calibrating a position of the projector. The calibration may comprise calibrating a reference pattern by performing the following steps: i) capturing at least one calibration image at a relative distance between the device and at least one reference object by using the image generation unit while the projector projecting the plurality of light beams; ii) determining a reflection pattern by identifying light spots on the captured calibration image generated by the reference object in response to illumination by the plurality of light beams; iii) matching the light spots of the reflection pattern to features of a reference pattern considering an estimate of the different relative distances between the device and the reference object while capturing the calibration image, thereby determining pairs of matched reflection features; iv) recalculating the reference pattern from image coordinates of the corresponding matched light spots.

[0094] In step i) a plurality of calibration images may be captured at different relative distances, e.g. at least one calibration image at each of the relative distances. The determining of the calibration images may comprise imaging at least one two-dimensional images of a white surface for different relative distances by using the image generation unit with the projector projecting the plurality of light beams. For example, calibration may comprise identification of at least one flat, white surface (e.g. wall), e.g. by using a selfie camera while the smartphone is being carried around. For example, the user is requested to capture at least two calibration images of a wall with a coarse distance.

[0095] At least one evaluation device, such as the processor, may be configured for performing an image analysis and for identifying light spots of the calibration image.

[0096] The term “calibration image” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to an image used for calibration. The reference object may be an arbitrary object such a wall. The calibration image may be a far field image.

[0097] The calibration may be performed using triangulation. The calibration may use information about a reference pattern and a relative position of the projector. The reference pattern may be a pattern of the projector at a hypothetic distance of infinity observed from the image generation unit. The reflection pattern may be captured by the image generation unit at a known distance zO. If the reflection pattern is known for different distance, e.g. 20 cm, 1 m, 2 m or infinity then the reference pattern can be recalculated.

[0098] Many applications relate to a near field (e.g. 0.15 m to 0.6 m). A coarse estimate of the distance outside the near field (> 1.5m) can nevertheless provide a good approximation for the near field. This is because the triangulation error is quadratic with distance. For example, a target with a distance of 2 m is estimated to a coarse distance of 1 .5 m. The error for the distance may be 0.5 m. The triangulation error for 0.2m may be simply given by the quadratic error propagation, i.e. 0.5 m / (2 m*2 m) * (0.2m*0.2m) = 5mm.

[0099] The evaluation device may be configured for selecting the light spots and matching them with corresponding features of the reference pattern. As used herein, the term “matching” refers to determining and / or evaluating corresponding features of the reference pattern and light spots of the calibration image. The matching may be performed as follows. The evaluation device may be configured for identifying at least one feature in the reference pattern having an essentially identical longitudinal coordinate as the selected light spot. The term “essentially identical” refers to identical within 10%, preferably 5%, most preferably 1 %. The feature of the reference pattern corresponding to the light spot may be determined using epipolar geometry. For description of epipolar geometry reference is made, for example, to chapter 2 in X. Jiang, H. Bunke: „Dreidi- mensionales Computersehen", Springer, Berlin Heidelberg, 1997. Epipolar geometry may assume that the reference pattern and the calibration image may be images determined at different spatial positions and / or spatial orientations having a fixed distance. The evaluation device may be configured for matching respectively one of the light spots of the calibration image with respectively one of the reference features within a displacement region by using at least one linear scaling algorithm considering a distance information.

[0100] In particular, step iv) may comprise considering an estimated relative distance of the reference object for recalculating the reference pattern. The distance estimate may be assumed or a distance sensor may be used. The distance information may be an estimate of the different relative distances between the device and the reference object while capturing the calibration image.

[0101] The evaluation device may be configured for determining an epipolar line in the reference pattern. The evaluation device may be configured for determining a straight line extending from a feature of the reference pattern. The straight line may comprise possible features corresponding to the selected light spot. The straight line and the baseline span an epipolar plane. As the reference pattern is determined at a different relative position from the calibration image, the corresponding possible features may be imaged on a straight line, called epipolar line, in the reference pattern. Thus, a feature of the reference pattern corresponding to the selected light spot is assumed to lie on the epipolar line. As outlined above, the evaluation device, may be configured for pre-classifying the selected light spots using information about distance, in particular an estimate of the different relative distances between the device and the reference object. This can allow an unambiguous assignment to one feature of the reference pattern. Moreover, in particular, features of the projected pattern may be arranged such that corresponding features of the reference pattern have a relative distance to each other as large as possible on the epipolar line. The features of the projected pattern may be arranged such that only few features of the reference pattern are positioned on the epipolar line. After matching the light spots and the features of the reference pattern, the reference pattern is recalculated. The recalculation may be performed by using image coordinates of the corresponding matched light spots. The image coordinates of light spots may be mapped to projector coordinates. This yields the calibrated reference pattern.

[0102] For example, the calibration of the reference pattern may be performed as follows. In step i), the reflection pattern may be captured on a far distance (e.g. 1 .5 m - 3 m) and the light spots may be identified. The calibration may comprise checking if the pattern is complete. Next, in step iii), the reflection pattern may be matched with the reference pattern. As outlined above, step iv) may comprise considering an estimated relative distance of the reference object for recalculating the reference pattern. For example, the projected target may be assumed to be at a nominal distance or an estimate of the distance, e.g. by the illumination of the light spots, may be used. In step iv), the image coordinates of light spots are mapped to projector coordinates. This yields the reference pattern.

[0103] The calibration may comprise calibrating a position of the projector. The calibrating of the position of the projector may be performed after calibrating the reference pattern.

[0104] The calibrating of the position of the projector may comprise the following steps:

[0105] I) capturing at least one calibration face image at a relative distance between the device and a user’s face by using the image generation unit while the projector projecting the plurality of light beams;

[0106] II) estimating the relative distance by analyzing the calibration face image, wherein the analyzing comprises extracting landmarks of the face by using a two-dimensional face detection;

[0107] III) identifying light spots on the user’s face on the calibration face image and matching the identified light spots to features of a reference pattern considering the estimated relative distance, thereby determining pairs of matched reflection features;

[0108] IV) determining a translation vector describing the position of the projector by determining an epipolar line distance for each of the pairs of the matched reflection features.

[0109] The term “calibration face image” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to an image of at least a part of the user’s face used for calibration. The determining of the calibration face image may comprise imaging at least one two-dimensional image of the user’s face with a relative distance between the projector and the user’s face with the projector projecting the plurality of light beams by using the image generation unit. The calibration face image may be captured e.g. during enrollment or during an unlock. For example, the 2D image generated while a selfie camera can be used. The calibration face image may be analyzed by extracting landmarks of a face by a 2d face detection. The extraction of landmarks may be performed as described in e.g. en.wikipe- dia.org / wiki / Landmark_detection or as described in OpenCV: Face landmark detection in an image. This may allow for an estimation of the face distance. Additionally a distance sensor may be used.

[0110] The light pattern of the calibration face image may be analyzed. For example, in step III) only light spots on the face may be used. The light spots may be matched to the reference pattern, the matching can be performed as described above. This is possible since the face distance is known from using the landmarks.

[0111] After matching the light spots to the reference pattern, the epipolar line distance d can be determined. The calibration may comprise determining an epipolar line of the matched feature of the reference pattern for each of the pairs of matched light spot and feature of the reference pattern. In particular, the epipolar line used for matching may be used as epipolar line of said pair. As used herein, the term “epipolar line distance” may refer to a distance of the feature of the reference pattern and the epipolar line used for matching, denoted as corresponding epipolar line. The distance may be determined by determining image coordinates of the calibration face image and of the corresponding epipolar line and comparing the image coordinates. A minimum distance to the corresponding epipolar line may be used as epipolar line distance. In case of a good extrinsic calibration the epipolar line distance is close to zero.

[0112] The epipolar line distance may be defined as a function d(x,y) on the position (x,y) of the reference pattern. The epipolar line distance function d can be analyzed to compute the correction for rotation and / or translation. In case of a decalibrated system, the function d may generate a geometric pattern. The shape of the geometric pattern may show uniquely the degree of decalibration. Geometric pattern may be or may comprise one or more of repetitions, steepness, discontinuities, and curvatures in the function d(x,y) can be used for calibration. If the rotation and / or translation of the projector and / or image generation unit changes, this result can be observed in the function d as geometric pattern. The evaluation device may be configured for performing an algorithm designed for analyzing d(x,y) and for computing a correction for the rotation and / or translation. The evaluation device may be configured for determining the correction of the reflection image by evaluating one or more of shape, repetitions, steepness, discontinuities, and curvatures of the geometric pattern.

[0113] This can yield the translation vector that locate the projector position. The length of the translation vector may be the baseline length. This value may be already known by the hardware design. With this step, the calibration is completed. The reference pattern and the projector position (rotation and / or translation) are calibrated.

[0114] Additionally or alternatively, the calibration may comprise obtaining reference data at different temperatures of the hardware used for authentication, such as of the projector and / or image generation unit. This may allow correcting for temperature related drift. Thus, the calibration may comprise at least one temperature calibration. For example, the temperature calibration comprises the following steps: a) capturing at least one temperature calibration image of at least one reference object at least a plurality of temperatures while the projector projecting the plurality of light beams; b) identifying light spots on the temperature calibration images and matching the identified light spots to features of a reference pattern considering an estimated relative distance between the reference object and the projector, thereby determining pairs of matched reflection features for each temperature; c) determining a temperature dependent correction of the reference pattern by determining deviations between the position of the matched reflection features for each temperature, and storing the temperature dependent correction of the reference pattern in at least one database.

[0115] Additionally or alternatively, the calibration comprises at least one diffraction calibration. For example, a selfie camera (which might be located behind a punchhole, which means not suffering from diffraction effects in the receiving path) may be used to get diffraction pattern correction data. The projection and imaging through the display may have diffraction effects for the TX and the RX. By providing a selfie camera image of the projected pattern on a surface and comparing the RX image and the selfie camera image, the influence of RX diffraction and TX diffraction can be sorted out. This may support simplification of underlying algorithms and / or more robust as well as giving the potential to correct for changes during the lifetime of the product.

[0116] For example, the diffraction calibration comprises the following steps:

[0117] A) capturing at least one diffraction calibration image of at least one reference object while the projector projecting the plurality of light beams by using a camera behind a punchhole;

[0118] B) identifying light spots on the diffraction calibration image and matching the identified light spots to features of a reference pattern considering an estimated relative distance between the reference object and the projector, thereby determining pairs of matched reflection features;

[0119] C) determining a diffraction correction of the reference pattern by determining deviations between the position of the matched light spots and features, and storing the diffraction correction of the reference pattern in at least one database.

[0120] The control unit may configured for repeatedly automatically triggering the calibration, in particular a calibration measurement. The control unit may be configured for performing status checks at predetermined points in time. For example, the status checks each comprise receiving the at least one item of status information on the current environmental status and checking whether the at least one predetermined environmental status condition is fulfilled. The control unit further may be configured for triggering the calibration measurement depending on whether the at least one environmental status condition is fulfilled. The term “fulfillment” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to at least one condition which must be met. As an example, the at least one environmental status condition may be at least one condition that must be fulfilled by the at least one item of status information on the current environmental status. As an example, the item of status information may be compared with at least one maximum or minimum threshold value, and the condition may be fulfilled and the calibration, as an example, be automatically triggered when the item of status information is above or below the minimum or maximum threshold value, respectively. Additionally or alternatively, the environmental status condition may be fulfilled when the at least one item of status information on the current environmental status and / or at least one secondary value derived from the at least one item of status information by using a predetermined relationship or function are within at least one predetermined range, fulfills the environmental status condition, e.g. is above or below the minimum or maximum threshold value, respectively. As an example, the control unit may obtain the at least one item of status information on a current environmental status from the status inquiry device, may optionally transform the at least one item of status information and to at least one secondary value and may then check whether the at least one item of status information and / or the at least one secondary value fulfills the environmental status condition, such as has a predetermined target value or is within a predetermined range. If this environmental status condition is fulfilled, the control unit may automatically trigger, with or without delay, the at least one reference measurement, such as by triggering the device to capture images and / or for issuing indications for a user to capture images.

[0121] Summarizing, surprisingly it was found, that automatically determining the right timing for performing a calibration can be advantageously used for calibration of hardware for face authentication built into a smartphone. This can allow reducing costs for a factory calibration. Moreover, calibration for a 3D imager behind OLED can be done in a complete assembled smartphone.

[0122] In a further aspect of the present invention, a method of performing at least one reference measurement with a device according to the present invention such as according to any one of the embodiments described above and / or according to any one of the embodiments described in further detail below, is disclosed.

[0123] The method comprising: i. retrieving at least one item of status information on a current environmental status of the device;

[0124] II. checking if at least one environmental status condition for performing a calibration, in particular at least one calibration measurement, is fulfilled; and ill. automatically triggering the calibration, in particular the at least one calibration measurement, if, in step ii. , the environmental status condition is fulfilled.

[0125] The method steps may be performed in the given order or may be performed in a different order. Further, one or more additional method steps may be present which are not listed. Further, one, more than one or even all of the method steps may be performed repeatedly. Thus, specifically, method steps i.-iii. may be performed repeatedly, such as in the given order. Moreover as an example and as also outlined above, the repetitions may take place in regular time intervals and / or at determinable or predetermined points in time. The predetermined points in time, as an example, may comprise points of time when the device is started or may comprise points in time when an app on the device, such as a mobile communication device like e.g. a smart phone, is started.

[0126] The method may comprise projecting a plurality of light beams through at least one display onto the user by using at least one projector; generating a pattern image showing the projecting of the plurality of light beams onto the user by using at least one image generation unit; extracting liveness data from the pattern image and allowing the user to perform an operation on the device that requires authentication based on the liveness data by using at least one processor; retrieving at least one item of status information on a current environmental status of the device by using at least one control unit and at least one status inquiry device, and automatically triggering at least one calibration depending on the fulfillment of at least one predetermined environmental status condition.

[0127] With respect to definitions and embodiments reference is made to definitions and embodiments as described with respect to the device above, or as described in more detail below.

[0128] The method may be computer-implemented. The term "computer implemented " as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a method involving at least one computer and / or at least one computer network. The computer and / or computer network may comprise at least one processor which is configured for performing at least one of the method steps of the method according to the present invention. Specifically, each of the method steps is performed by the computer and / or computer network. The method may be performed completely automatically, specifically without user interaction.

[0129] All described method steps may be performed by using the device. Therefore, a single processing device may be configured to exclusively perform at least one computer program, in particular at least one line of computer program code configured to execute at least one algorithm, as used in at least one of the embodiments of the method according to the present invention. Herein, the computer program as executed on the single processing device may comprise all instructions causing the computer to carry out the described method. Alternatively, or in addition, at least one method step may be performed by using at least one remote device, especially selected from at least one of a server or a cloud server, particularly when the device and the remote device may be part of a computer network. In this case, the computer program may comprise at least one remote component to be executed by the at least one remote processing device to carry out the at least one method step. The remote component may have the functionality of performing the identifying of the user and / or the extraction of the material data. Further, the computer program may comprise at least one interface configured to forward to and / or receive data from the at least one remote component of the computer program.

[0130] Further disclosed and proposed herein is a computer program including computer-executable instructions for performing the method according to the present invention in one or more of the embodiments enclosed herein when the program is executed on a computer or computer network. Specifically, the computer program may be stored on a computer-readable data carrier and / or on a computer-readable storage medium.

[0131] As used herein, the terms “computer-readable data carrier” and “computer-readable storage medium” specifically may refer to non-transitory data storage means, such as a hardware storage medium having stored thereon computer-executable instructions. The computer-readable data carrier or storage medium specifically may be or may comprise a storage medium such as a random-access memory (RAM) and / or a read-only memory (ROM).

[0132] Thus, specifically, one, more than one or even all of method steps i. to ill. as indicated above may be performed by using a computer or a computer network, preferably by using a computer program.

[0133] Further disclosed and proposed herein is a computer program product having program code means, in order to perform the method according to the present invention in one or more of the embodiments enclosed herein when the program is executed on a computer or computer network. Specifically, the program code means may be stored on a computer-readable data carrier and / or on a computer-readable storage medium.

[0134] Further disclosed and proposed herein is a data carrier having a data structure stored thereon, which, after loading into a computer or computer network, such as into a working memory or main memory of the computer or computer network, may execute the method according to one or more of the embodiments disclosed herein.

[0135] Further disclosed and proposed herein is a computer program product with program code means stored on a machine-readable carrier, in order to perform the method according to one or more of the embodiments disclosed herein, when the program is executed on a computer or computer network. As used herein, a computer program product refers to the program as a tradable product. The product may generally exist in an arbitrary format, such as in a paper format, or on a computer-readable data carrier and / or on a computer-readable storage medium. Specifically, the computer program product may be distributed over a data network. Finally, disclosed and proposed herein is a modulated data signal which contains instructions readable by a computer system or computer network, for performing the method according to one or more of the embodiments disclosed herein.

[0136] Referring to the computer-implemented aspects of the invention, one or more of the method steps or even all of the method steps of the method according to one or more of the embodiments disclosed herein may be performed by using a computer or computer network. Thus, generally, any of the method steps including provision and / or manipulation of data may be performed by using a computer or computer network. Generally, these method steps may include any of the method steps, typically except for method steps requiring manual work, such as providing the samples and / or certain aspects of performing the actual measurements.

[0137] Specifically, further disclosed herein are: a computer or computer network comprising at least one processor, wherein the processor is adapted to perform the method according to one of the embodiments described in this description, a computer loadable data structure that is adapted to perform the method according to one of the embodiments described in this description while the data structure is being executed on a computer, a computer program, wherein the computer program is adapted to perform the method according to one of the embodiments described in this description while the program is being executed on a computer, a computer program comprising program means for performing the method according to one of the embodiments described in this description while the computer program is being executed on a computer or on a computer network, a computer program comprising program means according to the preceding embodiment, wherein the program means are stored on a storage medium readable to a computer, a storage medium, wherein a data structure is stored on the storage medium and wherein the data structure is adapted to perform the method according to one of the embodiments described in this description after having been loaded into a main and / or working storage of a computer or of a computer network, and a computer program product having program code means, wherein the program code means can be stored or are stored on a storage medium, for performing the method according to one of the embodiments described in this description, if the program code means are executed on a computer or on a computer network.

[0138] As used herein, the terms “have”, “comprise” or “include” or any arbitrary grammatical variations thereof are used in a non-exclusive way. Thus, these terms may both refer to a situation in which, besides the feature introduced by these terms, no further features are present in the entity described in this context and to a situation in which one or more further features are present. As an example, the expressions “A has B”, “A comprises B” and “A includes B” may both refer to a situation in which, besides B, no other element is present in A (i.e. a situation in which A solely and exclusively consists of B) and to a situation in which, besides B, one or more further elements are present in entity A, such as element C, elements C and D or even further elements.

[0139] Further, it shall be noted that the terms “at least one”, “one or more” or similar expressions indicating that a feature or element may be present once or more than once typically are used only once when introducing the respective feature or element. In most cases, when referring to the respective feature or element, the expressions “at least one” or “one or more” are not repeated, nonwithstanding the fact that the respective feature or element may be present once or more than once.

[0140] Further, as used herein, the terms "preferably", "more preferably", "particularly", "more particularly", "specifically", "more specifically" or similar terms are used in conjunction with optional features, without restricting alternative possibilities. Thus, features introduced by these terms are optional features and are not intended to restrict the scope of the claims in any way. The invention may, as the skilled person will recognize, be performed by using alternative features. Similarly, features introduced by "in an embodiment of the invention" or similar expressions are intended to be optional features, without any restriction regarding alternative embodiments of the invention, without any restrictions regarding the scope of the invention and without any restriction regarding the possibility of combining the features introduced in such way with other optional or non-optional features of the invention.

[0141] Summarizing and without excluding further possible embodiments, the following embodiments may be envisaged:

[0142] Embodiment 1 . A device for authenticating a user, the device comprising: at least one projector configured for projecting a plurality of light beams through at least one display onto the user, at least one image generation unit configured for generating a pattern image showing the projecting of the plurality of light beams onto the user; at least one processor configured for extracting liveness data from the pattern image and allowing the user to perform an operation on the device that requires authentication based on the liveness data; at least one control unit and at least one status inquiry device configured for retrieving at least one item of status information on a current environmental status of the device, wherein the control unit is configured for automatically triggering at least one calibration depending on the fulfillment of at least one predetermined environmental status condition.

[0143] Embodiment 2. The device according to the preceding embodiment, wherein the item of status information on a current environmental status refers to one or more of: an environmental lighting condition such a level of ambient light and / or spatial location and / or orientation with respect to an external light source; a weather condition, an item of temperature information representing a current temperature; an item of position information of the device; an item of orientation information of the device; a relative orientation and / or a relative location between the device and at least one item in the environment; an item of handling information representing a current mode of handling of the device by a user; at least one item of information available via at least one network; a current environmental status approximated from the analysis of previously recorded environmental status information.

[0144] Embodiment 3. The device according to any one of the preceding embodiments, wherein the predetermined environmental status condition comprises at least one condition selected from the group consisting of: a level of ambient light is within a predetermined suitable level range for performing a calibration measurement; the device is in a suitable location for performing a calibration measurement; the device is in a suitable orientation for performing a calibration measurement; the device is not faced towards an external light source; the device is not in proximity to an object or pointed to an object; the device is not located within a pocket; the device is at a temperature within a predetermined temperature range suited for a calibration measurement; a temporal temperature change is within a predetermined range suited for a calibration measurement; weather conditions are within a predetermined range suited for a calibration measurement; the device presently is not used for another function; the device is available via at least one network.

[0145] Embodiment 4. The device according to any one of the preceding embodiments, wherein the device is configured for using at least one integrated sensor device of the device as at least a part of the at least one status inquiry device.

[0146] Embodiment 5. The device according to any one of the preceding embodiments, wherein the status inquiry device comprises at least one device selected from the group consisting of: a front camera being positioned on the same side as the display; a rear camera being positioned on the opposing side as the display; a location sensor; an illumination sensor configured for determining at least one state of illumination in an environment of the device; a temperature sensor; a motion sensor; a gyroscopic sensor; a magnetic sensor; a material sensor configured for determining at least one material property of at least one object in proximity of the device; a spectrometer device configured for acquiring at least one item of spectral information; at least one software sensor configured for generating information about the status of the device by processing input from a plurality of physical sensors.

[0147] Embodiment 6. The device according to any one of the preceding embodiments, wherein the status inquiry device is configured for determining if the device is placed on a table in a room by using at least one gyroscopic sensor, and / or wherein the status inquiry device is configured for determining an ambient light level by determining at least one dark image without the projector projecting the plurality of light beams, and / or wherein the status in- quiry device is configured for determining if a ceiling is visible by capturing at least one image with the projector projecting the plurality of light beams, and / or wherein the status inquiry device is configured for determining presence of an obstacle in proximity of the device by using at least one proximity sensor.

[0148] Embodiment 7. The device according to any one of the preceding embodiments, wherein the control unit is configured for repeatedly automatically triggering the calibration, wherein the control unit is configured for performing status checks at predetermined points in time, wherein the status checks each comprise receiving the at least one item of status information on the current environmental status and checking whether the at least one predetermined environmental status condition is fulfilled, and wherein the control unit further is configured for triggering the calibration depending on whether the at least one environmental status condition is fulfilled.

[0149] Embodiment 8. The device according to any one of the preceding embodiments, wherein the calibration comprises at least one user guided calibration and / or an automated calibration, wherein the calibration comprises one or more of calibrating a reference pattern, calibrating a position of the projector, at least one temperature calibration, or at least one diffraction calibration.

[0150] Embodiment 9. The device according to any one of the preceding embodiments, wherein the calibration comprises calibrating a reference pattern by performing the following steps: i) capturing at least one calibration image at a relative distance between the device and at least one reference object by using the image generation unit while the projector projecting the plurality of light beams; ii) determining a reflection pattern by identifying light spots on the captured calibration image generated by the reference object in response to illumination by the plurality of light beams; iii) matching the light spots of the reflection pattern to features of a reference pattern considering an estimate of the different relative distances between the device and the reference object while capturing the calibration image, thereby determining pairs of matched reflection features; iv) recalculating the reference pattern from image coordinates of the corresponding matched light spots.

[0151] Embodiment 10. The device according to the preceding embodiment, wherein the determining of the calibration images comprises imaging at least one two-dimensional images of a white surface for different relative distances by using the image generation unit with the projector projecting the plurality of light beams.

[0152] Embodiment 11 . The device according to any one of the preceding embodiments, wherein the calibration comprises calibrating a position of the projector by performing the following steps: I) capturing at least one calibration face image at a relative distance between the device and a user’s face by using the image generation unit while the projector projecting the plurality of light beams;

[0153] II) estimating the relative distance by analyzing the calibration face image, wherein the analyzing comprises extracting landmarks of the face by using a two-dimensional face detection;

[0154] III) identifying light spots on the user’s face on the calibration face image and matching the identified light spots to features of a reference pattern considering the estimated relative distance, thereby determining pairs of matched reflection features;

[0155] IV) determining a translation vector describing the position of the projector by determining an epipolar line distance for each of the pairs of the matched reflection features.

[0156] Embodiment 12. The device according to the preceding embodiment, wherein the determining of the calibration face image comprises imaging at least one two-dimensional image of the user’s face with a relative distance between the projector and the user’s face with the projector projecting the plurality of light beams by using the image generation unit.

[0157] Embodiment 13. The device according to any one of the preceding embodiments, wherein the calibration comprises at least one temperature calibration, wherein the temperature calibration comprises the following steps: a) capturing at least one temperature calibration image of at least one reference object at least a plurality of temperatures while the projector projecting the plurality of light beams; b) identifying light spots on the temperature calibration images and matching the identified light spots to features of a reference pattern considering an estimated relative distance between the reference object and the projector, thereby determining pairs of matched reflection features for each temperature; c) determining a temperature dependent correction of the reference pattern by determining deviations between the position of the matched reflection features for each temperature, and storing the temperature dependent correction of the reference pattern in at least one database.

[0158] Embodiment 14. The device according to any one of the preceding embodiments, wherein the calibration comprises at least one diffraction calibration, wherein the diffraction calibration comprises the following steps:

[0159] A) capturing at least one diffraction calibration image of at least one reference object while the projector projecting the plurality of light beams by using a camera behind a punchhole;

[0160] B) identifying light spots on the diffraction calibration image and matching the identified light spots to features of a reference pattern considering an estimated relative distance between the reference object and the projector, thereby determining pairs of matched reflection features; C) determining a diffraction correction of the reference pattern by determining deviations between the position of the matched light spots and features, and storing the diffraction correction of the reference pattern in at least one database.

[0161] Embodiment 15. The device according to any one of the preceding embodiments, wherein the device is selected from the group consisting of: a television device; a game console; a personal computer; a mobile device, particularly a mobile communication device such as a cell phone, a smart phone, a tablet computer, a laptop, a tablet, a virtual reality device, or a wearable such as a smart watch; or another type of portable computer.

[0162] Embodiment 16. The device according to any one of the preceding embodiments, wherein the display is or comprises at least one organic light-emitting diode (OLED) display and / or at least one quantum-dot light emitting diode (QLED) display.

[0163] Embodiment 17. The device according to any one of the preceding embodiments, wherein extracting liveness data comprises extracting material data and / or extracting blood perfusion data, wherein extracting material data comprises providing the pattern image to a model and / or receiving material data from the model, wherein extracting blood perfusion data comprises determining a speckle contrast of the pattern image and determining a blood perfusion measure based on the determined speckle contrast, wherein a speckle contrast represents a measure for a mean contrast of an intensity distribution within an area of a speckle pattern.

[0164] Embodiment 18. The device according to any one of the preceding embodiments, wherein the automatically triggering at least one calibration comprises the process of starting the calibration without the necessity of human interaction, specifically without any human interaction or with only optional human interaction.

[0165] Embodiment 19. A method of performing at least one reference measurement with a device according to any one of the preceding embodiments, the method comprising: i. retrieving at least one item of status information on a current environmental status of the device;

[0166] II. checking if at least one environmental status condition for performing at least one calibration is fulfilled; and ill. automatically triggering at least one calibration if, in step ii. , the environmental status condition is fulfilled.

[0167] Embodiment 20. The method according to the preceding embodiment, wherein the method steps i.-iii. are performed repeatedly.

[0168] Embodiment 21 . The method of performing at least one reference measurement with a device according to any one of the preceding embodiments referring to a method, the method comprising projecting a plurality of light beams through at least one display onto the user by using at least one projector; generating a pattern image showing the projecting of the plurality of light beams onto the user by using at least one image generation unit; extracting liveness data from the pattern image and allowing the user to perform an operation on the device that requires authentication based on the liveness data by using at least one processor; retrieving at least one item of status information on a current environmental status of the device by using at least one control unit and at least one status inquiry device, and automatically triggering at least one calibration depending on the fulfillment of at least one predetermined environmental status condition.

[0169] Embodiment 22. A computer program comprising instructions which, when the program is executed by the control unit of the device according to any one of the preceding embodiments referring to a device, cause the control unit to perform the method according to any one of the preceding embodiments referring to a method.

[0170] Embodiment 23. A computer-readable storage medium comprising instructions which, when the instructions are executed by the control unit of the device according to any one of the preceding embodiments referring to a device, cause the control unit to perform the method according to any one of the preceding embodiments referring to a method.

[0171] Short description of the Figures

[0172] Further optional features and embodiments will be disclosed in more detail in the subsequent description of embodiments, preferably in conjunction with the dependent claims. Therein, the respective optional features may be realized in an isolated fashion as well as in any arbitrary feasible combination, as the skilled person will realize. The scope of the invention is not restricted by the preferred embodiments. The embodiments are schematically depicted in the Figures. Therein, identical reference numbers in these Figures refer to identical or functionally comparable elements.

[0173] In the Figures:

[0174] Figure 1 shows an embodiment of a device according to the present invention;

[0175] Figure 2 shows a flowchart of an embodiment of a method according to the present invention; Figures 3A to 3D show an embodiment of calibrating a reference pattern; and

[0176] Figures 4A and 4D show an embodiment of calibrating a position of the projector.

[0177] Detailed description of the embodiments

[0178] Figure 1 shows in a highly schematic fashion an embodiment of a device 110 for authenticating a user 112 according to the present invention.

[0179] The device 110 may be selected from the group consisting of: a television device; a game console; a personal computer; a mobile device, particularly a cell phone, and / or a smart phone, and / or, and / or a tablet computer, and / or a laptop, and / or a tablet, and / or a virtual reality device, and / or a wearable, such as a smart watch; or another type of portable computer. The device, in particular, may be a portable device.

[0180] The authenticating may comprise verifying an identity of a user 112. Specifically, the authentication may comprise distinguishing between the user 112 from other humans or objects, in particular between an authorized access from a non-authorized access. The authentication may comprise verifying identity of a respective user and / or assigning identity to a user 112. The authentication may comprise generating and / or providing identity information, e.g. to other devices or units such as to at least one authorization unit 114 for authorization for providing access to the device. The identify information may be proofed by the authentication. For example, the identity information may be and / or may comprise at least one identity token. In case of successful authentication an image of a face recorded by at least one image generation unit may be verified to be an image of the user’s face and / or the identity of the user is verified. The authenticating may be performed using at least one authentication process. The authentication process may comprise a plurality of steps such as at least one face detection, e.g. on at least one flood image as will be described in more detail below, and at least one identification step in which an identity is assigned to the detected face and / or at least one identity check and / or verifying an identity of the user is performed.

[0181] The device 110 comprises at least one projector 116 configured for projecting a plurality of light beams through at least one display 118 onto the user 112. The projecting may comprise providing at least one light beam, in particular a light pattern onto at least one surface. The projector 116 may be an optical device configured for projecting at least one light beam onto a surface. The projector 116 is configured for projecting a plurality of light beams. The plurality of light beams may form a light pattern. The projector 116 may be configured for generating and / or for providing at least one light pattern, in particular at least one infrared light pattern. The infrared light pattern may be a near infrared light pattern. The light pattern may comprise at least one regular and / or constant and / or periodic pattern such as a triangular pattern, a rectangular pattern, a hexagonal pattern or a pattern comprising further convex tilings. For example, the light pattern is a hexagonal pattern, preferably a hexagonal light pattern, preferably a 2 / 5 hexagonal infrared light pattern. Using a periodical 2 / 5 hexagonal pattern can allow distinguishing between artefacts and usable signal. The light pattern may comprise at least one point pattern.

[0182] The projector 116 may comprise at least one least one emitter, in particular a plurality of emitters. The emitter may be selected from the group consisting of: at least one laser source such as at least one semi-conductor laser, at least one double heterostructure laser, at least one external cavity laser, at least one separate confinement heterostructure laser, at least one quantum cascade laser, at least one distributed Bragg reflector laser, at least one polariton laser, at least one hybrid silicon laser, at least one extended cavity diode laser, at least one quantum dot laser, at least one volume Bragg grating laser, at least one Indium Arsenide laser, at least one Gallium Arsenide laser, at least one transistor laser, at least one diode pumped laser, at least one distributed feedback lasers, at least one quantum well laser, at least one interband cascade laser, at least one semiconductor ring laser, at least one vertical cavity surface emitting laser (VCSEL); at least one non-laser light source such as at least one LED or at least one light bulb; at least one edge-emitting laser.

[0183] The display 118 may be or may comprise at least one organic light-emitting diode (OLED) display and / or at least one quantum-dot light emitting diode (QLED). The display may be at least partially transparent. The display may be at least partially transparent in at least one continuous areas covering the projector 116, a flood illumination source 120 and / or an image generation unit 122. The display 118 may have a transmission below or equal to 20 %, preferably below or equal to 15 %, more preferably below or equal to 10 %. For example, an intensity of a light beam after being projected through the display 118 may correspond to < 10 % of the intensity associated with the light beam when being emitted.

[0184] The device 110 comprises at least one image generation unit 122 configured for generating a pattern image showing the projecting of the plurality of light beams onto the user 112. The image generation unit 122 may be configured for generating at least one image. The image may be generated via a hardware and / or a software interface, which may be considered as the image generation unit 122. The image generation unit 122 may comprise at least one optical sensor, in particular at least one pixelated optical sensor. The image generation unit 120 may comprise at least one CMOS sensor or at least one CCD chip. For example, the image generation unit 122 may comprise at least one CMOS sensor, which may be sensitive in the infrared spectral range. For example, the image generation unit 122 may comprise one or more of at least one monochrome camera e.g. comprising monochrome pixels, at least one color (e.g. RGB) camera e.g. comprising color pixels, at least one IR camera. The camera may be a CMOS camera. The camera may comprise at least one monochrome camera chip, e.g. a CMOS chip. The camera may comprise at least one color camera chip, e.g. an RGB CMOS chip. The camera may comprise at least one IR camera chip, e.g. an IR CMOS chip. For example, the camera may comprise monochrome, e.g. black and white, pixels and color pixels. The color pixels and the monochrome pixels may be combined internally in the camera. The camera generally may comprise a one-dimensional or two-dimensional array of image sensors, such as pixels. For example, the camera may be an internal and / or external camera of the device. As described in the above, the internal and / or external camera of the device may be accessed via a hardware and / or a software interface, which is used as the image generation unit. In case, the device 110 is or comprises a smartphone the image generating unit 122 may be a front camera, such as a selfie camera, and / or back camera of the smartphone.

[0185] The pattern image may be an image generated by the image generation unit while illuminating with the light pattern, e.g. on an object and / or a user. The pattern image may comprise an image showing a user, in particular at least parts of the face of the user, while the user is being illuminated with the light pattern, particularly on a respective area of interest comprised by the image. The pattern image may be generated by imaging and / or recording light reflected by an object and / or user which is illuminated by the light pattern. The pattern image showing the user may comprise at least a portion of the illuminated light pattern on at least a portion the user. For example, the projection by the projector and the imaging by using the image generation unit 122 may be synchronized, e.g. by using at least one control unit of the device 110.

[0186] The device 110 may further comprise at least one flood illumination source 120 configured for emitting flood light. The image generation unit 122 may be configured for generating at least one flood image while the flood illumination source 120 is emitting flood light. The emitting of the flood light and the illumination of the light pattern may be performed subsequently or at at least partially overlapping times. For example, the flood light and the light pattern may be emitted at the same time. For example, one of the flood light or the light pattern may be emitted with a lower intensity compared to the other one.

[0187] The flood illumination source 120 may be configured for emitting flood light by and the image generation unit 122 may be configured for generating at least one flood image while the flood illumination source 120 is emitting flood light. The flood image may comprise an image showing a user, in particular the face of the user, while the user is being illuminated with the flood light. The flood image may be generated by imaging and / or recording light reflected by an object and / or user which is illuminated by the flood light. The flood image showing the user may comprise at least a portion of the flood light on at least a portion the user. For example, the illumination by the flood illumination source 120 and the imaging by using the image generation unit 122 may be synchronized, e.g. by using at least one control unit of the device 110.

[0188] The device 110 comprises the at least one processor 124 configured for extracting liveness data from the pattern image and allowing the user to perform an operation on the device 110 that requires authentication based on the liveness data. In particular, the device 110 may be configured for authenticating a user of the device 110 to perform at least one operation on the device that requires authentication. The authenticating may be performed by using the processor 124. The processor 124 may be or may be a part of at least one authentication unit configured for performing at least one authentication process of a user. The authentication unit may be configured for allowing the user to perform an operation on the device that requires authenti- cation based on the liveness data. Specifically, the authentication unit may be configured for using a facial recognition authentication process operating on the flood image, the pattern image and / or extracted liveness data, particularly derived from the pattern image.

[0189] For example, the authentication unit may perform at least one face detection using the flood image. The face detection may be performed locally on the device 110. Face identification, i.e. assigning an identity to the detected face, however, may be performed remotely, e.g. in the cloud, e.g. especially when identification needs to be done and not only verification. User templates can be stored at the remote device, e.g. in the cloud, and would not need to be stored locally. This can be an advantage in view of storage space and security. The authentication unit may be configured for identifying the user based on the flood image. Particularly therefore, the authentication unit may forward data to a remote device. Alternatively or in addition, the authentication unit may perform the identification of the user based on the flood image, particularly by running an appropriate computer program having a respective functionality.

[0190] The authentication process may comprise a plurality of steps. For example, the authentication process may comprise performing at least one face detection. The face detection step may comprise analyzing the flood image. In addition, for example, the authentication process may comprise identifying. The identifying may comprise assigning an identity to a detected face and / or at least one identity check and / or verifying an identity of the user. The identifying may comprise performing a face verification of the imaged face to be the user’s face. The identifying the user may comprise matching the flood image, e.g. showing a contour of parts of the user, in particular parts of the user’s face, with a template. The identifying of the user may comprise determining if the imaged face is the face of the user, in particular if the imaged face corresponds to at least one image of the user’s face stored in at least one memory, e.g. of the device. Authentication may be unsuccessful if the flood image cannot be matched with an image template.

[0191] The authentication process may comprise analyzing of the flood image, e.g. by one or more of the following: a filtering; a selection of at least one region of interest; a formation of a difference image between the flood image and at least one offset; an inversion of flood image; a background correction; a decomposition into color channels; a decomposition into hue; saturation; and brightness channels; a frequency decomposition; a singular value decomposition; applying a Canny edge detector; applying a Laplacian of Gaussian filter; applying a Difference of Gaussian filter; applying a Sobel operator; applying a Laplace operator; applying a Scharr operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transformation; applying a Radon- transformation; applying a Hough-transformation; applying a wavelet-transformation; a thresholding; creating a binary image. The region of interest may be determined manually by a user or may be determined automatically, such as by recognizing the user within the image. In particular, the analyzing of the flood image may comprise using at least one image recognition technique, in particular a face recognition technique. An image recognition technique comprises at least one process of identifying the user in an image. The image recognition may comprise us- ing at least one technique selected from the technique consisting of: color-based image recognition, e.g. using features such as template matching; segmentation and / or blob analysis e.g. using size, or shape; machine learning and / or deep learning e.g. using at least one convolutional neural network.

[0192] Authentication process may comprise extracting liveness data. The liveness data may comprise blood perfusion data and / or material data. Extracting liveness data may comprises extracting material data and / or extracting blood perfusion data. The liveness data may comprise information about a material of the surface of the user on which the spots are projected. The liveness data may comprise information about at least one vital sign. The extracting of liveness data, e.g. by using the authentication unit, may comprise extracting the material data from the pattern image by beam profile analysis of the light spots. With respect to beam profile analysis reference is made to WO 2018 / 091649 A1 , WO 2018 / 091638 A1 and WO 2018 / 091640 A1 , the full content of which is included by reference. Extracting material data from the pattern image may comprise generating the material type and / or data derived from the material type. Additionally or alternatively of using material data, the extracting of liveness data may comprise extracting blood perfusion data.

[0193] At least one operation on the device 110 that requires authentication may be access to the device 110, e.g. unlocking the device 110, and / or access to an application, preferably associated with the device 110 and / or access to a part of an application, preferably associated with the device 110.

[0194] The device 110 comprises the at least one control unit 126 and the at least one status inquiry device 128 configured for retrieving at least one item of status information on a current environmental status of the device 110. The control unit 126 is configured for automatically triggering at least one calibration depending on the fulfillment of at least one predetermined environmental status condition

[0195] The status inquiry device 128 may be a device or a combination of devices capable of our configured for retrieving the at least one item of status information. Specifically, the status inquiry device 128 may comprise at least one of an interface for retrieving the item of status information in an electronic format, such as a data format, such as a wireless or wire bound interface, and / or at least one device configured for generating the item of status information, such as at least one sensor device, as will be outlined in further detail below. The status inquiry device may 128 also fully or partially be integrated into the control unit 126. Additionally or alternatively, the status inquiry device 128 may comprise one or more devices integrated into the device for authenticating, such as one or more integrated sensors.

[0196] As an example, the item of status information on a current environmental status refers to one or more of: an environmental lighting condition such a level of ambient light and / or spatial location and / or orientation with respect to an external light source; a weather condition, an item of temperature information representing a current temperature; an item of position information of the device; an item of orientation information of the device; a relative orientation and / or a relative location between the device and at least one item in the environment; an item of handling information representing a current mode of handling of the device by a user; at least one item of information available via at least one network; a current environmental status approximated from the analysis of previously recorded environmental status information. For example, the current environmental status may be approximated from the analysis of previously recorded environmental status information. For determining the current environmental status measurements from the past may be used and / or considered. Actual measurements and measurements from the past can be combined.

[0197] The predetermined environmental status condition may be a condition of the environment the device for authenticating is in and / or a condition of a relationship between the device for authenticating and the environment which has been predetermined to be sufficient for performing the calibration As an example, the at least one environmental status condition may be at least one condition that must be fulfilled by the at least one item of status information on the current environmental status of the device for authenticating. As an example, the item of status information may be compared with at least one maximum or minimum threshold value, and the condition may be fulfilled and the calibration may, as an example, be automatically triggered when the item of status information is above or below the minimum or maximum threshold value, respectively. Additionally or alternatively, the environmental status condition may be fulfilled when the at least one item of status information on the current environmental status of the device for authenticating and / or at least one secondary value derived from the at least one item of status information by using a predetermined relationship or function are within at least one predetermined range. Thus, as an example, the control unit 126 may obtain the at least one item of status information on a current environmental status from the status inquiry device, may optionally transform the at least one item of status information and to at least one secondary value and may then check whether the at least one item of status information and / or the at least one secondary value fulfills the environmental status condition, such as has a predetermined target value or is within a predetermined range. If this environmental status condition is fulfilled, the control unit may automatically trigger, with or without delay, the at least one calibration.

[0198] The predetermined environmental status condition may comprise at least one condition selected from the group consisting of: a level of ambient light is within a predetermined suitable level range for performing a calibration measurement; the device is in a suitable location for performing a calibration measurement; the device is in a suitable orientation for performing a calibration measurement; the device is not faced towards an external light source; the device is not in proximity to an object or pointed to an object; the device is not located within a pocket; the device is at a temperature within a predetermined temperature range suited for a calibration measurement; a temporal temperature change is within a predetermined range suited for a calibration measurement; weather conditions are within a predetermined range suited for a calibration measurement; the device presently is not used for another function; the device is available via at least one network. The use of the at least one item of status information on a current environmental status of the device for authenticating may, thus, enable the device 110 for authenticating to perform calibrations repeatedly, when the predetermined environmental status condition indicates that the environmental conditions are suited for the at least one calibration. The at least one item of status information specifically may be retrieved by using integrated means of the device for authenticating.

[0199] The device 110 may be configured for using at least one integrated sensor device of the device as at least a part of the at least one status inquiry device. Thus, one or more integrated sensor devices may be used which are present in the device anyway, such as for one or more other purposes. The status inquiry device 128 may comprise at least one device selected from the group consisting of: a front camera being positioned on the same side as the display; a rear camera being positioned on the opposing side as the display; a location sensor; an illumination sensor configured for determining at least one state of illumination in an environment of the device; a temperature sensor; a motion sensor; a gyroscopic sensor; a magnetic sensor; a material sensor configured for determining at least one material property of at least one object in proximity of the device; a spectrometer device configured for acquiring at least one item of spectral information; at least one software sensor configured for generating information about the status of the device by processing input from a plurality of physical sensors.

[0200] The status inquiry device 128 may be configured for determining if the device is placed on a table in a room by using at least one gyroscopic sensor, and / or wherein the status inquiry device 128 is configured for determining an ambient light level by determining at least one dark image without the projector 116 projecting the plurality of light beams, and / or wherein the status inquiry device 128 is configured for determining if a ceiling is visible by capturing at least one image with the projector 116 projecting the plurality of light beams, and / or wherein the status inquiry device 128 is configured for determining presence of an obstacle in proximity of the device 110 by using at least one proximity sensor.

[0201] For example, the device 110 may try to detect if it is on a table in a room. A gyroscopic sensor may support to detect if the device, e.g. a smartphone, lies flat on a table. For example, the device may capture a frame of images without any flood light and light from the projector in order to determine if it is dark in the room, i.e. less sunlight. For example, proximity sensors can check that there lies no obstacle on the device, e.g. a smartphone. For example, the device 110 may capture a frame of images with projector on. If there is a ceiling this would be visible on the laser frame. The device 110 may detect the light pattern on that frame for calibration.

[0202] For example, the calibration comprises one or more of calibrating a reference pattern, calibrating a position of the projector, at least one temperature calibration, or at least one diffraction calibration. Figure 2 shows a flowchart of an embodiment of a method of performing at least one reference measurement with a device according to the present invention. In the method a device according to the present invention such as according to any one of the embodiments described above and / or according to any one of the embodiments described in further detail below, is used.

[0203] The method comprising: i. (130) retrieving at least one item of status information on a current environmental status of the device 110;

[0204] II. (132) checking if at least one environmental status condition for performing a calibration, in particular at least one calibration measurement, is fulfilled; and ill. (134) automatically triggering a calibration, in particular at least one calibration measurement, if, in step ii. , the environmental status condition is fulfilled.

[0205] The method steps may be performed in the given order or may be performed in a different order. Further, one or more additional method steps may be present which are not listed. Further, one, more than one or even all of the method steps may be performed repeatedly.

[0206] The calibration comprises at least one user guided calibration and / or an automated calibration.

[0207] The calibration may comprise a process as described in WO 2022 / 253777, the content of which is included by reference.

[0208] Figures 3A to 3D and Figures 4A to 4D show embodiments of calibrations. In Figures 3A to 3D an embodiment of calibrating a reference pattern is shown and in Figures 4A and 4D an embodiment of calibrating a position of the projector is depicted.

[0209] The calibration may comprise calibrating a reference pattern and / or, in particular subsequently, calibrating a position of the projector.

[0210] The calibration may comprise calibrating a reference pattern by performing the following steps: i) capturing at least one calibration image at a relative distance between the device 110 and at least one reference object 136 by using the image generation unit 122 while the projector 116 projecting the plurality of light beams; ii) determining a reflection pattern by identifying light spots on the captured calibration image generated by the reference object 136 in response to illumination by the plurality of light beams; iii) matching the light spots of the reflection pattern to features of a reference pattern considering an estimate of the different relative distances between the device 110 and the reference object 136 while capturing the calibration image, thereby determining pairs of matched reflection features; iv) recalculating the reference pattern from image coordinates of the corresponding matched light spots. In step i), as shown in Figures 3A and 3B, a plurality of calibration images may be captured at different relative distances, e.g. at least one calibration image at each of the relative distances. The determining of the calibration images may comprise imaging at least one two-dimensional images of a white surface for different relative distances by using the image generation unit with the projector projecting the plurality of light beams. For example, calibration may comprise identification of at least one flat, white surface (e.g. wall), e.g. by using a selfie camera while the smartphone is being carried around. For example, as shown in Figures 3A and 3B, the user 112 is requested to capture at least two calibration images of a wall with a coarse distance. The calibration image may be captured on a far distance ( 1 ,5m - 3m). The imaging of the grid of light spots with the device 110 is shown in Figure 3C.

[0211] At least one evaluation device, such as the processor 124, may be configured for performing an image analysis and for identifying light spots of the calibration image. The evaluation device may check if the imaged grid of the calibration image is complete. Then the light spots of said grid are matched with a reference pattern, such as a nominal known grid. In step iv) an estimated relative distance of the reference object 136 to the device 110 may be used. The estimated relative distance may be assumed to be at a nominal distance or determined, e.g. by the illumination of the laser spots. Next, the reference pattern is recalculated. The recalculation may be performed by using image coordinates of the corresponding matched light spots. The image coordinates of light spots may be mapped to projector coordinates. This yields a calibrated reference grid. Figure 3D shows a comparison of an initial reference pattern and the grid of light spots of the calibration image. A decalibration can be observed which can be corrected by recalculating the reference pattern.

[0212] For example, a complete automatic calibration may be performed as follows and may comprise the followings steps:

[0213] The smartphone tries to detect if it is on a table in a room. Gyro sensors can support to detect if the smartphone lies flat on a table.

[0214] Capturing an image frame with the image generation unit without any flood and projector in order to determine if it is dark in the room, i.e. less sunlight.

[0215] Proximity sensors can check that there lies no obstacle on the smartphone.

[0216] Capturing an image frame with the image generation unit with projector on. If there is a ceiling this is visible on the image frame, image the calibration image on that frame for calibration and performing steps i) to iv) of the calibration for calibrating a reference pattern .

[0217] The calibration may comprise calibrating a position of the projector 116. The calibrating of the position of the projector 116 may be performed after calibrating the reference pattern. The calibrating of the position of the projector 116 may comprise the following steps:

[0218] I) capturing at least one calibration face image at a relative distance between the device 110 and a user’s 112 face by using the image generation unit 122 while the projector 116 projecting the plurality of light beams; II) estimating the relative distance by analyzing the calibration face image, wherein the analyzing comprises extracting landmarks of the face by using a two-dimensional face detection;

[0219] III) identifying light spots on the user’s 112 face on the calibration face image and matching the identified light spots to features of a reference pattern considering the estimated relative distance, thereby determining pairs of matched reflection features;

[0220] IV) determining a translation vector describing the position of the projector by determining an epipolar line distance for each of the pairs of the matched reflection features.

[0221] The calibration face image may be an image of at least a part of the user’s 112 face used for calibration. The determining of the calibration face image may comprise imaging at least one two-dimensional image of the user’s face with a relative distance between the projector 116 and the user’s 112 face with the projector projecting the plurality of light beams by using the image generation unit 122. The calibration face image may be captured e.g. during enrollment or during an unlock. For example, the 2D image generated while a selfie camera can be used. Figures 4A and B shows imaging of two calibration face images of the user’s face at different distance between user 112 and device 110 by using the selfie camera.

[0222] The calibration face image, e.g. as shown in Figure 4C, may be analyzed by extracting landmarks of a face by a 2d face detection. The extraction of landmarks may be performed as described in e.g. en.wikipedia.org / wiki / Landmark_detection. This may allow for an estimation of the face distance. Additionally a distance sensor may be used.

[0223] The light pattern of the calibration face image may be analyzed. For example, in step III) only light spots on the face may be used. The light spots may be matched to the reference pattern, the matching can be performed as described above. This is possible since the face distance is known from using the landmarks. The face may be in the near field.

[0224] By matching the light spots to the reference pattern, the epipolar line direction can be extracted. Figure 4D shows matching of the light spots on the face with the reference pattern which yields the epipolar line directions. This yields the relative position of the projector 116, in particular a translation vector that locate the projector position. The length of the translation vector is the baseline length. This value is already known by the hardware design. The calibration is completed. The reference grid and the projector position (translation) is estimated. List of reference numbers device user authorization unit projector display flood illumination source image generation unit processor control unit status inquiry device retrieving checking automatically triggering a calibration reference object

Claims

Claims1 . A device (110) for authenticating a user (112), the device (110) comprising: at least one projector (116) configured for projecting a plurality of light beams through at least one display (118) onto the user (112), at least one image generation unit (122) configured for generating a pattern image showing the projecting of the plurality of light beams onto the user (112); at least one processor (124) configured for extracting liveness data from the pattern image and allowing the user (112) to perform an operation on the device (110) that requires authentication based on the liveness data; at least one control unit (126) and at least one status inquiry device (128) configured for retrieving at least one item of status information on a current environmental status of the device (110), wherein the control unit (126) is configured for automatically triggering at least one calibration depending on the fulfillment of at least one predetermined environmental status condition by the item of status information on the current environmental status.

2. The device (110) according to the preceding claim, wherein the item of status information on a current environmental status refers to one or more of: an environmental lighting condition such a level of ambient light and / or spatial location and / or orientation with respect to an external light source; a weather condition, an item of temperature information representing a current temperature; an item of position information of the device (110); an item of orientation information of the device (110); a relative orientation and / or a relative location between the device (110) and at least one item in the environment; an item of handling information representing a current mode of handling of the device (110) by a user (112); at least one item of information available via at least one network; a current environmental status approximated from the analysis of previously recorded environmental status information.

3. The device (110) according to any one of the preceding claims, wherein the predetermined environmental status condition comprises at least one condition selected from the group consisting of: a level of ambient light is within a predetermined suitable level range for performing a calibration measurement; the device (110) is in a suitable location for performing a calibration measurement; the device (110) is in a suitable orientation for performing a calibration measurement; the device (110) is not faced towards an external light source; the device (110) is not in proximity to an object or pointed to an object; the device (110) is not located within a pocket; the device (110) is at a temperature within a predetermined temperature range suited for a calibration measurement; a temporal temperaturechange is within a predetermined range suited for a calibration measurement; weather conditions are within a predetermined range suited for a calibration measurement; the device (110) presently is not used for another function; the device (110) is available via at least one network.

4. The device (110) according to any one of the preceding claims, wherein the status inquiry device (128) comprises at least one device selected from the group consisting of: a front camera being positioned on the same side as the display (118); a rear camera being positioned on the opposing side as the display (118); a location sensor; an illumination sensor configured for determining at least one state of illumination in an environment of the device; a temperature sensor; a motion sensor; a gyroscopic sensor; a magnetic sensor; a material sensor configured for determining at least one material property of at least one object in proximity of the device (110); a spectrometer device configured for acquiring at least one item of spectral information; at least one software sensor configured for generating information about the status of the device (110) by processing input from a plurality of physical sensors.

5. The device (110) according to any one of the preceding claims, wherein the calibration comprises at least one user guided calibration and / or an automated calibration, wherein the calibration comprises one or more of calibrating a reference pattern, calibrating a position of the projector, at least one temperature calibration, or at least one diffraction calibration.

6. The device (110) according to any one of the preceding claims, wherein the calibration comprises calibrating a reference pattern by performing the following steps: i) capturing at least one calibration image at a relative distance between the device (110) and at least one reference object (136) by using the image generation unit (122) while the projector (116) projecting the plurality of light beams; ii) determining a reflection pattern by identifying light spots on the captured calibration image generated by the reference object (136) in response to illumination by the plurality of light beams; iii) matching the light spots of the reflection pattern to features of a reference pattern considering an estimate of the different relative distances between the device (110) and the reference object (136) while capturing the calibration image, thereby determining pairs of matched reflection features; iv) recalculating the reference pattern from image coordinates of the corresponding matched light spots.

7. The device (110) according to any one of the preceding claims, wherein the calibration comprises calibrating a position of the projector by performing the following steps:I) capturing at least one calibration face image at a relative distance between the device (110) and a user’s (112) face by using the image generation unit (122) while the projector (116) projecting the plurality of light beams;II) estimating the relative distance by analyzing the calibration face image, wherein the analyzing comprises extracting landmarks of the face by using a two-dimensional face detection;III) identifying light spots on the user’s (112) face on the calibration face image and matching the identified light spots to features of a reference pattern considering the estimated relative distance, thereby determining pairs of matched reflection features;IV) determining a translation vector describing the position of the projector (116) by determining an epipolar line distance for each of the pairs of the matched reflection features.

8. The device (110) according to any one of the preceding claims, wherein the calibration comprises at least one temperature calibration, wherein the temperature calibration comprises the following steps: a) capturing at least one temperature calibration image of at least one reference object at least a plurality of temperatures while the projector (116) projecting the plurality of light beams; b) identifying light spots on the temperature calibration images and matching the identified light spots to features of a reference pattern considering an estimated relative distance between the reference object and the projector (116), thereby determining pairs of matched reflection features for each temperature; c) determining a temperature dependent correction of the reference pattern by determining deviations between the position of the matched reflection features for each temperature, and storing the temperature dependent correction of the reference pattern in at least one database.

9. The device (110) according to any one of the preceding claims, wherein the calibration comprises at least one diffraction calibration, wherein the diffraction calibration comprises the following steps:A) capturing at least one diffraction calibration image of at least one reference object while the projector (116) projecting the plurality of light beams by using a camera behind a punchhole;B) identifying light spots on the diffraction calibration image and matching the identified light spots to features of a reference pattern considering an estimated relative distance between the reference object and the projector (116), thereby determining pairs of matched reflection features;C) determining a diffraction correction of the reference pattern by determining deviations between the position of the matched light spots and features, and storing the diffraction correction of the reference pattern in at least one database.

10. The device (110) according to any one of the preceding claims, wherein the device (110) is selected from the group consisting of: a television device; a game console; a personal computer; a mobile device, particularly a mobile communication device such as a cellphone, a smart phone, a tablet computer, a laptop, a tablet, a virtual reality device, or a wearable such as a smart watch; or another type of portable computer.11 . The device (110) according to any one of the preceding claims, wherein the display (118) is or comprises at least one organic light-emitting diode (OLED) display and / or at least one quantum-dot light emitting diode (QLED) display.

12. The device (110) according to any one of the preceding claims, wherein extracting liveness data comprises extracting material data and / or extracting blood perfusion data, wherein extracting material data comprises providing the pattern image to a model and / or receiving material data from the model, wherein extracting blood perfusion data comprises determining a speckle contrast of the pattern image and determining a blood perfusion measure based on the determined speckle contrast, wherein a speckle contrast represents a measure for a mean contrast of an intensity distribution within an area of a speckle pattern.

13. A method of performing at least one reference measurement with a device (110) according to any one of the preceding claims, the method comprising: i. (130) retrieving at least one item of status information on a current environmental status of the device (110);II. (132) checking if at least one environmental status condition for performing at least one calibration measurement is fulfilled; and ill. (134) automatically triggering at least one calibration measurement if, in step ii., the environmental status condition is fulfilled by the item of status information on a current environmental status.

14. A computer program comprising instructions which, when the program is executed by the control unit of the device according to any one of the preceding claims referring to a device, cause the control unit to perform the method according to any one of the preceding claims referring to a method.

15. A computer-readable storage medium comprising instructions which, when the instructions are executed by the control unit of the device according to any one of the preceding claims referring to a device, cause the control unit to perform the method according to any one of the preceding claims referring to a method.

Images