Optical skin detection for facial unlocking

Abstract

A method for facial authentication is presented. The method comprises the following steps: a) at least one face detection step (110); b) at least one skin detection step (116); c) at least one 3D detection step (120); d) at least one authentication step (122).

Description

[0001] This application is a divisional application of the Chinese patent application "Optical Skin Detection for Facial Unlock" with application number 202280015245.4 (filing date: February 17, 2022). Technical Field

[0002] The present invention relates to a method for face authentication, a mobile device, and various uses of the method. The devices, methods, and uses according to the present invention can be specifically used in, for example, the following various fields: daily life, security technology, gaming, transportation technology, production technology, photography (such as digital photography or video photography for artistic, documentary, or technical purposes), security technology, information technology, agriculture, crop protection, maintenance, cosmetics, medical technology, or science. However, other applications are also possible. Background Art

[0003] In today's digital world, secure access to information technology is a fundamental requirement for any state-of-the-art system. Standard concepts such as passphrases or PIN codes have now been extended, combined, or even replaced by biometric methods such as fingerprint or face recognition. Although a passphrase can provide a high level of security if carefully chosen and of sufficient length, this step requires attention and the memorization of several potentially long phrases depending on the IT environment. In addition, there is never a guarantee that the person providing the passphrase is authorized by the passphrase or digital device owner. In contrast, biometric features such as face or fingerprint are unique and specific to an individual. Therefore, using functions derived from these is not only more convenient than passphrases / PIN codes but also more secure because they combine personal identity with the unlocking process.

[0004] Unfortunately, similar to stealing a password, fingerprints and faces can also be artificially created to impersonate legitimate users. The first generation of automatic face recognition tools used digital camera images or image streams, applying 2D image processing methods to extract characteristic features while using machine learning techniques to generate face templates for identity recognition based on these features. The second generation of face recognition algorithms uses deep convolutional neural networks instead of handcrafted image features to generate classification models.

[0005] However, both of these methods can be attacked, for example, by using high-quality photos that are currently freely downloadable from the Internet. Therefore, the idea of presentation attack detection (PAD) becomes significant. Early methods aimed to prevent simple attacks, such as presenting a photo of a legitimate user by recording a series of images and testing time-related features, such as the position of the head or minute but natural changes in blinking. These methods can be deceived again by playing a pre-recorded video of the user or by a carefully crafted animation generated from publicly available photos. To rule out the display as a potential object of deception, a near-infrared (NIR) camera can be used because the display emits photons only in the visible range of the electromagnetic spectrum. As a further countermeasure, 3D cameras were introduced, which can clearly distinguish a flat photo or a tablet with a played video from a 3D face. Nevertheless, these systems can still be attacked by high-quality masks, such as those generated by methods such as 3D printing, carefully 3D arranging 2D photos, or handcrafted silicone or latex masks. Since humans usually wear masks, liveness detection based on small movements also fails. However, these types of masks may be rejected by a PAD system that is capable of classifying human skin from other materials, as described in European Patent Application No. 20159984.2 filed on February 28, 2020, and European Patent Application No. 20154961.5190679 filed on January 31, 2020, the entire contents of which are incorporated by reference.

[0006] Another problem may stem from differences in the optical properties of the skin with respect to ethnic origin. Reliable recognition and PAD technologies need to be completely unaware of these different sources.

[0007] In addition to security considerations, the speed of the unlocking process and the required computational power are also important for providing an acceptable user experience. After successfully unlocking the device, high-speed face recognition can be used to perform multiple tasks, such as checking whether the user is still in front of the display or initiating further security applications, such as a banking application, by performing an instant check on the person in front of the display. Again, speed and computational resources have a significant impact on the user experience.

[0008] Current 3D algorithms are very computationally demanding, and presentation attack detection requires processing multiple video frames. Therefore, expensive hardware is needed to provide acceptable unlocking performance. In addition, the result is high power consumption.

[0009] In summary, current methods for face unlocking cannot reliably detect spoofing attacks with 3D masks and cannot perform this task at a speed below the human detection limit.

[0010] US2019 / 213309 A1 describes a system and method for authenticating a user's face using a range sensor. The range sensor includes a time-of-flight sensor and a reflectivity sensor. The range sensor emits a signal that is reflected off the user and received back at the range sensor. The received signal can be used to determine the distance between the user and the sensor and the reflectivity value of the user. Using the distance or reflectivity, a processor can activate a face recognition process in response to the distance and reflectivity.

[0011] Problems to be Solved by the Invention

[0012] Accordingly, an object of the present invention is to provide a device and method in the face of the above technical challenges of known devices and methods. Although simple presentation attacks using legitimate face photos and videos can be detected, there is still a lack of a method for reliably detecting presentation attacks using 3D masks. Specifically, another layer of security is needed so that a passphrase / PIN code can be replaced with a biometric derived from the face to unlock a digital device. In addition, a method that is completely agnostic to different skin types from different ethnic groups is needed. Summary of the Invention

[0013] This problem is solved by the present invention having the features of the independent patent claims. Advantageous developments of the present invention that can be implemented individually or in combination are presented in the dependent claims and / or the following description and detailed embodiments.

[0014] As used hereinafter, the terms "having", "comprising", or "including" or any arbitrary grammatical variants thereof are used in a non-exclusive manner. Thus, these terms can refer both to a situation where no other features exist in the entity described in the context except for the features introduced by these terms, and to a situation where one or more other features exist. As an example, the expressions "A has B", "A comprises B", and "A includes B" can all refer to a situation where no other elements exist in A except for B (i.e., the situation where A consists entirely of B), or to a situation where one or more other elements exist in entity A in addition to B, such as element C, elements C and D, or even additional elements.

[0015] Furthermore, it should be noted that the terms "at least one", "one or more", or similar expressions indicating that a feature or element can occur once or more than once are usually used only once when introducing the corresponding feature or element. Hereinafter, in most cases, when referring to individual features or elements, the expressions "at least one" or "one or more" will not be repeated, although the individual features or elements may occur once or more than once.

[0016] In addition, as used hereinafter, the terms "preferably", "more preferably", "specifically", "more specifically", "in particular", "even more particularly" or similar terms are used in conjunction with optional features and do not limit the possibilities of alternatives. Thus, the features introduced by these terms are optional features and are not intended to limit the scope of the claims in any way. As will be recognized by those skilled in the art, the present invention can be carried out by using alternative features. Similarly, features introduced by "in an embodiment of the present invention" or similar expressions are intended to be optional features without any limitation on alternative embodiments of the present invention, without any limitation on the scope of the present invention, and without any limitation on the possibility of combining the features introduced in this way with other optional or non-optional features of the present invention.

[0017] In a first aspect of the present invention, a method for face authentication is disclosed. The face to be authenticated may specifically be a human face. The term "face authentication" as used herein is a broad term and shall have the ordinary and customary meaning given to it by those of ordinary skill in the art and is not limited to a special or customized meaning. The term may specifically refer to, but is not limited to, verifying an identified object or a part of the identified object as a human face. Specifically, the authentication may include distinguishing a real human face from attack materials generated to mimic a human face. The authentication may include verifying the identity of the corresponding user and / or assigning an identity to the user. The authentication may include generating and / or providing identity information, for example, to other devices, such as to at least one authorized device, for authorizing access to a mobile device, a machine, a vehicle, a building, etc. The identity information may be proven through authentication. For example, the identity information may be and / or may include at least one identity token. In the case of successful authentication, the identified object or a part of the identified object is verified as a real face and / or the identity of the object, particularly the user, is verified.

[0018] The method comprises the following steps:

[0019] a) at least one face detection step, wherein the face detection step comprises determining at least one first image by using at least one camera, wherein the first image comprises at least one two-dimensional image of a scene suspected of including a face, and wherein the face detection step comprises detecting a face in the first image by using at least one processing unit to identify at least one predefined or predetermined geometric feature characteristic of the face in the first image;

[0020] b) At least one skin detection step, wherein the skin detection step includes projecting at least one illumination pattern including a plurality of illumination features onto a scene by using at least one illumination unit and determining at least one second image by using at least one camera, wherein the second image includes a plurality of reflection features generated by the scene in response to the illumination of the illumination features, wherein each reflection feature includes at least one beam profile, and wherein the skin detection step includes: using a processing unit to determine first beam profile information of at least one of the reflection features by analyzing the beam profile of at least one of the reflection features located within an image region of the second image, the image region of the second image corresponding to the image region of the first image including the identified geometric feature, and determining at least one material property of the reflection feature from the first beam profile information, wherein if the material property corresponds to at least one property characteristic of skin, the detected face is characterized as skin;

[0021] c) At least one 3D detection step, wherein the 3D detection step includes: using a processing unit to determine second beam profile information of at least four reflection features by analyzing the beam profiles of at least four reflection features located within an image region of the second image corresponding to the image region of the first image including the identified geometric feature, and determining at least one depth level from the second beam profile information of the reflection features, wherein if the depth level deviates from the depth level of a planar object, the detected face is characterized as a 3D object;

[0022] d) At least one authentication step, wherein the authentication step includes: authenticating the detected face by using at least one authentication unit if the face detected in step b) is characterized as skin and the face detected in step c) is characterized as a 3D object.

[0023] The method steps may be performed in the given order or may be performed in a different order. Additionally, there may be one or more additional method steps not listed. Further, one, more, or even all of the method steps may be repeated.

[0024] The term "camera" as used herein is a broad term and shall be given its ordinary and customary meaning by a person of ordinary skill in the art and is not limited to a special or customized meaning. The term may specifically refer to, but is not limited to, a device having at least one imaging element configured to record or capture spatially resolved one-dimensional, two-dimensional, or even three-dimensional optical data or information. The camera may be a digital camera. By way of example, the camera may include at least one camera chip, such as at least one CCD chip and / or at least one CMOS chip configured to record an image. The camera may be or may include at least one near-infrared camera.

[0025] As used herein, but not limited to, the term "image" can specifically relate to data recorded by using a camera, such as multiple electronic readings from an imaging device, such as pixels of a camera chip. In addition to at least one camera chip or imaging chip, the camera can also include additional elements, such as one or more optical elements, such as one or more lenses. As an example, the camera can be a fixed-focus camera having at least one lens that is fixedly adjusted relative to the camera. However, alternatively, the camera can also include one or more variable lenses that can be adjusted automatically or manually.

[0026] The camera can be a camera of a mobile device, such as a laptop, a tablet computer, or specifically a mobile phone, such as a smart phone, etc. Thus, specifically, the camera can be part of a mobile device that, in addition to at least one camera, also includes one or more data processing devices, such as one or more data processors. However, other cameras are also feasible. The term "mobile device" as used herein is a broad term and will be given its ordinary and customary meaning by those of ordinary skill in the art and is not limited to a special or customized meaning. The term can specifically refer to, but is not limited to, mobile electronic devices, and more specifically to mobile communication devices, such as cellular phones or smart phones. Additionally or alternatively, the mobile device can also refer to a tablet computer or another type of portable computer.

[0027] Specifically, the camera can be or can include at least one optical sensor having at least one photosensitive area. As used herein, an "optical sensor" generally refers to a photosensitive device for detecting a light beam, such as for detecting illumination and / or light spots generated by at least one light beam. As further used herein, a "photosensitive area" generally refers to an area of the optical sensor that can be externally irradiated by at least one light beam, and in response to such irradiation, generates at least one sensor signal. The photosensitive area can specifically be located on the surface of each optical sensor. However, other embodiments are also feasible. The camera can include multiple optical sensors, each optical sensor having a photosensitive area. As used herein, the term "optical sensor each having at least one photosensitive area" refers to a configuration having multiple individual optical sensors each having one photosensitive area and a configuration having one combined optical sensor having multiple photosensitive areas. Additionally, the term "optical sensor" refers to a photosensitive device configured to generate one output signal. In the case where the camera includes multiple optical sensors, each optical sensor can be implemented such that there is precisely one photosensitive area in the corresponding optical sensor, for example, by providing precisely one photosensitive area that can be illuminated, and in response to such illumination, precisely creating a uniform sensor signal for the entire optical sensor. Thus, each optical sensor can be a single-area optical sensor. However, the use of single-area optical sensors makes the setup of the camera particularly simple and efficient. Thus, as an example, off-the-shelf light sensors, such as off-the-shelf silicon photodiodes, each having precisely one photosensitive area, can be used in this setup. However, other embodiments are also feasible.

[0028] Specifically, the optical sensor can be or can include at least one photodetector, preferably an inorganic photodetector, more preferably an inorganic semiconductor photodetector, and most preferably a silicon photodetector. Specifically, the optical sensor can be sensitive in the infrared spectral range. The optical sensor can include at least one sensor element that includes a matrix of pixels. All pixels of the matrix or at least one set of optical sensors of the matrix can specifically be the same. Specifically, the same pixel set of the matrix can be provided for different spectral ranges, or all pixels can be the same in terms of spectral sensitivity. Additionally, the pixels can be the same in terms of size and / or their electronic or photoelectric properties. Specifically, the optical sensor can be or can include at least one array of inorganic photodiodes that is sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 microns. Specifically, the optical sensor can be sensitive in the part of the near-infrared region where silicon photodiodes are applicable, specifically in the range of 700 nm to 1100 nm. The infrared optical sensors that can be used for the optical sensor can be off-the-shelf infrared optical sensors, such as those available from trinamiX, Ludwigshafen am Rhein, D-67056, GermanyTM The infrared optical sensors commercially available under the Hertzstueck brand by GmbH. TM Thus, by way of example, the optical sensor may include at least one optical sensor of the intrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of: Ge photodiodes, InGaAs photodiodes, extended InGaAs photodiodes, InAs photodiodes, InSb photodiodes, HgCdTe photodiodes. Additionally or alternatively, the optical sensor may include at least one extrinsic photovoltaic type optical sensor, more preferably at least one semiconductor photodiode selected from the group consisting of: Ge:Au photodiodes, Ge:Hg photodiodes, Ge:Cu photodiodes, Ge:Zn photodiodes, Si:Ga photodiodes, Si:As photodiodes. Additionally or alternatively, the optical sensor may include at least one photoconductive sensor, such as a PbS or PbSe sensor, bolometer, preferably a bolometer selected from the group consisting of: VO bolometers and amorphous Si bolometers.

[0029] The optical sensor may be sensitive in one or more of the ultraviolet, visible or infrared spectral ranges. Specifically, the optical sensor may be sensitive in the visible spectral range from 500 nm to 780 nm, most preferably at 650 nm to 750 nm, or at 690 nm to 700 nm. Specifically, the optical sensor may be sensitive in the near-infrared region. Specifically, the optical sensor may be sensitive in the part of the near-infrared region where silicon photodiodes are applicable, specifically in the range from 700 nm to 1000 nm. Specifically, the optical sensor may be sensitive in the infrared spectral range, specifically in the range from 780 nm to 3.0 microns. For example, each optical sensor independently may be or may include at least one element selected from the group consisting of: photodiodes, optoelectronic elements, photoconductors, phototransistors or any combination thereof. For example, the optical sensor may be or may include at least one element selected from the group consisting of: CCD sensor elements, CMOS sensor elements, photodiodes, optoelectronic units, photoconductors, phototransistors or any combination thereof. Any other type of photosensitive element may be used. The photosensitive element may generally be made entirely or partially of inorganic materials and / or may be made entirely or partially of organic materials. Most commonly, one or more photodiodes may be used, such as commercially available photodiodes, such as inorganic semiconductor photodiodes.

[0030] An optical sensor may include at least one sensor element, which includes a matrix of pixels. Thus, by way of example, the optical sensor may be part of or constitute a pixelated optical device. For example, the optical sensor may be and / or may include at least one CCD and / or CMOS device. By way of example, the optical sensor may be part of or constitute at least one CCD and / or CMOS device having a matrix of pixels, with each pixel forming a photosensitive area.

[0031] As used herein, the term "sensor element" generally refers to a device or combination of devices configured to sense at least one parameter. At this time, the parameter may specifically be an optical parameter, and the sensor element may specifically be an optical sensor element. The sensor element may be formed as an integral, single device or as a combination of several devices. The sensor element includes an optical sensor matrix. The sensor element may include at least one CMOS sensor. The matrix may be composed of independent pixels, such as independent optical sensors. Thus, a matrix of inorganic photodiodes may be constituted. However, alternatively, a commercially available matrix may be used, such as one or more of a CCD detector (e.g., a CCD detector chip) and / or a CMOS detector (e.g., a CMOS detector chip). Thus, generally speaking, the sensor element may be and / or may include at least one CCD and / or CMOS device and / or the optical sensor may form a sensor array or may be part of a sensor array, such as the above matrix. Thus, by way of example, the sensor element may include a pixel array, such as a rectangular array having m rows and n columns, where m and n are independently positive integers. Preferably, more than one column and more than one row are given, i.e., n > 1, m > 1. Thus, by way of example, n may be from 2 to 16 or higher, and m may be from 2 to 16 or higher. Preferably, the ratio of the number of rows to the number of columns is close to 1. By way of example, n and m may be selected such that 0.3 ≤ m / n ≤ 3, such as by selecting m / n = 1:1, 4:3, 16:9 or a similar ratio. By way of example, the array may be a square array, having the same number of rows and columns, such as by selecting m = 2, n = 2 or m = 3, n = 3, etc.

[0032] The matrix may be composed of independent pixels, such as independent optical sensors. Thus, a matrix of inorganic photodiodes may be constituted. However, alternatively, a commercially available matrix may be used, such as one or more of a CCD detector (e.g., a CCD detector chip) and / or a CMOS detector (e.g., a CMOS detector chip). Thus, generally, the optical sensor may be and / or may include at least one CCD and / or CMOS device and / or the optical sensor of a camera forms a sensor array or may be part of a sensor array, such as the above matrix.

[0033] The matrix can specifically be a rectangular matrix having at least one row, preferably multiple rows, and multiple columns. As an example, the rows and columns can be oriented substantially vertically. As used herein, the term "substantially vertical" refers to a vertically oriented situation with a tolerance, for example, of ±20° or less, preferably a tolerance of ±10° or less, and more preferably a tolerance of ±5° or less. Similarly, the term "substantially parallel" refers to a parallel oriented condition with a tolerance, for example, of ±20° or less, preferably a tolerance of ±10° or less, and more preferably a tolerance of ±5° or less. Thus, as an example, a tolerance of less than 20°, specifically less than 10°, or even less than 5° can be acceptable. To provide a wider field of view, the matrix can specifically have at least 10 rows, preferably at least 500 rows, and more preferably at least 1000 rows. Similarly, the matrix can have at least 10 columns, preferably at least 500 columns, and more preferably at least 1000 columns. The matrix can include at least 50 optical sensors, preferably at least 100,000 optical sensors, and more preferably at least 5,000,000 optical sensors. The matrix can include a plurality of pixels in the megapixel range. However, other embodiments are also feasible. Thus, in a desired axially rotationally symmetric setting, a circular or concentric arrangement of the optical sensors (which can also be referred to as pixels) of the matrix may be preferred.

[0034] Thus, as an example, the sensor element can be part of or constitute a pixelated optical device. For example, the sensor element can be and / or can include at least one CCD and / or CMOS device. As an example, the sensor element can be part of or constitute at least one CCD and / or CMOS device having a pixel matrix, with each pixel forming a photosensitive area. The sensor element can read out the matrix of the optical sensors using a rolling shutter or global shutter method.

[0035] The camera may also include at least one transfer device. The camera may also include one or more additional elements, such as one or more additional optical elements. The camera may include at least one optical element selected from the group consisting of: a transfer device, such as at least one lens and / or at least one lens system, at least one diffractive optical element. The term "transfer device", also denoted as "transfer system", generally may refer to one or more optical elements adapted to modify a light beam, such as by modifying one or more of the light beam parameters, the width of the light beam, or the direction of the light beam. The transfer device may be adapted to direct the light beam onto an optical sensor. Specifically, the transfer device may include one or more of the following: at least one lens, such as at least one lens selected from the group consisting of: at least one adjustable-focus lens, at least one aspherical lens, at least one spherical lens, at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one light beam deflection element, preferably at least one mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitter; at least one multi-lens system. The transfer device may have a focal length. As used herein, the "focal length" of the transfer device refers to the distance at which incident collimated light rays that can impinge on the transfer device enter the "focus", which may also be denoted as the "focal point". Thus, the focal length constitutes a measure of the ability of the transfer device to converge the incident light beam. Thus, the transfer device may include one or more imaging elements, which may have the effect of a converging lens. For example, the transfer device may have one or more lenses, particularly one or more refractive lenses, and / or one or more convex mirrors. In this example, the focal length may be defined as the distance from the center of a thin refractive lens to the principal focus of the thin lens. For a converging thin refractive lens, such as a convex or biconvex thin lens, the focal length may be considered positive and may provide the distance at which a collimated light beam irradiates the thin lens when the transfer device can be focused into a single light spot. Additionally, the transfer device may include at least one wavelength selection element, such as at least one filter. Additionally, the transfer device may be designed to impose a predetermined light beam profile on the electromagnetic radiation (e.g., in the sensor region and particularly at the location of the sensor region). In principle, the above optional embodiments of the transfer device may be implemented individually or in any desired combination.

[0036] The transfer device may have an optical axis. As used herein, the term "optical axis of the transfer device" generally refers to the mirror symmetry axis or the rotational symmetry axis of a lens or a lens system. As an example, the transfer system may include at least one beam path, wherein the elements of the transfer system in the beam path are positioned in a rotationally symmetric manner with respect to the optical axis. However, one or more optical elements located within the beam path may also be eccentric or tilted with respect to the optical axis. However, in such a case, the optical axis may be defined sequentially, for example, by interconnecting the centers of the optical elements in the beam path, such as by interconnecting the centers of the lenses, where, in such a case, the optical sensor is not counted as an optical element. The optical axis can generally represent the beam path. Wherein, the camera may have a single beam path along which the beam can travel from the object to the optical sensor, or may have multiple beam paths. As an example, a single beam path may be given or the beam path may be divided into two or more partial beam paths. In the latter case, each partial beam path may have its own optical axis. In the case of multiple optical sensors, the optical sensors may be located in the same beam path or partial beam path. However, optionally, the optical sensors may also be located in different partial beam paths.

[0037] The transfer device may constitute a coordinate system, wherein the longitudinal coordinate is the coordinate along the optical axis, and wherein d is the spatial offset from the optical axis. The coordinate system may be a polar coordinate system, wherein the optical axis of the transfer device forms the z-axis, and wherein the distance from the z-axis and the polar angle may be used as additional coordinates. A direction parallel or antiparallel to the z-axis may be considered the longitudinal direction, and the coordinate along the z-axis may be considered the longitudinal coordinate. Any direction perpendicular to the z-axis may be considered the transverse direction, and the polar coordinates and / or the polar angle may be considered the transverse coordinates.

[0038] The camera is configured to determine at least one image of a scene, particularly a first image. As used herein, the term "scene" may refer to a spatial region. The scene may include a face and the surrounding environment in authentication. The first image itself may include pixels, and the pixels of the image are related to the pixels of the matrix of the sensor elements. Thus, when referring to "pixels", it either refers to the unit of image information generated by a single pixel of the sensor element or directly refers to a single pixel of the sensor element. The first image is at least one two-dimensional image. As used herein, the term "two-dimensional image" generally may refer to an image having information about transverse coordinates (such as dimensions of height and width). The first image may be an RGB (red, green, blue) image. The term "determine at least one first image" may refer to capturing and / or recording the first image.

[0039] The face detection step includes detecting a face in a first image by using at least one processing unit to identify at least one predefined or predetermined geometric feature characteristic of the face in the first image. Specifically, the face detection step includes detecting a face in the first image by using at least one processing unit to identify at least one predefined or predetermined geometric feature as a face characteristic in the first image.

[0040] As further used herein, the term "processing unit" generally refers to any data processing device adapted to perform specified operations, for example, by using at least one processor and / or at least one application specific integrated circuit. Thus, by way of example, at least one processing unit may include software code stored thereon that includes a plurality of computer commands. The processing unit may provide one or more hardware elements for performing one or more specified operations and / or may provide one or more processors on which software for performing one or more specified operations runs. Operations including evaluating an image may be performed by at least one processing unit. Thus, by way of example, one or more instructions may be implemented in software and / or hardware. Thus, by way of example, the processing unit may include one or more programmable devices, such as one or more computers, application specific integrated circuits (ASICs), digital signal processors (DSPs), or field programmable gate arrays (FPGAs), configured to perform the above evaluation. However, additionally or alternatively, the processing unit may also be implemented fully or partially in hardware. The processing unit and the camera may be fully or partially integrated into a single device. Thus, generally, the processing unit may also form part of the camera. Or, the processing unit and the camera may be fully or partially embodied as separate devices.

[0041] The processing unit may be or may include one or more integrated circuits, such as one or more application specific integrated circuits (ASICs), and / or one or more data processing devices, such as one or more computers, preferably one or more microcomputers and / or microcontrollers, field programmable arrays, or digital signal processors. Additional components may be included, such as one or more preprocessing devices and / or data acquisition devices, such as one or more devices for receiving and / or preprocessing sensor signals, such as one or more AD converters and / or one or more filters. In addition, the processing unit may include one or more measuring devices, such as one or more measuring devices for measuring current and / or voltage. In addition, the processing unit may include one or more data storage devices. In addition, the processing unit may include one or more interfaces, such as one or more wireless interfaces and / or one or more wired interfaces.

[0042] The processing unit can be configured to display, visualize, analyze, distribute, transmit, or further process one or more of information (such as information obtained by a camera). As an example, the processing unit can be connected or integrated with at least one of the following: a display, a projector, a monitor, an LCD, a TFT, a speaker, a multi-channel sound system, an LED pattern, or another visualization device. It can also be connected or integrated with at least one of the following: a communication device or communication interface, a connector or port capable of sending encrypted or unencrypted information using one or more of email, text message, telephone, Bluetooth, Wi-Fi, infrared, or an Internet interface, port, or connection. It can further be connected to or integrated with at least one of the following: a processor, a graphics processor, a CPU, an Open Multimedia Application Platform (OMAPTM), an integrated circuit, a system-on-chip (such as products from the Apple A series or the Samsung S3C2 series), a microcontroller or microprocessor, one or more storage blocks (such as ROM, RAM, EEPROM, or flash memory), a timing source (such as an oscillator or phase-locked loop, counter timer, real-time timer, or power-on reset generator), a voltage regulator, a power management circuit, or a DMA controller. The individual units can also be connected via a bus (such as the AMBA bus) or integrated into an Internet of Things or Industry 4.0 type network.

[0043] The processing unit can be connected or have additional external interfaces or ports through additional external interfaces or ports, such as one or more of the following: serial or parallel interfaces or ports, USB, Centronics port, FireWire, HDMI, Ethernet, Bluetooth, RFID, Wi-Fi, USART, or SPI, or analog interfaces or ports, such as one or more ADCs or DACs, or a standardized interface or port to other devices, such as a 2D camera device using an RGB interface (such as CameraLink). The processing unit can also be connected through one or more of an inter-processor interface or port, an FPGA-FPGA interface, or a serial or parallel interface port. The processing unit can also be connected to one or more of the following: an optical disc drive, a CD-RW drive, a DVD+RW drive, a flash drive, a memory card, a disk drive, a hard disk drive, a solid state drive, or a solid state disk.

[0044] The processing unit may be connected or have one or more additional external connectors, such as one or more of the following: telephone connectors, RCA connectors, VGA connectors, genderless connectors, USB connectors, HDMI connectors, 8P8C connectors, BNC connectors, IEC 60320 C14 connectors, fiber optic connectors, D-subminiature connectors, RF connectors, coaxial connectors, SCART connectors, XLR connectors, and / or may include at least one suitable socket for one or more of these connectors.

[0045] Detecting a face in the first image may include identifying at least one predefined or predetermined geometric feature characteristic of the face. The term "geometric feature characteristic of the face" as used herein is a broad term and shall have the ordinary and customary meaning given to it by a person of ordinary skill in the art and is not limited to a special or customized meaning. The term may specifically refer to, but is not limited to, at least one geometry-based feature that describes the shape of the face and its components, particularly one or more of the nose, eyes, mouth, or eyebrows. The processing unit may include at least one database in which the geometric feature characteristics of the face are stored, such as in a look-up table. Techniques for identifying at least one predefined or predetermined geometric feature characteristic of the face are generally known to those skilled in the art. For example, face detection may be performed as described in Masi, Lacopo et al. "Deep face recognition: A survey" 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), IEEE, 2018, the entire content of which is incorporated by reference.

[0046] The processing unit may be configured to perform at least one image analysis and / or image processing to identify geometric features. The image analysis and / or image processing may use at least one feature detection algorithm. The image analysis and / or image processing may include one or more of the following: filtering; selecting at least one region of interest; background correction; decomposition into color channels; decomposition into hue, saturation, and / or luminance channels; frequency decomposition; singular value decomposition; applying a blob detector; applying a corner detector; applying the determinant of a Hessian filter; applying a region detector based on principal curvature; applying a histogram of gradient positions and orientations algorithm; applying a histogram of orientation gradient descriptors; applying an edge detector; applying a differential edge detector; applying a Canny edge detector; applying a Laplacian of Gaussian filter; applying a Difference of Gaussians filter; applying a Sobel operator; applying a Laplacian operator; applying a Scharr operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying a Radon transform; applying a Hough transform; applying a wavelet transform; thresholding; creating a binary image. The region of interest may be determined manually by a user or may be determined automatically, for example, by identifying features within a first image.

[0047] Specifically, after the face detection step, a skin detection step may be performed, including projecting at least one illumination pattern including a plurality of illumination features onto the scene by using at least one illumination unit. However, embodiments in which the skin detection step is performed before the face detection step are feasible.

[0048] As used herein, the term "illumination unit", also referred to as an illumination source, generally may refer to at least one arbitrary device configured to generate at least one illumination pattern. The illumination unit may be configured to provide an illumination pattern for illuminating the scene. The illumination unit may be adapted to directly or indirectly illuminate the scene, where the illumination pattern is remitted by the surface of the scene, in particular reflected or scattered, and thereby at least partially directed towards the camera. The illumination unit may be configured to illuminate the scene, for example, by directing a light beam towards the scene and reflecting the light beam. The illumination unit may be configured to generate an illumination light beam for illuminating the scene.

[0049] The lighting unit may include at least one light source. The lighting unit may include a plurality of light sources. The lighting unit may include an artificial light source, in particular at least one laser source and / or at least one incandescent lamp and / or at least one semiconductor light source, such as at least one light-emitting diode, in particular an organic light-emitting diode and / or an inorganic light-emitting diode. As an example, the light emitted by the lighting unit may have a wavelength in the range of 300 to 1100 nm, in particular 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range may be used, such as light in the range of 780 nm to 3.0 μm. Specifically, light in the portion of the near-infrared region particularly suitable for silicon photodiodes may be used, especially light in the range of 700 nm to 1100 nm.

[0050] The lighting unit may be configured to generate at least one illumination pattern in the infrared region. The illumination feature may have a wavelength in the near-infrared (NIR) range. The illumination feature may have a wavelength of approximately 940 nm. At this wavelength, melanin absorption is depleted, so the dark and bright complexes reflect light almost identically. However, other wavelengths in the NIR region are also possible, such as one or more of 805 nm, 830 nm, 835 nm, 850 nm, 905 nm, or 980 nm. Additionally, using light in the near-infrared region makes the light undetectable or only weakly detectable by the human eye and still detectable by silicon sensors, in particular standard silicon sensors.

[0051] The lighting unit may be configured to emit light of a single wavelength. In other embodiments, the lighting unit may be configured to emit light having multiple wavelengths, thereby allowing additional measurements in other wavelength channels.

[0052] As used herein, the term "ray" generally refers to a line perpendicular to the wavefront of light and pointing in the direction of the energy flow. As used herein, the term "beam" generally refers to a collection of rays. Hereinafter, the terms "ray" and "beam" will be used synonymously. As further used herein, the term "light beam" generally refers to an amount of light, specifically an amount of light traveling substantially in the same direction, including the possibility that the light beam has an expansion angle or a widening angle. The light beam may have a spatial extent. Specifically, the light beam may have a non-Gaussian beam profile. The beam profile may be selected from the group consisting of: a trapezoidal beam profile; a triangular beam profile; a conical beam profile. The trapezoidal beam profile may have a plateau region and at least one edge region. The light beam may specifically be a Gaussian beam or a linear combination of Gaussian beams, as will be further outlined in detail below. However, other embodiments are also feasible.

[0053] The illumination unit can be or can include at least one multi-beam light source. For example, the illumination unit can include at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the illumination unit can include at least one laser and / or laser source. Various types of lasers can be employed, such as semiconductor lasers, double heterostructure lasers, external cavity lasers, separate confinement heterostructure lasers, quantum cascade lasers, distributed Bragg reflector lasers, polariton lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, bulk Bragg grating lasers, indium arsenide lasers, transistor lasers, diode-pumped lasers, distributed feedback lasers, quantum well lasers, interband cascade lasers, gallium arsenide lasers, semiconductor ring lasers, extended cavity diode lasers, or vertical cavity surface emitting lasers. Additionally or alternatively, non-laser light sources can be used, such as LEDs and / or light bulbs. The illumination unit can include one or more diffractive optical elements (DOEs) adapted to generate an illumination pattern. For example, the illumination unit can be adapted to generate and / or project a point cloud, such as the illumination unit can include one or more of the following: at least one digital light processing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one light emitting diode array; at least one laser light source array. Due to its generally defined beam profile and other properties of operability, it is particularly preferred to use at least one laser source as the illumination unit. The illumination unit can be integrated into the housing of the camera or can be separate from the camera.

[0054] Furthermore, the illumination unit can be configured to emit modulated or unmodulated light. In the case of using multiple illumination units, different illumination units can have different modulation frequencies, which can later be used to distinguish the light beams.

[0055] One or more light beams generated by the illumination unit can generally propagate parallel to the optical axis or inclined with respect to the optical axis, for example including an angle with respect to the optical axis. The illumination unit can be configured such that one or more light beams propagate from the illumination unit towards the scene along the optical axis of the illumination unit and / or the camera. For this purpose, the illumination unit and / or the camera can include at least one reflecting element, preferably at least one prism, for deflecting the illumination light beam onto the optical axis. As an example, one or more light beams, such as laser beams, can include an angle with respect to the optical axis of less than 10°, preferably less than 5°, or even less than 2°. However, other embodiments are also feasible. Additionally, one or more light beams can be on the optical axis or off the optical axis. As an example, one or more light beams can be parallel to the optical axis and have a distance from the optical axis of less than 10 mm, preferably a distance from the optical axis of less than 5 mm, or even less than 1 mm, or can even coincide with the optical axis.

[0056] As used herein, the term "at least one illumination pattern" refers to at least one arbitrary pattern that includes at least one illumination feature adapted to illuminate at least a portion of a scene. As used herein, the term "illumination feature" refers to at least one at least partially extended feature of a pattern. An illumination pattern can include a single illumination feature. An illumination pattern can include multiple illumination features. An illumination pattern can be selected from the group consisting of: at least one dot pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; at least one pattern including an arrangement of periodic or aperiodic features. An illumination pattern can include regular and / or constant and / or periodic patterns, such as triangular patterns, rectangular patterns, hexagonal patterns, or patterns including additional convex tilings. An illumination pattern can exhibit at least one illumination feature selected from the group consisting of: at least one dot; at least one row; at least two lines, such as parallel lines or intersecting lines; at least one dot and one line; at least one arrangement of periodic or aperiodic features; at least one feature of any shape. An illumination pattern can include at least one pattern selected from the group consisting of: at least one dot pattern, particularly a pseudo-random dot pattern; a random dot pattern or a quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern including at least one known feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern, including at least one pattern of convex uniform tiling; at least one line pattern including at least one line; at least one line pattern including at least two lines (such as parallel lines or intersecting lines). For example, an illumination unit can be adapted to generate and / or project a point cloud. The illumination unit can include at least one light projector adapted to generate a point cloud such that the illumination pattern can include multiple dot patterns. The illumination pattern can include a periodic grid of laser spots. The illumination unit can include at least one mask adapted to generate an illumination pattern from at least one light beam generated by the illumination unit.

[0057] The distance between two features of an illumination pattern and / or the area of at least one illumination feature can depend on the circle of confusion in an image. As described above, the illumination unit can include at least one light source configured to generate at least one illumination pattern. Specifically, the illumination unit includes at least one laser source and / or at least one laser diode designated for generating laser radiation. The illumination unit can include at least one diffractive optical element (DOE). The illumination unit can include at least one point projector, such as at least one laser source and DOE, adapted to project at least one periodic dot pattern. As further used herein, the term "project at least one illumination pattern" can refer to providing at least one illumination pattern for illuminating at least one scene.

[0058] The skin detection step includes determining at least one second image, also referred to as a reflection image, using a camera. The method may include determining a plurality of second images. The reflection features of the plurality of second images can be used for skin detection in step b) and / or for 3D detection in step c).

[0059] The second image includes a plurality of reflection features generated by the scene in response to illumination with illumination features. As used herein, the term "reflection feature" may refer to a feature in the image plane generated by the scene in response to illumination (specifically having at least one illumination feature). Each reflection feature includes at least one beam profile, also referred to as a reflected beam profile. As used herein, the term "beam profile" of a reflection feature generally may refer to at least one intensity distribution of the reflection feature, such as the intensity distribution of a spot on an optical sensor, as a function of pixels. The beam profile may be selected from the group consisting of: a trapezoidal beam profile; a triangular beam profile; a conical beam profile; and a linear combination of Gaussian beam profiles.

[0060] The evaluation of the second image may include identifying reflection features of the second image. The processing unit may be configured to perform at least one image analysis and / or image processing to identify the reflection features. The image analysis and / or image processing may use at least one feature detection algorithm. The image analysis and / or image processing may include one or more of the following: filtering; selecting at least one region of interest; forming a difference image between the image created by the sensor signal and at least one offset; inverting the sensor signal by inverting the image created by the sensor signal; forming a difference image between images created by the sensor signal at different times; background correction; decomposition into color channels; decomposition into hues; saturation; and brightness channels; frequency decomposition; singular value decomposition; applying a blob detector; applying a corner detector; applying the determinant of a Hessian filter; applying a region detector based on principal curvature; applying a maximally stable extremal region detector; applying a generalized Hough transform; applying a ridge detector; applying an affine invariant feature detector; applying an affine adaptive interest point operator; applying a Harris affine region detector; applying a Hessian affine region detector; applying a scale-invariant feature transform; applying a scale-space extremum detector; applying a local feature detector; applying an accelerated robust features algorithm; applying a histogram of gradient positions and orientations algorithm; applying a histogram of orientation gradient descriptors; applying a Deriche edge detector; applying a differential edge detector; applying a spatio-temporal interest point detector; applying a Moravec corner detector; applying a Canny edge detector; applying a Laplacian of Gaussian filter; applying a difference of Gaussian filter; applying a Sobel operator; applying a Laplacian operator; applying a Scharr operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying a Radon transform; applying a Hough transform; applying a wavelet transform; thresholding; creating a binary image. The region of interest may be determined manually by a user or may be determined automatically, for example, by identifying features within the image generated by the optical sensor.

[0061] For example, the illumination unit may be configured to generate and / or project a point cloud such that multiple illumination regions are generated on the optical sensor (e.g., a CMOS detector). Additionally, there may be interference on the optical sensor, such as interference due to speckles and / or foreign light and / or multiple reflections. The processing unit may be adapted to determine at least one region of interest, such as one or more pixels illuminated by a light beam, which is used to determine the longitudinal coordinates of the corresponding reflection features, which will be described in more detail below. For example, the processing unit may be adapted to perform filtering methods, such as speckle analysis and / or edge filtering and / or object recognition methods.

[0062] The processing unit may be configured to perform at least one image correction. The image correction may include at least one background subtraction. The processing unit may be adapted to remove the influence of background light from the beam profile, for example, by imaging without further illumination.

[0063] The processing unit may be configured to determine the beam profile of a corresponding reflection feature. As used herein, the term "determine the beam profile" refers to identifying at least one reflection feature provided by an optical sensor and / or selecting at least one reflection feature provided by an optical sensor and evaluating at least one intensity distribution of the reflection feature. As an example, regions of a matrix may be used and evaluated to determine the intensity distribution, such as a three-dimensional intensity distribution or a two-dimensional intensity distribution, for example, along an axis or line passing through the matrix. As an example, the illumination center of a beam may be determined, such as by determining at least one pixel with the highest illumination, and a cross-sectional axis passing through the illumination center may be selected. The intensity distribution may be an intensity distribution as a function of coordinates along the cross-sectional axis passing through the illumination center. Other evaluation algorithms are also feasible.

[0064] The reflection feature may cover at least one pixel of the second image or may extend over at least one pixel of the second image. For example, the reflection feature may cover multiple pixels or may extend over multiple pixels. The processing unit may be configured to determine and / or select all pixels connected to and / or belonging to the reflection feature (such as a light spot). The processing unit may be configured to determine the intensity center by:

[0065]

[0066] where R coi is the position of the intensity center, r pixel is the pixel position, and l = ∑ j I total where j is the number of pixels j connected to and / or belonging to the reflection feature, and I total is the total intensity.

[0067] The processing unit is configured to determine first beam profile information of at least one reflection feature by analyzing the beam profile of at least one reflection feature located within an image region of the second image corresponding to the image region of the first image including the identified geometric feature. The method may include identifying the image region of the second image corresponding to the image region of the first image including the identified geometric feature. Specifically, the method may include matching the pixels of the first image and the second image and selecting the pixels of the second image corresponding to the image region of the first image including the identified geometric feature. The method may include additionally considering additional reflection features located outside the image region of the second image.

[0068] As used herein, the term "beam profile information" may refer to any information and / or property derived from and / or associated with the beam profile of a reflection feature. The first and second beam profile information may be the same or may be different. For example, the first beam profile information may be an intensity distribution, a reflection profile, an intensity center, a material characteristic. For skin detection in step b), beam profile analysis may be used. Specifically, beam profile analysis uses the reflection properties of coherent light projected onto the surface of an object to classify materials. The classification of materials may be performed as described in WO 2020 / 187719, EP application 20159984.2 filed on February 28, 2020, and / or EP application 20154961.5 filed on January 31, 2020, the entire contents of which are incorporated by reference. Specifically, a grid of periodically projected laser spots, such as the hexagonal grid described in EP application 20170905.2 filed on April 22, 2020, is projected and the reflected image is recorded with a camera. The beam profile of each reflection feature recorded by the camera may be analyzed by a feature-based method. The feature-based method may be explained below. The feature-based method may be used in combination with a machine learning method, which may allow the parameterization of a skin classification model. Alternatively or in combination, a convolutional neural network may be used to classify the skin by using the reflected image as an input.

[0069] Other methods for verifying a user's face are known, such as from US2019 / 213309 A1. However, these methods use time-of-flight (ToF) sensors. The well-known working principle of a ToF sensor is to emit light and measure the time span until the reflected light is received. In contrast, the proposed beam profile analysis uses a projected illumination pattern. It is not possible to use such a projected pattern with a ToF sensor. For example, considering coverage and thus allowing different positions on the face to be considered, using an illumination pattern may be advantageous. This may enhance the reliability and security of user face authentication.

[0070] The skin detection step may include determining at least one material property of a reflection feature from beam profile information by using a processing unit. Specifically, the processing unit is configured to identify a reflection feature that will be generated by irradiating biological tissue, particularly human skin, when its reflected beam profile meets at least one predetermined or predefined criterion. As used herein, the term "at least one predetermined or predefined criterion" refers to at least one property and / or value suitable for distinguishing biological tissue (particularly human skin) from other materials. The predetermined or predefined criterion may be or may include at least one predetermined or predefined value and / or threshold and / or threshold range related to a material property. When the reflected beam profile meets at least one predetermined or predefined criterion, the reflection feature may be indicated as being generated by biological tissue. As used herein, the term "indicate" refers to any indication, such as an electronic signal and / or at least one visual or auditory indication. The processing unit is configured to otherwise identify the reflection feature as non-skin. As used herein, the term "biological tissue" generally refers to a biological material containing living cells. Specifically, the processing unit may be configured for skin detection. The term "identify" as generated by biological tissue (particularly human skin) may refer to determining and / or verifying whether the surface to be examined or tested is or includes biological tissue (particularly human skin), and / or distinguishing biological tissue (particularly human skin) from other tissues (particularly other surfaces). The method according to the present invention may allow distinguishing human skin from one or more of inorganic tissue, metal surfaces, plastic surfaces, foams, papers, woods, displays, screens, fabrics. The method according to the present invention may allow distinguishing human biological tissue from the surface of a man-made or non-living object.

[0071] The processing unit can be configured to determine the material property m of the surface emitting the reflection feature by evaluating the beam profile of the reflection feature. As used herein, the term "material property" refers to at least one arbitrary property of a material that is configured to characterize and / or identify and / or classify the material. For example, the material property can be a property selected from the group consisting of: roughness, penetration depth of light into the material, property characterizing the material as a biological or non-biological material, reflectivity, specular reflectivity, diffuse reflectivity, surface property, measurement of translucency, scattering, especially backscattering behavior, etc. At least one material property can be a property selected from the group consisting of: scattering coefficient, translucency, transparency, deviation from Lambertian surface reflection, speckle, etc. As used herein, the term "determine at least one material property" can refer to assigning a material property to the corresponding reflection feature, especially to the detected face. The processing unit can include at least one database that includes a list and / or table of predefined and / or pre-determined material properties, such as a look-up list or look-up table. The list and / or table of material properties can be determined and / or generated by performing at least one test measurement, such as by performing a material test using a sample with known material properties. The list and / or table of material properties can be determined and / or generated at the manufacturer's and / or by the user. Material properties can also be assigned to a material classifier, such as one or more of the following: material name, material group, such as biological or non-biological material, translucent or non-translucent material, metal or non-metal, skin or non-skin, fur or non-fur, carpet or non-carpet, reflective or non-reflective, specular reflective or non-specular reflective, foam or non-foam, hair or non-hair, roughness group, etc. The processing unit can include at least one database that includes a list and / or table that includes material properties and associated material names and / or material groups.

[0072] The reflection properties of the skin can be characterized by the simultaneous occurrence of surface direct reflection (Lambertian-like) and subsurface scattering (volume scattering). This results in a wider laser spot on the skin compared to the above materials.

[0073] The first beam profile information may be a reflection profile. For example, without wishing to be bound by theory, human skin may have a reflection profile, also referred to as a backscatter profile, including a portion generated by back reflection from the surface (referred to as surface reflection) and a portion generated by very diffuse reflection of light penetrating the skin, referred to as the diffuse portion of the backscatter. Regarding the reflection profile of human skin, reference is made to "Lasertechnik in der Medizin: Grundlagen, Systeme, Anwendungen", "Wirkung von Laserstrahlung auf Gewebe", 1991, pages 10171 to 266, Jürgen Eichler, Theo Seiler, Springer Verlag, ISBN 0939-0979. The surface reflection of the skin may increase as the wavelength increases towards the near infrared. In addition, the penetration depth may increase as the wavelength increases from visible light to near infrared. The diffuse portion of the backscatter may increase with the penetration depth of the light. By analyzing the backscatter distribution, these properties can be used to distinguish skin from other materials.

[0074] Specifically, the processing unit may be configured to compare the reflected beam profile with at least one predetermined and / or pre-recorded and / or predefined beam profile. The predetermined and / or pre-recorded and / or predefined beam profile may be stored in a table or look-up table and may be determined empirically, for example, and may be stored in at least one data storage device of the detector as an example. For example, the predetermined and / or pre-recorded and / or predefined beam profile may be determined during the initial startup of the device performing the method according to the invention. For example, the predetermined and / or pre-recorded and / or predefined beam profile may be stored in at least one data storage device of the processing unit or the device by software, specifically by an application downloaded from an app store or the like. In the case where the reflected beam profile is the same as the predetermined and / or pre-recorded and / or predefined beam profile, the reflection feature may be identified as being generated by biological tissue. The comparison may include overlapping the reflected beam profile and the predetermined or predefined beam profile such that their intensity centers match. The comparison may include determining the deviation between the reflected beam profile and the predetermined and / or pre-recorded and / or predefined beam profile, such as the sum of the squares of the point-to-point distances. The processing unit may be adapted to compare the determined deviation with at least one threshold, wherein in the case where the determined deviation is below and / or equal to the threshold, the surface is indicated as biological tissue and / or the detection of biological tissue is confirmed. The threshold may be stored in a table or look-up table and may be determined empirically, for example, and may be stored in at least one data storage device of the processing unit as an example.

[0075] Additionally or alternatively, the first beam profile information can be determined by applying at least one image filter to the image of the region. As further used herein, the term "image" refers to a two-dimensional function f(x, y), where the luminance and / or color value is given for any x, y position in the image. This position can correspond to the discretization of the recording pixels. The luminance and / or color can be discretized corresponding to the bit depth of the optical sensor. As used herein, the term "image filter" refers to at least one mathematical operation applied to the beam profile and / or at least one specific region of the beam profile. Specifically, the image filter Ф maps the image f or the region of interest in the image to a real number where represents a feature, in particular a material feature. Images may be affected by noise, and so may features. Thus, features may be random variables. These features can be normally distributed. If the features are not normally distributed, they can be transformed to a normal distribution, for example, by a Box-Cox transformation.

[0076] The processing unit can be configured to determine at least one material feature by applying at least one material-related image filter Ф2 to the image As used herein, the term "material-related" image filter refers to an image having a material-related output. The output of the material-related image filter is denoted herein as "material feature" " or "material-related feature" ". The material feature can be or can include at least one information about at least one material property of the surface of the scene for which the reflection feature has been generated.

[0077] The material-related image filter can be at least one filter selected from the group consisting of: a luminance filter; a point shape filter; a squared norm gradient; a standard deviation; a smoothing filter, such as a Gaussian filter or a median filter; a contrast filter based on grey-level occurrence; an energy filter based on grey-level occurrence; a homogeneity filter based on grey-level occurrence; a dissimilarity filter based on grey-level occurrence; an energy filter of Law; a threshold region filter; or a linear combination thereof; or further a material-related image filter Ф 2other , which is related to one or more of the following: a luminance filter, a spot shape filter, a squared norm gradient, a standard deviation, a smoothness filter, an energy filter based on grey-level occurrence, a homogeneity filter based on grey-level occurrence, a dissimilarity filter based on grey-level occurrence, an energy filter of Law, or a threshold region filter, or a linear combination thereof |ρ Ф2other,Фm |≥0.40, where Фm One or more of the following: a luminance filter, a spot shape filter, a squared norm gradient, a standard deviation, a smoothing filter, an energy filter based on gray level occurrence, a uniformity filter based on gray level occurrence, a dissimilarity filter based on gray level occurrence, an energy filter of a law, or a threshold region filter, or a linear combination thereof. Another material-related image filter Ф 2other may be related to one or more of the material-related image filters Ф by |ρ Ф2other,Фm |≥0.60, preferably by |ρ Ф2other,Фm |≥0.80 m and one or more in the material-related image filter Ф

[0078] The material-related image filter may be at least one arbitrary filter Φ by hypothesis testing. As used herein, the term "by hypothesis testing" refers to the fact of rejecting the null hypothesis H0 and accepting the alternative hypothesis H1. Hypothesis testing may include testing the material relevance of an image filter by applying the image filter to a predefined data set. The data set may include a plurality of beam profile images. As used herein, the term "beam profile image" refers to the sum of N B Gaussian radial basis functions

[0079]

[0080] wherein each N B Gaussian radial basis function is defined by a center (x lk , y lk ), a prefactor a lk , and an exponential factor α = 1 / ∈. The exponential factor is the same for all Gaussian functions in all images. For all images f k , the center positions x lk , y lk are the same: Each beam profile image in the data set may correspond to a material classifier and a distance. The material classifier may be labels such as "Material A", "Material B", etc. The beam profile image may be generated by using the above formula f k (x, y) and combining the following parameter table:

[0081]

[0082] The values of x and y are integers corresponding to the pixels having . The image may have a pixel size of 32x32. The data set of beam profile images may be generated by using the above formula f k and combining a set of parameters to obtain a continuous description of f k . The value of each pixel in the 32x32 image may be obtained by evaluating f kis obtained by inserting integer values from 0, …, 31 for x and y in (x,y). For example, for the pixel (6,9), the value f can be calculated k (6,9).

[0083] Subsequently, for each image f k , the eigenvalue corresponding to the filter Φ can be calculated where z k is the f corresponding to an image from a predefined data set k distance value. This results in a data set with the corresponding generated eigenvalues The hypothesis test can use the null hypothesis that the filter does not distinguish between material classifiers. The null hypothesis can be given by H0: μ1 = μ2 = … = μ J where μ m is the expected value for each material group corresponding to the eigenvalue . The index m represents the material group. The hypothesis test can use the alternative hypothesis that the filter does distinguish between at least two material classifiers. The alternative hypothesis can be given by H1: . As used herein, the term “does not distinguish between material classifiers” means that the expected values of the material classifiers are the same. As used herein, the term “distinguish between material classifiers” means that at least two of the expected values of the material classifiers are different. As used herein, “distinguish between at least two material classifiers” is synonymous with “suitable material classifier”. The hypothesis test can include at least one analysis of variance (ANOVA) of the generated eigenvalues. Specifically, the hypothesis test can include determining the mean of the eigenvalues for each of the J materials, i.e., the overall J mean, for m ∈ [0, 1, …, J−1], where N m gives the number of eigenvalues for each of the J materials in the predefined data set. The hypothesis test can include determining the mean of all N eigenvalues The hypothesis test can include determining the sum of squares within:

[0084]

[0085] The hypothesis test can include determining the sum of squares between,

[0086]

[0087] The hypothesis test can include performing an F-test:

[0088] where d1 = N−J, d2 = J−1,

[0089] F(x) = 1 – CDF(x)

[0090] p = F(mssb / mssw)

[0091] Here, I x is the regularized incomplete Beta function, where the Euler Beta function and is the incomplete Beta function. If the p-value p is less than or equal to a predefined significance level, the image filter can pass the hypothesis test. If p ≤ 0.075, preferably p ≤ 0.05, more preferably p ≤ 0.025, and most preferably p ≤ 0.01, the filter can pass the hypothesis test. For example, in the case where the predefined significance level is α = 0.075, if the p-value is less than α = 0.075, the image filter can pass the hypothesis test. In this case, the null hypothesis H0 can be rejected and the alternative hypothesis H1 can be accepted. The image filter thus distinguishes at least two material classifiers. Thus, the image filter has passed the hypothesis test.

[0092] Next, assume that the reflected image includes at least one reflection feature, in particular a specular spot image, to describe the image filter. The specular spot image f can be given by the function where the background of the image f may have been subtracted. However, other reflection features are also possible.

[0093] For example, the material-dependent image filter can be a luminance filter. The luminance filter can return the luminance measurement of the specular spot as a material feature. The material feature can be determined by

[0094]

[0095] where f is the specular spot image. The distance of the specular spot is denoted by z, where z can be obtained, for example, by using defocus depth or depth-from-photon ratio techniques and / or by using triangulation techniques. The surface normal of the material is given by and can be obtained as the normal of the surface spanned by at least three measured specular spots. The vector is the direction vector of the light source. Since the position of the specular spot is known by using defocus depth or depth-from-photon ratio techniques and / or by using triangulation techniques, where the position of the light source is known as a parameter of the detector system, d ray , is the difference vector between the specular spot position and the light source position.

[0096] For example, the material-dependent image filter can be a filter with an output depending on the shape of the specular spot. The material-dependent image filter can return a value related to the translucency of the material as a material feature. The translucency of the material affects the shape of the specular spot. The material feature can be given by

[0097]

[0098] where 0 < α, β < 1 are the weights of the spot height h, and H represents the Heaviside function, i.e., H(x) = 1 for x ≥ 0 and H(x) = 0 for x < 0. The spot height h can be determined by the following formula

[0099]

[0100] where B r is the inner circle of the spot with radius r.

[0101] For example, the material - related image filter can be the squared - norm gradient. The material - related image filter can return a value related to the measurement of the soft and hard transitions and / or roughness of the spot as a material feature. The material feature can be defined as

[0102]

[0103] For example, the material - related image filter can be the standard deviation. The standard deviation of the spot can be determined by the following formula

[0104]

[0105] where μ is the mean value given by: μ = ∫(f(x))dx.

[0106] For example, the material - related image filter can be a smoothing filter, such as a Gaussian filter or a median filter. In one embodiment of the smoothing filter, the image filter can refer to the observation that the volume scattering exhibits a smaller speckle contrast compared to the diffuse scattering material. The image filter can quantify the smoothness of the spot corresponding to the speckle contrast as a material feature. The material feature can be determined by the following formula

[0107]

[0108] where is a smoothing function, such as a median filter or a Gaussian filter. The image filter can include dividing by the distance z, as described in the formula above. The distance z can be determined, for example, using the defocus depth or the photon depth ratio technique and / or by using triangulation techniques. This can allow the filter to be insensitive to distance. In one embodiment of the smoothing filter, the smoothing filter can be based on the standard deviation of the extracted speckle noise pattern. The speckle noise pattern N can be empirically described as

[0109] f(x) = f0(x)·(N(X)+1),

[0110] Among them, f0 is the image of the despeckle light spot. N(X) is the noise term of the simulated speckle pattern. Calculating the despeckle image can be difficult. Therefore, the despeckle image can be approximated by a smoothed version of f, that is Among them, is a smoothing operator, such as a Gaussian filter or a median filter. Therefore, the approximation of the speckle pattern can be given by the following formula

[0111]

[0112] The material characteristics of this filter can be determined by the following formula

[0113]

[0114] Among them, Var represents the variance function.

[0115] For example, the image filter can be a contrast filter based on the occurrence of gray levels. This material filter can be based on the matrix M of the occurrence of gray levels f,ρ (g1g2) = [p g1,g2 , and p g1,g2 is the occurrence rate of the gray level combination (g1, g2) = [f(x1, y1), f(x2, y2)], and the relationship ρ defines the distance between (x1, y1) and (x2, y2), that is, ρ(x, y) = (x + a, y + b), where a and b are selected from 0 and 1.

[0116] The material characteristics of the contrast filter based on the occurrence of gray levels can be given by the following formula

[0117]

[0118] For example, the image filter can be an energy filter based on the occurrence of gray levels. This material filter is based on the matrix of the occurrence of gray levels defined above.

[0119] The material characteristics of the energy filter based on the occurrence of gray levels can be given by the following formula

[0120]

[0121] For example, the image filter can be a uniformity filter based on the occurrence of gray levels. This material filter is based on the matrix of the occurrence of gray levels defined above.

[0122] The material characteristics of the uniformity filter based on the occurrence of gray levels can be given by the following formula

[0123]

[0124] For example, the image filter can be a dissimilarity filter based on the gray-level occurrence. The material filter is based on the gray-level occurrence matrix defined above.

[0125] The material characteristics of the dissimilarity filter based on the gray-level occurrence can be given by the following formula

[0126]

[0127] For example, the image filter can be an energy filter of a law. The material filter can be based on the law vectors L5 = [1, 4, 6, 4, 1] and E5 = [-1, -2, 0, -2, -1] and the matrices L5(E5) T and E5(L5) T .

[0128] Image f k is convolved with these matrices:

[0129]

[0130] and

[0131]

[0132] And the material characteristics of the energy filter of the law can be determined by the following formula:

[0133]

[0134] For example, the material-related image filter can be a threshold region filter. The material characteristics can associate two regions in the image plane. The first region Ω1 can be the region where the function f is greater than α times the maximum value of f. The second region Ω2 can be the region where the function f is less than α times the maximum value of f but greater than the threshold ε times the maximum value of f. Preferably, α can be 0.5 and ε can be 0.05. Due to speckle or noise, these regions may not simply correspond to the inner and outer circles around the center of the light spot. As an example, Ω1 can include speckles or unconnected regions in the outer circle. The material characteristics can be determined by the following formula

[0135]

[0136] where Ω1 = {x|f(x)>α·max(f(x))} and Ω2 = {x|ε·max(f(x))<f(x)<α·max(f(x))}.

[0137] The processing unit can be configured to use the material characteristics Determine the material properties of the surface with the generated reflection features based on at least one predetermined relationship between the material properties of the surface with the generated reflection features. The predetermined relationship can be one or more of an empirical relationship, a semi-empirical relationship, and an analytically derived relationship. The processing unit may include at least one data storage device for storing the predetermined relationship, such as a lookup list or a lookup table.

[0138] Although the feature-based method described above is sufficient to accurately distinguish skin and only surface scattering materials, the distinction between skin and carefully selected attack materials (which also involve volume scattering) is more challenging. Step b) may include using artificial intelligence, particularly a convolutional neural network. Using the reflection image as the input to the convolutional neural network can generate a classification model with sufficient accuracy to distinguish skin and other volume scattering materials. Since only physically valid information is passed to the network by selecting important regions in the reflection image, only a compact training dataset may be required. In addition, a very compact network architecture can also be generated.

[0139] Specifically, in the skin detection step, at least one parametric skin classification model can be used. The parametric skin classification model can be configured to classify skin and other materials by using a second image as input. The skin classification model can be parameterized by using one or more of machine learning, deep learning, neural networks, or other forms of artificial intelligence. The term "machine learning" as used herein is a broad term and shall be given its ordinary and customary meaning to a person of ordinary skill in the art and is not limited to a special or customized meaning. This term can specifically refer to, but is not limited to, a method of using artificial intelligence (AI) for automatic model building, particularly parametric models. The term "skin classification model" can refer to a classification model configured to distinguish human skin from other materials. The nature characteristics of the skin can be determined by applying an optimization algorithm according to at least one optimization objective on the skin classification model. Machine learning can be based on at least one neural network, particularly a convolutional neural network. The weights and / or topology of the neural network can be predetermined and / or predefined. Specifically, machine learning can be used to perform the training of the skin classification model. The skin classification model can include at least one machine learning architecture and model parameters. For example, the machine learning architecture can be or can include one or more of the following: linear regression, logistic regression, random forest, naive bayes classification, nearest neighbor, neural network, convolutional neural network, generative adversarial network, support vector machine, or gradient boosting algorithm, etc. As used herein, the term "training" also means learning, is a broad term and will be given its ordinary and customary meaning to a person of ordinary skill in the art and is not limited to a special or customized meaning. This term can specifically refer to, but is not limited to, the process of building a skin classification model, particularly determining and / or updating the parameters of the skin classification model. The skin classification model can be at least partially data-driven. For example, the skin classification model can be based on experimental data, such as data determined by illuminating multiple people and artificial objects such as masks and recording the reflection patterns. For example, the training can include using at least one training dataset, where the training dataset includes images of multiple people and artificial objects with known material properties, particularly the second image.

[0140] The skin detection step may include using at least one 2D face and facial landmark detection algorithm, which is configured to provide at least two positions of characteristic points of a human face. For example, the positions may be eye positions, forehead or cheeks. The 2D face and facial landmark detection algorithm may provide the positions of characteristic points of a human face, such as eye positions. Due to slight differences in reflections in different regions of the face (such as the forehead or cheeks), region-specific models can be trained. In the skin detection step, preferably at least one region-specific parametric skin classification model is used. The skin classification model may include multiple region-specific parametric skin classification models, for different regions, and / or region-specific data may be used to train the skin classification model, such as by filtering the images used for training. For example, for training, two different regions may be used, such as the eye-cheek region below the nose, and especially in cases where insufficient reflection features can be identified within this region, the region of the forehead may be used. However, other regions are also possible.

[0141] If the material properties correspond to at least one property characteristic of the skin, the detected face is characterized as skin. The processing unit may be configured to identify reflection features generated by illuminating biological tissue, especially skin, when its corresponding material properties meet at least one predetermined or predefined criterion. In the case where the material properties indicate "human skin", the reflection features may be identified as being generated by human skin. If the material properties are within at least one threshold and / or at least one range, the reflection features may be identified as being generated by human skin. The at least one threshold and / or range may be stored in a table or look-up table, and may be determined empirically, for example, and may be stored in at least one data storage device of the processing unit as an example. The processing unit is configured to otherwise identify the reflection features as background. Thus, the processing unit may be configured to assign material properties, such as skin yes or no, to each projection point.

[0142] The 3D detection step may be performed after the skin detection step and / or the face detection step. However, other embodiments are also possible, where the 3D detection step is performed before the skin detection step and / or the face detection step. After determining the longitudinal coordinate z in step d), the material properties can be determined by subsequently evaluating such that information about the longitudinal coordinate z can be considered for evaluating

[0143] The 3D detection step includes determining second beam profile information of at least four reflection features by analyzing the beam profiles of at least four reflection features within an image region of a second image corresponding to an image region of a first image including the identified geometric features. The 3D detection step may include determining the second beam profile information of them by analyzing the beam profiles of each of the at least four reflection features. The second beam profile information may include a quotient Q of the area of the beam profile.

[0144] As used herein, the term "analysis of the beam profile" generally may refer to the evaluation of the beam profile and may include at least one mathematical operation and / or at least one comparison and / or at least symmetrization and / or at least one filtering and / or at least one normalization. For example, the analysis of the beam profile may include at least one of a histogram analysis step, the calculation of a difference measurement, the application of a neural network, the application of a machine learning algorithm. The processing unit may be configured to symmetrize and / or normalize and / or filter the beam profile, in particular to remove noise or asymmetry from recordings at large angles, recording edges, etc. The processing unit may filter the beam profile by removing high spatial frequencies (e.g., by spatial frequency analysis and / or median filtering, etc.). Summation may be performed by the intensity center of the light spot and averaging all intensities at the same distance from the center. The processing unit may be configured to normalize the beam profile to the maximum intensity, in particular taking into account the intensity differences due to the recorded distance. The processing unit may be configured to remove the influence of background light from the beam profile, for example by imaging without illumination.

[0145] The processing unit may be configured to determine at least one longitudinal coordinate z of a corresponding reflection feature by analyzing the beam profile of the reflection feature within an image region of a second image corresponding to an image region of a first image including the identified geometric features. DPR The processing unit may be configured to determine the longitudinal coordinate z of the reflection feature by using a so-called photon depth ratio technique (also denoted as beam profile analysis). DPR Reference is made to WO 2018 / 091649A1, WO 2018 / 091638A1 and WO 2018 / 091640A1 regarding the photon depth ratio (DPR) technique, the entire contents of which are incorporated herein by reference.

[0146] The processing unit may be configured to determine at least one first region and at least one second region of the reflected beam profile of each reflection feature and / or the reflection features in at least one region of interest. The processing unit is configured to integrate the first region with the second region.

[0147] Analysis of a beam profile, which is one of the reflection characteristics, may include determining at least one first region and at least one second region of the beam profile. The first region of the beam profile may be region A1 and the second region of the beam profile may be region A2. The processing unit may be configured to integrate the first region and the second region. The processing unit may be configured to derive a combined signal, in particular a quotient Q, by one or more of the following: dividing the integrated first region by the integrated second region, dividing the integrated first region by a multiple of the integrated second region, dividing the integrated first region by a linear combination of the integrated first region and the integrated second region. The processing unit may be configured to determine at least two regions of the beam profile and / or divide the beam profile into at least two segments including different regions of the beam profile, where overlap of the regions is possible as long as the regions are not identical. For example, the processing unit may be configured to determine a plurality of regions, such as two, three, four, five or up to ten regions. The processing unit may be configured to divide the light spot into at least two regions of the beam profile and / or divide the beam profile into at least two segments including different regions of the beam profile. The processing unit may be configured to determine the integral of the beam profile over the respective regions for at least two regions. The processing unit may be configured to compare at least two of the determined integrals. Specifically, the processing unit may be configured to determine at least one first region and at least one second region of the beam profile. As used herein, the term "region of the beam profile" generally refers to any region of the beam profile at the location of the optical sensor for determining the quotient Q. The first region of the beam profile and the second region of the beam profile may be one or both of adjacent or overlapping regions. The first region of the beam profile and the second region of the beam profile may differ in area. For example, the processing unit may be configured to divide the sensor area of a CMOS sensor into at least two sub-regions, where the processing unit may be configured to divide the sensor area of the CMOS sensor into at least one left portion and at least one right portion and / or at least one upper portion and at least one lower portion and / or at least one inner portion and at least one outer portion. Additionally or alternatively, the camera may include at least two optical sensors, where the photosensitive regions of the first optical sensor and the second optical sensor may be arranged such that the first optical sensor is adapted to determine the first region of the beam profile of the reflection characteristic and the second optical sensor is adapted to determine the second region of the beam profile of the reflection characteristic. The processing unit may be adapted to integrate the first region and the second region. The processing unit may be configured to determine a longitudinal coordinate using at least one predetermined relationship between the quotient Q and the longitudinal coordinate. The predetermined relationship may be one or more of an empirical relationship, a semi-empirical relationship, and an analytically derived relationship. The processing unit may include at least one data storage device for storing the predetermined relationship, such as a look-up list or a look-up table.

[0148] The first region of the beam profile may substantially include the edge information of the beam profile, and the second region of the beam profile may substantially include the center information of the beam profile, and / or the first region of the beam profile may substantially include the information about the left portion of the beam profile, and the second region of the beam profile substantially includes the information about the right portion of the beam profile. The beam profile may have a center, i.e., the maximum value of the beam profile and / or the center point of the plateau of the beam profile and / or the geometric center of the spot, and a descending edge extending from the center. The second region may include the inner region of the cross-section and the first region may include the outer region of the cross-section. As used herein, the term "substantially center information" generally refers to a lower proportion of edge information (i.e., the proportion of the intensity distribution corresponding to the edge) compared to the proportion of center information (i.e., the proportion of the intensity distribution corresponding to the center). Preferably, the center information has a proportion of less than 10% of the edge information, more preferably less than 5%, and most preferably the center information does not include edge content. As used herein, the term "substantially edge information" generally refers to a lower proportion of center information compared to the proportion of edge information. The edge information may include the information of the entire beam profile, especially the information from the center and edge regions. The edge information may have a proportion of less than 10% of the center information, preferably less than 5%, and more preferably the edge information does not include center content. If at least one region of the beam profile is close to or around the center and substantially includes the center information, it may be determined and / or selected as the second region of the beam profile. If at least one region of the beam profile includes at least a part of the descending edge of the cross-section, it may be determined and / or selected as the first region of the beam profile. For example, the entire region of the cross-section may be determined as the first region.

[0149] Other selections of the first region A1 and the second region A2 are also feasible. For example, the first region may include the substantially outer region of the beam profile, and the second region may include the substantially inner region of the beam profile. For example, in the case of a two-dimensional beam profile, the beam profile may be divided into a left portion and a right portion, where the first region may substantially include the region of the left portion of the beam profile, and the second region may substantially include the region of the right portion of the beam profile.

[0150] The edge information may include information related to the number of photons in a first region of the beam profile, and the center information may include information related to the number of photons in a second region of the beam profile. The processing unit may be configured to determine the area integral of the beam profile. The processing unit may be configured to determine the edge information by integrating and / or summing the first region. The processing unit may be configured to determine the center information by integrating and / or summing the second region. For example, the beam profile may be a trapezoidal beam profile, and the processing unit may be configured to determine the integral of the trapezoid. Additionally, when a trapezoidal beam profile can be assumed, the determination of the edge and center signals may be replaced by an equivalent evaluation that utilizes the properties of the trapezoidal beam profile, such as the determination of the slope and position of the edge and the height of the central plateau, and the edge and center signals are derived through geometric considerations.

[0151] In one embodiment, A1 may correspond to the entire or complete region of a feature point on the optical sensor. A2 may be the central region of the feature point on the optical sensor. The central region may be a constant value. The central region may be smaller compared to the entire region of the feature point. For example, in the case of a circular feature point, the radius of the central region may be from 0.1 to 0.9 of the full radius of the feature point, preferably from 0.4 to 0.6 of the full radius.

[0152] In one embodiment, the illumination pattern may include at least one line pattern. A1 may correspond to the region of the full line width of the line pattern on the optical sensor, particularly on the photosensitive region of the optical sensor. The line pattern on the optical sensor may be widened and / or shifted compared to the line pattern of the illumination pattern, such that the line width on the optical sensor increases. Specifically, in the case of an optical sensor matrix, the line width of the line pattern on the optical sensor may vary from one column to another. A2 may be the central region of the line pattern on the optical sensor. The line width of the central region may be a constant value and particularly may correspond to the line width in the illumination pattern. The central region may have a smaller line width compared to the entire line width. For example, the line width of the central region may be from 0.1 to 0.9 of the full line width, preferably from 0.4 to 0.6 of the full line width. The line pattern may be segmented on the optical sensor. Each column of the optical sensor matrix may include central information of the intensity in the central region of the line pattern and edge information of the intensity from the region that extends further outwards from the central region of the line pattern to the edge region.

[0153] In one embodiment, the illumination pattern may include at least a dot pattern. A1 may correspond to the region of the full radius of the dot of the dot pattern on the optical sensor. A2 may be the central region of the dot in the dot pattern on the optical sensor. The central region may be a constant value. The central region may have a radius compared to the full radius. For example, the central region may have a radius from 0.1 to 0.9 of the full radius, preferably from 0.4 to 0.6 of the full radius.

[0154] The illumination pattern may include both at least one dot pattern and at least one line pattern. Other embodiments are possible in addition to or in place of the line pattern and the dot pattern.

[0155] The processing unit may be configured to derive a quotient Q by one or more of the following: dividing the integrated first region and the integrated second region, dividing a multiple of the integrated first region and the integrated second region, dividing a linear combination of the integrated first region and the integrated second region.

[0156] The processing unit may be configured to derive a quotient Q by one or more of the following: dividing the first region and the second region, dividing a multiple of the first region and the second region, dividing a linear combination of the first region and the second region. The processing unit may be configured to derive the quotient Q by:

[0157]

[0158] where x and y are lateral coordinates, A1 and A2 are the first and second regions of the beam profile respectively, and E(x, y) represents the beam profile.

[0159] Additionally or alternatively, the processing unit may be adapted to determine one or both of center information and edge information from at least one slice or cut of the light spot. This may be achieved, for example, by replacing the region integration in the quotient Q with a line integral along the slice or cut. To improve the accuracy, multiple slices or cuts through the light spot may be used and averaged. In the case of an elliptical light spot profile, averaging multiple slices or cuts may result in improved distance information.

[0160] For example, in the case where the optical sensor has a pixel matrix, the processing unit may be configured to evaluate the beam profile by:

[0161] - determining the pixels having the highest sensor signal and forming at least one center signal;

[0162] - evaluating the sensor signals of the matrix and forming at least one sum signal;

[0163] - determining the quotient Q by combining the center signal and the sum signal; and

[0164] - determining at least one longitudinal coordinate z of the object by evaluating the quotient Q.

[0165] As used herein, "sensor signal" generally refers to a signal generated by an optical sensor and / or at least one pixel of an optical sensor in response to illumination. Specifically, the sensor signal can be or can include at least one electrical signal, such as at least one analog electrical signal and / or at least one digital electrical signal. More specifically, the sensor signal can be or can include at least one voltage signal and / or at least one current signal. More specifically, the sensor signal can include at least one photocurrent. Additionally, the raw sensor signal can be used, or a display device, optical sensor, or any other element can be adapted to process or preprocess the sensor signal to generate an auxiliary sensor signal, which can also be used as a sensor signal, such as by preprocessing through filtering, etc. The term "central signal" generally refers to at least one sensor signal that substantially includes the central information of the beam profile. As used herein, the term "highest sensor signal" refers to a local maximum or one of the maxima or both in a region of interest. For example, the central signal can be the signal of a pixel having the highest sensor signal among a plurality of sensor signals generated by pixels of an entire matrix or a region of interest within the matrix, where the region of interest can be predetermined or determinable within an image generated by the pixels of the matrix. The central signal can come from a single pixel or a group of optical sensors, and in the latter case, as an example, the sensor signals of a group of pixels can be added, integrated, or averaged to determine the central signal. The group of pixels generating the central signal can be a group of adjacent pixels, such as pixels located at a distance less than a predetermined distance from the actual pixel having the highest sensor signal, or can be a group of pixels generating sensor signals within a predetermined range from the highest sensor signal. A group of pixels generating the central signal can be selected as large as possible to allow for the maximum dynamic range. The processing unit can be adapted to determine the central signal by integrating a plurality of sensor signals (such as a plurality of pixels around the pixel having the highest sensor signal). For example, the beam profile can be a trapezoidal beam profile, and the processing unit can be adapted to determine the integral of the trapezoid, particularly the integral of the plateau of the trapezoid.

[0166] As described above, the center signal can typically be a single sensor signal, such as a sensor signal from a pixel at the center of a light spot, or can be a combination of multiple sensor signals, such as a combination of sensor signals from pixels at the center of a light spot, or an auxiliary sensor signal derived by processing sensor signals derived from one or more of the above possibilities. The determination of the center signal can be performed electronically, since the comparison of sensor signals is quite simply achieved by conventional electronic devices, or can be performed entirely or in part by software. Specifically, the center signal can be selected from the group including the following: the highest sensor signal; the average value of a group of sensor signals within a predetermined tolerance range from the highest sensor signal; the average value of sensor signals from a group of pixels including the pixel with the highest sensor signal and a predetermined group of adjacent pixels; the sum of sensor signals from a group of pixels including the pixel with the highest sensor signal and a predetermined group of adjacent pixels; the sum of a group of sensor signals within a predetermined tolerance range from the highest sensor signal; the average value of a group of sensor signals above a predetermined threshold; the sum of a group of sensor signals above a predetermined threshold; the integral of sensor signals from a group of optical sensors including the optical sensor with the highest sensor signal and a predetermined group of adjacent pixels; the integral of a group of sensor signals within a predetermined tolerance range from the highest sensor signal; the integral of a group of sensor signals above a predetermined threshold.

[0167] Similarly, the term "sum signal" generally refers to a signal that substantially includes edge information of the beam profile. For example, the sum signal can be derived by adding sensor signals, integrating sensor signals, or averaging sensor signals over the entire matrix or a region of interest within the matrix, where the region of interest can be predetermined or determinable within an image generated by the optical sensors of the matrix. When adding, integrating, or averaging sensor signals, the actual optical sensors generating the sensor signals can be excluded from the addition, integration, or averaging, or alternatively, can be included in the addition, integration, or averaging. The processing unit can be adapted to determine the sum signal by integrating the signals over the entire matrix or a region of interest within the matrix. For example, the beam profile can be a trapezoidal beam profile and the processing unit can be adapted to determine the integral of the entire trapezoid. Additionally, when a trapezoidal beam profile can be assumed, the determination of the edge and center signals can be replaced by an equivalent evaluation that utilizes the properties of the trapezoidal beam profile, such as the determination of the slope and position of the edge and the height of the central platform, and the edge and center signals are derived through geometric considerations.

[0168] Similarly, the center signal and the edge signal can also be determined by using segments of the beam profile (e.g., circular segments of the beam profile). For example, the beam profile can be divided into two segments by a secant or chord that does not pass through the center of the beam profile. Thus, one segment will substantially contain edge information, while the other segment will substantially contain center information. For example, to further reduce the amount of edge information in the center signal, the edge signal can also be subtracted from the center signal.

[0169] The quotient Q can be a signal generated by combining the center signal and the sum signal. Specifically, the determination can include one or more of the following: forming the quotient of the center signal and the sum signal, or vice versa; forming the quotient of a multiple of the center signal and a multiple of the sum signal, or vice versa; forming the quotient of a linear combination of the center signal and a linear combination of the sum signal, or vice versa. Additionally or alternatively, the quotient Q can include any signal or combination of signals that contains at least one piece of information regarding the comparison between the center signal and the sum signal.

[0170] As used herein, the term "longitudinal coordinate of the reflection feature" refers to the distance between the optical sensor and the point in the scene that emits the corresponding illumination feature. The processing unit can be configured to determine the longitudinal coordinate by using at least one predetermined relationship between the quotient Q and the longitudinal coordinate. The predetermined relationship can be one or more of an empirical relationship, a semi-empirical relationship, and an analytically derived relationship. The processing unit can include at least one data storage device for storing the predetermined relationship, such as a look-up list or a look-up table.

[0171] The processing unit can be configured to perform at least one photon depth ratio algorithm that calculates the distances of all reflection features having zero order and higher orders.

[0172] The 3D detection step can include determining at least one depth level from the second beam profile information of the reflection feature by using the processing unit.

[0173] The processing unit may be configured to determine a depth map of at least a portion of a scene by determining at least one depth information of a reflection feature located within an image region of a second image corresponding to an image region of a first image including the identified geometric feature. As used herein, the term "depth" or depth information may refer to the distance between an object and an optical sensor and may be given by a longitudinal coordinate. As used herein, the term "depth map" may refer to the spatial distribution of depth. The processing unit may be configured to determine the depth information of the reflection feature by one or more of the following techniques: photon depth ratio, structured light, beam profile analysis, time-of-flight, shape-from-motion, depth-from-focus, triangulation, depth-from-defocus, stereo sensors. The depth map may be a sparsely populated depth map including several entries. Alternatively, the depth may be crowded, including a large number of entries.

[0174] If the depth level deviates from a predetermined or predefined depth level of a planar object, the detected face is characterized as a 3D object. Step c) may include using 3D topology data of a face in front of the camera. The method may include determining a curvature based on at least four reflection features located within an image region of a second image corresponding to an image region of a first image including the identified geometric feature. The method may include comparing the curvature determined from the at least four reflection features with a predetermined or predefined depth level of a planar object. If the curvature exceeds the assumed curvature of the planar object, the detected face may be characterized as a 3D object, otherwise it may be characterized as a planar object. The predetermined or predefined depth level of the planar object may be stored in at least one data memory of the processing unit, such as a lookup list or a lookup table. The predetermined or predefined level of the planar object may be determined experimentally and / or may be a theoretical level of the planar object. The predetermined or predefined depth level of the planar object may be at least one limit of at least one curvature and / or a range of at least one curvature.

[0175] The 3D features determined in step c) may allow for distinguishing between high-quality photos and 3D face-like structures. The combination of steps b) and c) may allow for enhancing the reliability of authentication against attacks. The 3D features may be combined with material features to increase the security level. Since the same computational pipeline can be used to generate the input data for skin classification and the generation of the 3D point cloud, these two properties can be computed from the same frame with a lower computational load.

[0176] Preferably, after steps a) to c), an authentication step may be performed. The authentication step may be partially performed after each of steps a) to c). If no face is detected in step a) and / or if it is determined in step b) that the reflection feature is not generated by the skin and / or if the depth map in step c) relates to a planar object, the authentication may be aborted. The authentication step includes authenticating the detected face by using at least one authentication unit if the face detected in step b) is characterized as skin and the face detected in step c) is characterized as a 3D object.

[0177] Steps a) to d) may be performed by using at least one device, such as at least one mobile device, such as a mobile phone, a smart phone, etc., where access to the device is protected by using face authentication. Other devices are also possible, such as access control devices that control access to buildings, machines, cars, etc. The method may include allowing access to the device if the detected face is authenticated.

[0178] The method may include at least one enrollment step. In the enrollment step, a user of the device may be enrolled. As used herein, the term "enroll" may refer to the process of registering and / or signing up and / or teaching a user for subsequent use of the device. Generally, enrollment may be performed at the first use of the device and / or when the device is started up. However, embodiments are feasible where multiple users may be enrolled, for example, continuously, such that enrollment may be performed and / or repeated at any time during the use of the device. Enrollment may include generating a user account and / or a user profile. Enrollment may include inputting and storing user data, particularly image data, via at least one user interface. Specifically, at least one 2D image of the user is stored in at least one database. The enrollment step may include imaging at least one image of the user, particularly a plurality of images. Images may be recorded from different directions and / or the user may change his orientation. Additionally, the enrollment step may include generating at least one 3D image and / or depth map of the user, which may be used for comparison in step d). The database may be a database of the device, such as a database of a processing unit, and / or may be an external database such as a cloud. The method includes identifying the user by comparing the 2D image of the user with a first image. The method according to the present invention may allow a significant improvement in the presentation attack detection ability of biometric authentication methods. To improve overall authentication, in addition to the 2D image of the user, a personal specific material fingerprint and 3D topological features may also be stored during the enrollment process. This may allow multi-factor authentication within a device by using 2D, 3D, and material-derived features.

[0179] The method using beam profile analysis technology according to the present invention can provide a concept for reliably detecting human skin by analyzing the reflection of laser spots on the face (especially in the NIR range) and differentiating it from the reflection from the attack material generated by mimicking the face. In addition, beam profile analysis provides depth information simultaneously by analyzing the same camera frame. Therefore, 3D and skin safety features can be provided by exactly the same technology.

[0180] Since a 2D image of the face can also be recorded simply by turning off the laser illumination, a completely secure face recognition pipeline can be established to solve the above problems.

[0181] When the laser wavelength moves towards the NIR region, the reflection properties of human skin of different ethnic groups become more similar. At a wavelength of 940 nm, the difference is minimal. Therefore, different ethnic groups do not play a role in skin authentication.

[0182] Time-consuming analysis of a series of frames may not be required because attack detection (through skin classification) is provided by only one frame. The time frame for performing the complete method can be ≤500 ms, preferably ≤250 ms. However, embodiments in which multiple frames can be used to perform skin detection may be feasible. Depending on the confidence in identifying the reflection features in the second image and the speed of the method, the method can include sampling the reflection features over multiple frames to achieve more stable classification.

[0183] In addition to accuracy, execution speed and power consumption are also important requirements. For security reasons, further restrictions on the availability of computing resources can be introduced. For example, steps a) to d) can be run in a secure area of the processing unit to avoid any software-based manipulation during program execution. The compact nature of the above-mentioned material detection network can solve this problem by exhibiting excellent runtime behavior in the said secure area, while traditional PAD solutions require checking several consecutive frames, which results in a large computational cost and a long response time for the algorithm.

[0184] In another aspect of the present invention, a computer program for face authentication is configured to cause a computer or a computer network to fully or partially execute the method according to the present invention when executed on the computer or the computer network, wherein the computer program is configured to execute and / or perform at least steps a) to d) of the method according to the present invention. Specifically, the computer program can be stored on a computer-readable data carrier and / or a computer-readable storage medium.

[0185] As used herein, the terms "computer-readable data carrier" and "computer-readable storage medium" specifically may refer to a non-transitory data storage device, such as a hardware storage medium storing computer-executable instructions. The computer-readable data carrier or storage medium specifically may be or may include storage media such as random access memory (RAM) and / or read-only memory (ROM).

[0186] Accordingly, specifically, one, more than one or even all of the above method steps may be performed by using a computer or a computer network, preferably by using a computer program.

[0187] On the other hand, the computer-readable storage medium includes instructions which, when executed by a computer or a computer network, cause at least steps a) to d) of the method according to the invention to be performed.

[0188] Further disclosed and proposed herein is a data carrier having a data structure stored thereon, which, after being loaded into a computer or a computer network, such as into the working memory or main memory of a computer or a computer network, can perform the method according to one or more embodiments disclosed herein.

[0189] Also disclosed and proposed herein is a computer program product having program code means stored on a machine-readable carrier for performing the method according to one or more embodiments disclosed herein when the program is executed on a computer or a computer network. As used herein, a computer program product refers to a program as a tradable product. The product generally may exist in any format, such as in a paper format, or on a computer-readable data carrier and / or a computer-readable storage medium. Specifically, the computer program product may be distributed via a data network.

[0190] Finally, disclosed and proposed herein is a modulated data signal comprising instructions readable by a computer system or a computer network for performing the method according to one or more embodiments disclosed herein.

[0191] Referring to the computer-implemented aspects of the present invention, one or more method steps or even all method steps of the method according to one or more embodiments disclosed herein may be performed by using a computer or a computer network. Accordingly, generally, any method step including the provision and / or manipulation of data may be performed by using a computer or a computer network. Generally, these method steps may include any method steps, typically except for method steps that require manual work.

[0192] Specifically, further disclosed by the present invention is:[[]]END]]

[0193] - A computer or computer network, including at least one processor, wherein the processor is adapted to execute a method according to one of the embodiments described in this specification,

[0194] - A computer-loadable data structure, which is adapted to execute a method according to one of the embodiments described in this specification when the data structure is executed on a computer,

[0195] - A computer program, wherein the computer program is adapted to execute a method according to one of the embodiments described in this specification when the program is executed on a computer,

[0196] - A computer program, including program means for executing a method according to one of the embodiments described in this specification when the computer program is executed on a computer or a computer network,

[0197] - A computer program, including program means according to the foregoing embodiments, wherein the program means is stored on a computer-readable storage medium,

[0198] - A storage medium, wherein a data structure is stored on the storage medium, and wherein the data structure is adapted to execute a method according to one of the embodiments described in this specification after being loaded into the main memory and / or working memory of a computer or a computer network, and

[0199] - A computer program product having program code means, wherein the program code means can be stored or is stored on a storage medium and is used to execute a method according to one of the embodiments described in this specification if the program code means is executed on a computer or a computer network.

[0200] In another aspect, a mobile device including at least one camera, at least one lighting unit, and at least one processing unit is disclosed. The mobile device is configured to perform at least steps a) to c) of the face authentication method according to the present invention and optionally perform step d). Step d) can be performed by using at least one authentication unit. The authentication unit can be a unit of the mobile device or can be an external authentication unit. For the definitions and embodiments of the mobile device, reference is made to the definitions and embodiments described with respect to the method.

[0201] In another aspect of the present invention, for the purpose of biometric presentation attack detection, the use of the method according to the present invention is proposed, for example, a method according to one or more of the embodiments given above or further detailed below.

[0202] Generally speaking, in the context of the present invention, the following embodiments are considered to be preferred:

[0203] Embodiment 1. A method for face authentication, including the following steps:

[0204] a) At least one face detection step, wherein the face detection step includes: using at least one camera to determine at least one first image, wherein the first image includes at least one two-dimensional image of a scene suspected of including a face, and wherein the face detection step includes: using at least one processing unit to detect the face in the first image by identifying at least one predefined or predetermined geometric feature characteristic for the face in the first image;

[0205] b) At least one skin detection step, wherein the skin detection step includes: using at least one lighting unit to project at least one lighting pattern including a plurality of lighting features onto the scene, and using at least one camera to determine at least one second image, wherein the second image includes a plurality of reflection features generated by the scene in response to the lighting of the lighting features, wherein each of the reflection features includes at least one beam profile, and wherein the skin detection step includes: using the processing unit to determine first beam profile information of at least one of the reflection features by analyzing the beam profile of at least one of the reflection features located within an image region of the second image corresponding to the image region of the first image including the identified geometric feature, and determining at least one material property of the reflection feature from the first beam profile information, wherein if the material property corresponds to at least one property characteristic for skin, the detected face is characterized as skin;

[0206] c) At least one 3D detection step, wherein the 3D detection step includes: using the processing unit to determine second beam profile information of at least four of the reflection features by analyzing the beam profiles of at least four of the reflection features located within the image region of the second image corresponding to the image region of the first image including the identified geometric feature, and determining at least one depth level from the second beam profile information of the reflection features, wherein if the depth level deviates from a predetermined or predefined depth level of a planar object, the detected face is characterized as a 3D object;

[0207] d) At least one authentication step, wherein the authentication step includes: if the face detected in step b) is characterized as skin and the face detected in step c) is characterized as a 3D object, authenticating the detected face by using at least one authentication unit.

[0208] Example 2. The method according to the preceding example, wherein steps a) to d) are performed by using at least one device, wherein access to the device is protected by using facial authentication, and wherein the method comprises: allowing access to the device if the detected face is authenticated.

[0209] Example 3. The method according to the preceding example, wherein the method comprises at least one registration step, wherein, in the registration step, the user of the device is registered, wherein at least one 2D image of the user is stored in at least one database, and wherein the method comprises: identifying the user by comparing the 2D image of the user with the first image.

[0210] Example 4. The method according to any one of the preceding examples, wherein, in the skin detection step, at least one parameterized skin classification model is used, and wherein the parameterized skin classification model is configured to classify skin and other materials by using the second image as input.

[0211] Example 5. The method according to the preceding example, wherein the skin classification model is parameterized by using machine learning, and wherein the property characteristics for the skin are determined by applying an optimization algorithm in terms of at least one optimization objective to the skin classification model.

[0212] Example 6. The method according to any one of the preceding two examples, wherein the skin detection step comprises: using at least one 2D face and facial landmark detection algorithm, the 2D face and facial landmark detection algorithm being configured to provide at least two positions of characteristic points of a human face, and wherein, in the skin detection step, at least one region-specific parameterized skin classification model is used.

[0213] Example 7. The method according to any one of the preceding examples, wherein the illumination pattern comprises a periodic grid of laser spots.

[0214] Example 8. The method according to any one of the preceding examples, wherein the illumination feature has a wavelength in the near-infrared (NIR) range.

[0215] Example 9. The method according to the preceding example, wherein the illumination feature has a wavelength of 940 nm.

[0216] Example 10. The method according to any one of the preceding examples, wherein a plurality of second images are determined, and wherein the reflection features of the plurality of second images are used for skin detection in step b) and / or for 3D detection in step c).

[0217] Example 11. The method according to any one of the preceding examples, wherein the camera is or comprises at least one near-infrared camera.

[0218] Example 12. A computer program for face authentication, which is configured to cause the computer or the computer network to fully or partially execute the method according to any one of the preceding examples when executed on the computer or the computer network, wherein the computer program is configured to execute and / or implement at least steps a) to d) of the method according to any one of the preceding examples.

[0219] Example 13. A computer-readable storage medium comprising instructions which, when executed by a computer or a computer network, cause at least steps a) to d) of the method according to any one of the preceding examples relating to the method to be executed.

[0220] Example 14. A mobile device comprising at least one camera, at least one lighting unit and at least one processing unit, the mobile device being configured to at least execute steps a) to c) of the method for face authentication according to any one of the preceding examples relating to the method, and optionally step d).

[0221] Example 15. Use of the method according to any one of the preceding examples for biometric presentation attack detection. Description of the Drawings

[0222] Further optional details and features of the invention are apparent from the description of the preferred exemplary embodiments in conjunction with the dependent claims. Here, the specific features can be implemented in isolation or in combination with other features. The invention is not limited to the exemplary embodiments. The exemplary embodiments are schematically illustrated in the drawings. The same reference numerals in the various drawings denote the same elements or elements having the same function, or elements corresponding to each other in terms of their function.

[0223] Specifically, in the figures:

[0224] Figure 1 An embodiment of the method for face authentication according to the invention is shown;

[0225] Figure 2 An embodiment of the mobile device according to the invention is shown; and

[0226] Figure 3 The experimental results are shown. Detailed Description

[0227] Figure 1A flowchart of a method for face authentication according to the present invention is shown. Face authentication may include verifying an identified object or a part of the identified object as a human face. Specifically, the authentication may include distinguishing a real human face from attack materials generated for mimicking the face. The authentication may include verifying the identity of the corresponding user and / or assigning an identity to the user. The authentication may include generating and / or providing identity information, such as to other devices, such as to at least one authorized device, for authorizing access to a mobile device, a machine, a vehicle, a building, etc. The identity information may be proven through authentication. For example, the identity information may be and / or may include at least one identity token. In the case of successful authentication, the identified object or the part of the identified object is verified as a real face and / or the identity of the object, especially the user, is verified.

[0228] The method includes the following steps:

[0229] a) (Reference numeral 110) At least one face detection step, wherein the face detection step includes determining at least one first image by using at least one camera 112, wherein the first image includes at least one two-dimensional image of a scene suspected of including a face, and wherein the face detection step includes detecting a face in the first image by using at least one processing unit 114 to identify at least one predefined or predetermined geometric feature characteristic of the face;

[0230] b) (Reference numeral 116) At least one skin detection step, wherein the skin detection step includes projecting at least one illumination pattern including a plurality of illumination features onto the scene by using at least one illumination unit 118 and using at least one camera 112 to determine at least one second image, wherein the second image includes a plurality of reflection features generated by the scene in response to the illumination of the illumination features, wherein each reflection feature includes at least one beam profile, and wherein the skin detection step includes determining first beam profile information of at least one reflection feature by analyzing at least one reflection feature of the reflection features located within an image region of the second image, the image region corresponding to the image region of the first image including the identified geometric feature, and determining at least one material property of the reflection feature from the first beam profile information by using a processing unit 114, wherein if the material property corresponds to at least one property characteristic of skin, the detected face is characterized as skin;

[0231] c) (reference numeral 120) at least one 3D detection step, wherein the 3D detection step determines second beam profile information of at least four reflective features by analyzing beam profiles of at least four reflective features located within an image region of the second image, the image region corresponding to an image region of the first image including the identified geometrical feature, wherein the detected face is characterized as a 3D object if the depth level deviates from a predetermined or predefined depth level of a planar object;

[0232] d) (reference numeral 122) at least one authentication step, wherein the authentication step comprises authenticating the detected face by using at least one authentication unit if the face detected in step b) is characterized as skin and the face detected in step c) is characterized as a 3D object.

[0233] The method steps may be performed in a given order or in a different order. In addition, there may be one or more additional method steps not listed. Further, one, more or even all of the method steps may be repeated.

[0234] The camera 112 may include at least one imaging element configured to record or capture spatially resolved one-dimensional, two-dimensional or even three-dimensional optical data or information. The camera 112 may be a digital camera. As an example, the camera 112 may include at least one camera chip, such as at least one CCD chip and / or at least one CMOS chip configured to record an image. The camera 112 may be or may include at least one near-infrared camera. The image may relate to data recorded using the camera 112, such as a plurality of electronic readings from an imaging device, such as pixels of a camera chip. In addition to at least one camera chip or imaging chip, the camera 112 may also include additional elements, such as one or more optical elements, such as one or more lenses. As an example, the camera 112 may be a fixed-focus camera having at least one lens that is fixedly adjusted relative to the camera. However, alternatively, the camera 112 may also include one or more variable lenses that can be adjusted automatically or manually.

[0235] The camera 112 may be a camera of a mobile device 124, such as a laptop, a tablet or in particular a cellular phone such as a smartphone. Thus, in particular, the camera 112 may be part of a mobile device 124, which in addition to at least one camera 112 also includes one or more data processing devices, such as one or more data processors. However, other cameras are also possible. The mobile device 124 may be a mobile electronic device, more particularly a mobile communication device such as a cellular phone or a smartphone. Additionally or alternatively, the mobile device 124 may also refer to a tablet computer or another type of portable computer.Figure 2 An embodiment of a mobile device according to the present invention is shown.

[0236] Specifically, the camera 112 may be or may include at least one optical sensor 126 having at least one photosensitive area. The optical sensor 126 may specifically be or may include at least one photodetector, preferably an inorganic photodetector, more preferably an inorganic semiconductor photodetector, and most preferably a silicon photodetector. Specifically, the optical sensor 126 may be sensitive in the infrared spectral range. The optical sensor 126 may include at least one sensor element that includes a pixel matrix. All pixels of the matrix or at least one set of optical sensors of the matrix may specifically be the same. Specifically, a set of the same pixels of the matrix may be provided for different spectral ranges, or all pixels may be the same in terms of spectral sensitivity. In addition, the size and / or their electronic or optoelectronic properties of the pixels may be the same. Specifically, the optical sensor 126 may be or may include at least one array of inorganic photodiodes that is sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 microns. Specifically, the optical sensor 126 may be sensitive in a part of the near-infrared region where silicon photodiodes are applicable, specifically in the range of 700 nm to 1100 nm. Infrared optical sensors that can be used for the optical sensor may be commercially available infrared optical sensors, such as the infrared optical sensors commercially available from trinamiX GmbH, D-67056 Ludwigshafen am Rhein, Germany, under the Hertzstueck brand. Thus, by way of example, the optical sensor 126 may include at least one intrinsic photovoltaic type optical sensor, more preferably at least one semiconductor photodiode selected from the group consisting of: Ge photodiode, InGaAs photodiode, extended InGaAs photodiode, InAs photodiode, InSb photodiode, HgCdTe photodiode. Additionally or alternatively, the optical sensor may include at least one extrinsic photovoltaic type optical sensor, more preferably at least one semiconductor photodiode selected from the group consisting of: Ge:Au photodiode, Ge:Hg photodiode, Ge:Cu photodiode, Ge:Zn photodiode, Si:Ga photodiode, Si:As photodiode. Additionally or alternatively, the optical sensor 126 may include at least one photoconductive sensor, such as a PbS or PbSe sensor, bolometer, preferably selected from the group of VO bolometers and amorphous Si bolometers. TM GmbH under the Hertzstueck TM brand. Thus, by way of example, the optical sensor 126 may include at least one intrinsic photovoltaic type optical sensor, more preferably at least one semiconductor photodiode selected from the group consisting of: Ge photodiode, InGaAs photodiode, extended InGaAs photodiode, InAs photodiode, InSb photodiode, HgCdTe photodiode. Additionally or alternatively, the optical sensor may include at least one extrinsic photovoltaic type optical sensor, more preferably at least one semiconductor photodiode selected from the group consisting of: Ge:Au photodiode, Ge:Hg photodiode, Ge:Cu photodiode, Ge:Zn photodiode, Si:Ga photodiode, Si:As photodiode. Additionally or alternatively, the optical sensor 126 may include at least one photoconductive sensor, such as a PbS or PbSe sensor, bolometer, preferably selected from the group of VO bolometers and amorphous Si bolometers.

[0237] Specifically, the optical sensor 126 can be sensitive in the near-infrared region. Specifically, the optical sensor 126 can be sensitive in a part of the near-infrared region applicable to silicon photodiodes, specifically in the range of 700 nm to 1000 nm. Specifically, the optical sensor 126 can be sensitive in the infrared spectral range, specifically in the range of 780 nm to 3.0 microns. For example, the optical sensor 126 can be or can include at least one element selected from the group consisting of: a CCD sensor element, a CMOS sensor element, a photodiode, a photocell, a photoconductor, a phototransistor, or any combination thereof. Any other type of photosensitive element can be used. The photosensitive element can generally be made entirely or partially of inorganic materials and / or can be made entirely or partially of organic materials. Most commonly, one or more photodiodes can be used, such as commercially available photodiodes, such as inorganic semiconductor photodiodes.

[0238] The camera 112 may further include at least one transfer device (not shown here). The camera 112 may include at least one optical element selected from the group consisting of: a transfer device, such as at least one lens and / or at least one lens system, at least one diffractive optical element. The transfer device may be adapted to direct a light beam onto the optical sensor 126. Specifically, the transfer device may include one or more of the following: at least one lens, such as at least one lens selected from the group consisting of: at least one focus-adjustable lens, at least one aspherical lens, at least one spherical lens, at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflection element, preferably at least one mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitter; at least one multi-lens system. The transfer device can have a focal length. Thus, the focal length constitutes a measure of the ability of the transfer device to converge an incident light beam. Therefore, the transfer device may include one or more imaging elements, which may have the effect of a converging lens. For example, the transfer device may have one or more lenses, particularly one or more refractive lenses, and / or one or more convex mirrors. In this example, the focal length can be defined as the distance from the center of a thin refractive lens to the principal focus of the thin lens. For a converging thin refractive lens, such as a convex or biconvex thin lens, the focal length can be considered positive and can provide the distance at which a collimated light beam illuminating the thin lens acting as the transfer device can be focused into a single spot. Additionally, the transfer device may include at least one wavelength selection element, such as at least one filter. Additionally, the transfer device can be designed to impose a predefined beam profile on the electromagnetic radiation (e.g., at the sensor area and particularly at the location of the sensor area). In principle, the above optional embodiments of the transfer device can be implemented individually or in any desired combination.

[0239] The transfer device may have an optical axis. The transfer device may form a coordinate system, where the longitudinal coordinate is the coordinate along the optical axis, and where d is the spatial offset relative to the optical axis. The coordinate system may be a polar coordinate system, where the optical axis of the transfer device forms the z-axis, and where the distance from the z-axis and the polar angle may be used as additional coordinates. A direction parallel or antiparallel to the z-axis may be considered the longitudinal direction, and the coordinate along the z-axis may be considered the longitudinal coordinate. Any direction perpendicular to the z-axis may be considered the transverse direction, and the polar coordinates and / or the polar angle may be considered the transverse coordinates.

[0240] The camera 112 is configured to determine at least one image of the scene, in particular a first image. The scene may refer to a spatial region. The scene may include a face in authentication and the surrounding environment. The first image itself may include pixels, and the pixels of the image are related to the pixels of the matrix of sensor elements. The first image is at least one two-dimensional image that has information about transverse coordinates, such as dimensions of height and width.

[0241] The face detection step 110 includes detecting a face in the first image by identifying at least one predefined or predetermined geometric feature characteristic of the face in the first image by using at least one processing unit 114. As an example, the at least one processing unit 114 may include software code stored thereon, which includes a plurality of computer commands. The processing unit 114 may provide one or more hardware elements for performing one or more specified operations and / or may provide one or more processors on which software for performing one or more specified operations runs. Operations including evaluating the image may be performed by the at least one processing unit 114. Thus, as an example, one or more instructions may be implemented in software and / or hardware. Thus, as an example, the processing unit 114 may include one or more programmable devices, such as one or more computers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), or field-programmable gate arrays (FPGAs), which are configured to perform the above evaluation. However, additionally or alternatively, the processing unit may also be implemented fully or partially by hardware. The processing unit 114 and the camera 112 may be fully or partially integrated into a single device. Thus, generally, the processing unit 114 may also form part of the camera 112. Alternatively, the processing unit 114 and the camera 112 may be fully or partially embodied as separate devices.

[0242] Detecting a face in a first image can include identifying at least one predefined or predetermined geometric feature characteristic of the face. The geometric feature characteristic of the face can be at least one geometry-based feature that describes the shape of the face and its components, particularly one or more of the nose, eyes, mouth, or eyebrows. The processing unit 114 can include at least one database in which the geometric feature characteristics of the face are stored, such as in a look-up table. Techniques for identifying at least one predefined or predetermined geometric feature characteristic of the face are generally known to those skilled in the art. For example, face detection can be performed as described in Masi, Lacopo et al., "Deep face recognition: A survey", 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), IEEE, 2018, the entire content of which is incorporated by reference.

[0243] The processing unit 114 can be configured to perform at least one image analysis and / or image processing to identify geometric features. The image analysis and / or image processing can use at least one feature detection algorithm. The image analysis and / or image processing can include one or more of the following: filtering; selecting at least one region of interest; background correction; decomposition into color channels; decomposition into hue, saturation, and / or brightness channels; frequency decomposition; singular value decomposition; applying a blob detector; applying a corner detector; applying the determinant of a Hessian filter; applying a region detector based on principal curvature; applying a histogram of gradient positions and orientations algorithm; applying a histogram of orientation gradient descriptors; applying an edge detector; applying a differential edge detector; applying a Canny edge detector; applying a Laplacian of Gaussian filter; applying a Difference of Gaussian filter; applying a Sobel operator; applying a Laplacian operator; applying a Scharr operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying a Radon transform; applying a Hough transform; applying a wavelet transform; thresholding; creating a binary image. The region of interest can be determined manually by a user or can be determined automatically, for example, by identifying features within the first image.

[0244] Specifically, after the face detection step 110, a skin detection step 116 can be performed, including projecting at least one illumination pattern including a plurality of illumination features onto the scene by using at least one illumination unit 118. However, embodiments in which the skin detection step 116 is performed before the face detection step 110 are feasible.

[0245] The illumination unit 118 may be configured to provide an illumination pattern for scene illumination. The illumination unit 118 may be adapted to illuminate the scene directly or indirectly, where the illumination pattern is affected by the surface of the scene, in particular reflected or scattered, and thereby at least partially directed towards the camera. The illumination unit 118 may be configured to illuminate the scene, for example, by directing a light beam onto the scene and reflecting the light beam. The illumination unit 118 may be configured to generate an illumination light beam for illuminating the scene.

[0246] The illumination unit 118 may include at least one light source. The illumination unit 118 may include a plurality of light sources. The illumination unit 118 may include an artificial light source, in particular at least one laser source and / or at least one incandescent lamp and / or at least one semiconductor light source, such as at least one light-emitting diode, in particular an organic and / or inorganic light-emitting diode. The illumination unit 118 may be configured to generate at least one illumination pattern in the infrared region. The illumination feature may have a wavelength in the near-infrared (NIR) range. The illumination feature may have a wavelength of approximately 940 nm. At this wavelength, melanin absorption is depleted, so that dark and bright complexes reflect light almost identically. However, other wavelengths in the NIR region are also possible, such as one or more of 805 nm, 830 nm, 835 nm, 850 nm, 905 nm or 980 nm. In addition, using light in the near-infrared region makes the light undetectable or only weakly detectable by the human eye, and still detectable by a silicon sensor, in particular a standard silicon sensor.

[0247] The illumination unit 118 can be or can include at least one multi-beam light source. For example, the illumination unit 118 can include at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the illumination unit 118 can include at least one laser and / or laser source. Various types of lasers can be employed, such as semiconductor lasers, double heterostructure lasers, external cavity lasers, separate confinement heterostructure lasers, quantum cascade lasers, distributed Bragg reflector lasers, polariton lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, bulk Bragg grating lasers, indium arsenide lasers, transistor lasers, diode-pumped lasers, distributed feedback lasers, quantum well lasers, interband cascade lasers, gallium arsenide lasers, semiconductor ring lasers, extended cavity diode lasers, or vertical cavity surface emitting lasers. Additionally or alternatively, non-laser light sources can be used, such as LEDs and / or light bulbs. The illumination unit 118 can include one or more diffractive optical elements (DOEs) adapted to generate an illumination pattern. For example, the illumination unit 118 can be adapted to generate and / or project a point cloud, such as the illumination unit 118 can include one or more of the following: at least one digital light processing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one light emitting diode array; at least one laser light source array. Due to its generally defined beam profile and other properties of operability, it is particularly preferred to use at least one laser source as the illumination unit 118. The illumination unit 118 can be integrated into the housing of the camera 112 or can be separated from the camera 112.

[0248] The illumination pattern includes at least one illumination feature adapted to illuminate at least a portion of the scene. The illumination pattern may include a single illumination feature. The illumination pattern may include multiple illumination features. The illumination pattern may be selected from the group consisting of: at least one dot pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; at least one pattern including an arrangement of periodic or aperiodic features. The illumination pattern may include regular and / or constant and / or periodic patterns, such as triangular patterns, rectangular patterns, hexagonal patterns, or patterns including additional convex tilings. The illumination pattern may exhibit at least one illumination feature selected from the group consisting of: at least one dot; at least one row; at least two lines, such as parallel lines or intersecting lines; at least one dot and one line; at least one arrangement of periodic or aperiodic features; at least one feature of any shape. The illumination pattern may include at least one pattern selected from the group consisting of: at least one dot pattern, particularly a pseudo-random dot pattern; a random dot pattern or a quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern including at least one known feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern, including at least one pattern with convex uniform tiling; at least one line pattern including at least one line; at least one line pattern including at least two lines (such as parallel lines or intersecting lines). For example, the illumination unit 118 may be adapted to generate and / or project a point cloud. The illumination unit 118 may include at least one light projector adapted to generate a point cloud such that the illumination pattern may include multiple dot patterns. The illumination pattern may include a periodic grid of laser dots. The illumination unit 118 may include at least one mask adapted to generate the illumination pattern from at least one light beam generated by the illumination unit 118.

[0249] The skin detection step 116 includes determining at least one second image, also referred to as a reflection image, using the camera 112. The method may include determining multiple second images. The reflection features of the multiple second images may be used for skin detection in step b) and / or for 3D detection in step c). The reflection features may be features in the image plane generated by the scene in response to illumination (specifically having at least one illumination feature). Each reflection feature includes at least one beam profile, also referred to as a reflected beam profile. The beam profile of a reflection feature generally may refer to at least one intensity distribution of the reflection feature, such as the intensity distribution of a spot on an optical sensor, as a function of pixels. The beam profile may be selected from the group consisting of: a trapezoidal beam profile; a triangular beam profile; a conical beam profile; and a linear combination of Gaussian beam profiles.

[0250] The evaluation of the second image may include identifying reflection features of the second image. The processing unit 114 may be configured to perform at least one image analysis and / or image processing to identify the reflection features. The image analysis and / or image processing may use at least one feature detection algorithm. The image analysis and / or image processing may include one or more of the following: filtering; selecting at least one region of interest; forming a difference image between the image created by the sensor signals and at least one offset; inverting the sensor signals by inverting the image created by the sensor signals; forming a difference image between images created by the sensor signals at different times; background correction; decomposition into color channels; decomposition into hues; saturation; and brightness channels; frequency decomposition; singular value decomposition; applying a blob detector; applying a corner detector; applying the determinant of a Hessian filter; applying a region detector based on principal curvature; applying a maximally stable extremal region detector; applying a generalized Hough transform; applying a ridge detector; applying an affine invariant feature detector; applying an affine adaptive interest point operator; applying a Harris affine region detector; applying a Hessian affine region detector; applying a scale-invariant feature transform; applying a scale-space extremum detector; applying a local feature detector; applying an accelerated robust features algorithm; applying a histogram of gradient positions and orientations algorithm; applying a histogram of orientation gradient descriptors; applying a Deriche edge detector; applying a differential edge detector; applying a spatio-temporal interest point detector; applying a Moravec corner detector; applying a Canny edge detector; applying a Laplacian of Gaussian filter; applying a difference of Gaussian filter; applying a Sobel operator; applying a Laplacian operator; applying a Scharr operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transform; applying a Radon transform; applying a Hough transform; applying a wavelet transform; thresholding; creating a binary image. The region of interest may be determined manually by the user or may be determined automatically, for example, by identifying features within the image generated by the optical sensor 126.

[0251] For example, the illumination unit 118 may be configured to generate and / or project a point cloud such that a plurality of illumination regions are generated on the optical sensor 126 (e.g., a CMOS detector). Additionally, there may be interference on the optical sensor 126, such as interference due to speckle and / or extraneous light and / or multiple reflections. The processing unit 114 may be adapted to determine at least one region of interest, such as one or more pixels illuminated by a light beam, which is used to determine the longitudinal coordinates of the corresponding reflection features, which will be described in more detail below. For example, the processing unit 114 may be adapted to perform filtering methods, such as speckle analysis and / or edge filtering and / or object recognition methods.

[0252] The processing unit 114 may be configured to perform at least one image correction. The image correction may include at least one background subtraction. The processing unit 114 may be adapted to remove the influence of background light from the beam profile, for example, by imaging without further illumination.

[0253] The processing unit 114 may be configured to determine the beam profile of a corresponding reflection feature. Determining the beam profile may include identifying at least one reflection feature provided by the optical sensor 126 and / or selecting at least one reflection feature provided by the optical sensor 126 and evaluating at least one intensity distribution of the reflection feature. As an example, regions of a matrix may be used and evaluated to determine the intensity distribution, such as a three-dimensional intensity distribution or a two-dimensional intensity distribution, for example, along an axis or line passing through the matrix. As an example, the illumination center of the beam may be determined, such as by determining at least one pixel with the highest illumination, and a cross-sectional axis passing through the illumination center may be selected. The intensity distribution may be an intensity distribution as a function of coordinates along the cross-sectional axis passing through the illumination center. Other evaluation algorithms are also feasible.

[0254] The processing unit 114 is configured to determine first beam profile information of at least one reflection feature by analyzing the beam profile of at least one reflection feature located within an image region of a second image corresponding to the image region of the first image including the identified geometric feature. The method may include identifying the image region of the second image corresponding to the image region of the first image including the identified geometric feature. Specifically, the method may include matching the pixels of the first image and the second image and selecting the pixels of the second image corresponding to the image region of the first image including the identified geometric feature. The method may include additionally considering additional reflection features located outside the image region of the second image.

[0255] The beam profile information can be or can include any information and / or property derived from and / or associated with the beam profile of the reflection feature. The first and second beam profile information can be the same or can be different. For example, the first beam profile information can be an intensity distribution, a reflection profile, an intensity center, a material feature. For skin detection in step b) 116, beam profile analysis can be used. Specifically, beam profile analysis classifies materials by utilizing the reflection properties of coherent light projected onto the surface of an object. The classification of materials can be performed as described in WO 2020 / 187719, EP application 20159984.2 filed on February 28, 2020, and / or EP application 20154961.5 filed on January 31, 2020, the entire contents of which are incorporated by reference. Specifically, a periodic grid of laser spots is projected, such as the hexagonal grid described in EP application 20170905.2 filed on April 22, 2020, and the reflected image is recorded with a camera. The beam profile of each reflection feature recorded by the camera can be analyzed by a feature-based method. Regarding the feature-based method, reference is made to the above description. The feature-based method can be used in combination with a machine learning method, which can allow for the parameterization of a skin classification model. Alternatively or in combination, a convolutional neural network can be utilized to classify the skin by using the reflected image as an input.

[0256] The skin detection step 116 may include determining at least one material property of the reflection feature based on the beam profile information by using the processing unit 114. Specifically, the processing unit 114 is configured to identify the reflection feature that will be generated by irradiating biological tissue (especially human skin) when its reflected beam profile meets at least one predetermined or predefined criterion. The at least one predetermined or predefined criterion may be at least one property and / or value suitable for distinguishing biological tissue (especially human skin) from other materials. The predetermined or predefined criterion may be or may include at least one predetermined or predefined value and / or threshold and / or threshold range related to the material property. In the case where the reflected beam profile meets at least one predetermined or predefined criterion, the reflection feature may be indicated as being generated by biological tissue. The processing unit is configured to otherwise identify the reflection feature as non-skin. Specifically, the processing unit 114 may be configured for skin detection, especially for identifying whether the detected face is human skin. If the material is biological tissue, especially human skin, the identification may include determining and / or verifying whether the surface to be examined or tested is or includes biological tissue (especially human skin), and / or distinguishing biological tissue (especially human skin) from other tissues (especially other surfaces). The method according to the present invention may allow distinguishing human skin from one or more of inorganic tissue, metal surfaces, plastic surfaces, foams, papers, woods, displays, screens, fabrics. The method according to the present invention may allow distinguishing human biological tissue from the surface of artificial or non-living objects.

[0257] The processing unit 114 may be configured to determine a material property m of a surface emitting a reflected feature by evaluating a beam profile of the reflected feature. The material property may refer to at least any one property of a material that is configured to characterize and / or identify and / or classify the material. For example, the material property may be a property selected from the group consisting of: roughness, penetration depth of light into the material, property characterizing the material as a biological or non-biological material, reflectivity, specular reflectivity, diffuse reflectivity, surface property, measurement of translucency, scattering, especially backscattering behavior, etc. At least one material property may be a property selected from the group consisting of: scattering coefficient, translucency, transparency, deviation from Lambertian surface reflection, speckle, etc. Determining at least one material property may include assigning the material property to the detected face. The processing unit 114 may include at least one database that includes a list and / or table of predefined and / or pre-determined material properties, such as a look-up list or a look-up table. The list and / or table of material properties may be determined and / or generated by performing at least one test measurement, such as by performing a material test using a sample with known material properties. The list and / or table of material properties may be determined and / or generated at the manufacturer's and / or by the user. The material property may also be assigned to a material classifier, such as one or more of the following: material name, material group, such as biological or non-biological material, translucent or non-translucent material, metal or non-metal, skin or non-skin, fur or non-fur, carpet or non-carpet, reflective or non-reflective, specular reflective or non-specular reflective, foam or non-foam, hair or non-hair, roughness group, etc. The processing unit 114 may include at least one database that includes a list and / or table that includes the material property and the associated material name and / or material group.

[0258] While feature-based methods are accurate enough to distinguish skin from only surface-scattering materials, the distinction between skin and carefully selected attack materials (which also involve volume scattering) is more challenging. Step b) 116 may include using artificial intelligence, especially a convolutional neural network. Using the reflected image as the input to the convolutional neural network may generate a classification model with sufficient accuracy to distinguish skin from other volume-scattering materials. Since only physically valid information is passed to the network by selecting important regions in the reflected image, only a compact training dataset may be required. In addition, a very compact network architecture may also be generated.

[0259] Specifically, in the skin detection step 116, at least one parameterized skin classification model can be used. The parameterized skin classification model can be configured to classify skin and other materials by using the second image as input. The skin classification model can be parameterized by using one or more of machine learning, deep learning, neural networks, or other forms of artificial intelligence. Machine learning can include methods of using artificial intelligence (AI) for automatic model building, particularly parameterized models. The skin classification model can include a classification model configured to distinguish human skin from other materials. The nature characteristics of the skin can be determined by applying an optimization algorithm according to at least one optimization objective on the skin classification model. Machine learning can be based on at least one neural network, particularly a convolutional neural network. The weights and / or topology of the neural network can be predetermined and / or predefined. Specifically, machine learning can be used to perform the training of the skin classification model. The skin classification model can include at least one machine learning architecture and model parameters. For example, the machine learning architecture can be or can include one or more of the following: linear regression, logistic regression, random forest, naive bayes classification, nearest neighbor, neural network, convolutional neural network, generative adversarial network, support vector machine, or gradient boosting algorithm, etc. As used herein, training can include the process of building the skin classification model, particularly determining and / or updating the parameters of the skin classification model. The skin classification model can be at least partially data-driven. For example, the skin classification model can be based on experimental data, such as data determined by illuminating multiple people and artificial objects such as masks and recording the reflection patterns. For example, training can include using at least one training data set, where the training data set includes images of multiple people and artificial objects with known material properties, particularly the second image.

[0260] The skin detection step 116 can include using at least one 2D face and facial landmark detection algorithm, which is configured to provide at least two positions of the characteristic points of the human face. For example, the positions can be the eye positions, forehead, or cheeks. The 2D face and facial landmark detection algorithm can provide the positions of the characteristic points of the human face, such as the eye positions. Since there are slight differences in the reflections of different regions of the face (such as the forehead or cheeks), region-specific models can be trained. In the skin detection step 116, preferably at least one region-specific parameterized skin classification model is used. The skin classification model can include multiple region-specific parameterized skin classification models, for example, for different regions, and / or region-specific data can be used to train the skin classification model, such as by filtering the images used for training. For example, for training, two different regions can be used, such as the eye-cheek region below the nose, and particularly in the case where insufficient reflection features can be identified within this region, the region of the forehead can be used. However, other regions are also possible.

[0261] If the material properties correspond to at least one property characteristic of the skin, the detected face is characterized as skin. The processing unit 114 may be configured to identify reflection features that will be generated by illuminating biological tissue, particularly skin, when its corresponding material properties meet at least one predetermined or predefined criterion. In the case where the material properties indicate "human skin", the reflection features may be identified as being generated by human skin. If the material properties are within at least one threshold and / or at least one range, the reflection features may be identified as being generated by human skin. The at least one threshold and / or range may be stored in a table or look-up table and may be determined empirically, for example, and stored in at least one data storage device of the processing unit as an example. The processing unit 114 is configured to otherwise identify the reflection features as background. Thus, the processing unit 114 may be configured to assign material properties, such as skin yes or no, to each projection point.

[0262] The 3D detection step 120 may be performed after the skin detection step 116 and / or the face detection step 110. However, other embodiments are also feasible, where the 3D detection step 120 is performed before the skin detection step 116 and / or the face detection step 110.

[0263] The 3D detection step 120 includes determining second beam profile information of at least four reflection features by analyzing the beam profiles of at least four reflection features located within an image region of a second image corresponding to the image region of the first image including the identified geometric features. The second beam profile information may include the quotient Q of the areas of the beam profiles.

[0264] The analysis of the beam profile may include the evaluation of the beam profile and may include at least one mathematical operation and / or at least one comparison and / or at least one symmetrization and / or at least one filtering and / or at least one normalization. For example, the analysis of the beam profile may include at least one of a histogram analysis step, the calculation of a difference measurement, the application of a neural network, the application of a machine learning algorithm. The processing unit 114 may be configured to symmetrize and / or normalize and / or filter the beam profile, particularly to remove noise or asymmetry from recordings at large angles, recording edges, etc. The processing unit 114 may filter the beam profile by removing high spatial frequencies (e.g., by spatial frequency analysis and / or median filtering, etc.). Summarization may be performed by the intensity center of the light spot and averaging all intensities at the same distance from the center. The processing unit 114 may be configured to normalize the beam profile to the maximum intensity, particularly taking into account intensity differences due to the recorded distance. The processing unit 114 may be configured to remove the influence of background light from the beam profile, for example, by imaging without illumination.

[0265] The processing unit 114 may be configured to determine at least one longitudinal coordinate z of the respective reflection feature by analyzing a beam profile of the reflection feature located within an image region of a second image corresponding to an image region of the first image including the identified geometric feature DPR The processing unit 114 may be configured to determine the longitudinal coordinate z of the reflection feature by using a so-called photon depth ratio technique (also denoted as beam profile analysis) DPR Regarding the photon depth ratio (DPR) technique, reference is made to WO2018 / 091649A1, WO 2018 / 091638A1 and WO 2018 / 091640A1, the entire contents of which are incorporated by reference

[0266] The longitudinal coordinate of the reflection feature can be the distance between the optical sensor 126 and the point in the scene that emits the corresponding illumination feature. The analysis of the beam profile of one of the reflection features can include determining at least one first region and at least one second region of the beam profile. The first region of the beam profile can be region A1 and the second region of the beam profile can be region A2. The processing unit 114 can be configured to integrate the first region and the second region. The processing unit 114 can be configured to derive a combined signal, in particular the quotient Q, by one or more of the following: dividing the integrated first region and the integrated second region, dividing the integrated first region and a multiple of the integrated second region, dividing the integrated first region and a linear combination of the integrated second region. The processing unit 114 can be configured to determine at least two regions of the beam profile and / or divide the beam profile into at least two segments including different regions of the beam profile, where overlap of the regions is possible as long as the regions are not the same. For example, the processing unit 114 can be configured to determine a plurality of regions, such as two, three, four, five or up to ten regions. The processing unit 114 can be configured to divide the light spot into at least two regions of the beam profile and / or divide the beam profile into at least two segments including different regions of the beam profile. The processing unit 114 can be configured to determine the integral of the beam profile over at least two regions. The processing unit can be configured to compare at least two of the determined integrals. Specifically, the processing unit 114 can be configured to determine at least one first region and at least one second region of the beam profile. The regions of the beam profile can be any regions of the beam profile at the location of the optical sensor for determining the quotient Q. The first region of the beam profile and the second region of the beam profile can be one or both of adjacent or overlapping regions. The first region of the beam profile and the second region of the beam profile can be different in area. For example, the processing unit 114 can be configured to divide the sensor area of the CMOS sensor into at least two sub-regions, where the processing unit can be configured to divide the sensor area of the CMOS sensor into at least one left portion and at least one right portion and / or at least one upper portion and at least one lower portion and / or at least one inner portion and at least one outer portion. Additionally or alternatively, the camera 112 can include at least two optical sensors 126, where the photosensitive regions of the first optical sensor 126 and the second optical sensor 126 can be arranged such that the first optical sensor 126 is adapted to determine the first region of the beam profile of the reflection feature, and the second optical sensor 126 is adapted to determine the second region of the beam profile of the reflection feature. The processing unit 114 can be adapted to integrate the first region and the second region. The processing unit 114 can be configured to determine the longitudinal coordinate using at least one predetermined relationship between the quotient Q and the longitudinal coordinate. The predetermined relationship can be one or more of an empirical relationship, a semi-empirical relationship, and an analytically derived relationship.The processing unit 114 may include at least one data storage device for storing predetermined relationships, such as a lookup list or a lookup table.

[0267] The 3D detection step may include determining at least one depth level from the second beam profile information of the reflection features by using the processing unit.

[0268] The processing unit 114 may be configured to determine a depth map of at least a portion of the scene by determining at least one depth information of the reflection features located within an image region of a second image corresponding to an image region of a first image including the identified geometric features. The processing unit 114 may be configured to determine the depth information of the reflection features by one or more of the following techniques: photon depth ratio, structured light, beam profile analysis, time of flight, shape from motion, depth from focus, triangulation, depth from defocus, stereo sensors. The depth map may be a sparsely populated depth map including several entries. Alternatively, the depth may be crowded, including a large number of entries.

[0269] If the depth level deviates from a predetermined or predefined depth level of a planar object, the detected face is characterized as a 3D object. Step c) 120 may include using the 3D topology data of the face in front of the camera. The method may include determining a curvature based on at least four reflection features located within an image region of a second image corresponding to an image region of a first image including the identified geometric features. The method may include comparing the curvature determined from at least four reflection features with a predetermined or predefined depth level of a planar object. If the curvature exceeds the assumed curvature of the planar object, the detected face may be characterized as a 3D object, otherwise it may be characterized as a planar object. The predetermined or predefined depth level of the planar object may be stored in at least one data memory of the processing unit, such as a lookup list or a lookup table. The predetermined or predefined level of the planar object may be determined experimentally and / or may be a theoretical level of the planar object. The predetermined or predefined depth level of the planar object may be at least one limit of at least one curvature and / or a range of at least one curvature.

[0270] The 3D features determined in step c) 120 may allow for distinguishing between a high-quality photo and a 3D face-like structure. The combination of steps b) 116 and c) 120 may allow for enhancing the reliability of authentication against attacks. The 3D features may be combined with material features to increase the security level. Since the same computational pipeline can be used to generate the input data for skin classification and the generation of the 3D point cloud, these two properties can be calculated from the same frame with a lower computational load.

[0271] Preferably, after steps a) 110, b) 116, and c) 120, an authentication step 122 may be performed. The authentication step 122 may be partially performed after each of steps a) to c). If no face is detected in step a) 110 and / or the reflection feature is determined not to be generated by the skin in step b) 116 and / or the depth map in step c) 120 relates to a planar object, the authentication may be aborted. The authentication step includes authenticating the detected face by using at least one authentication unit if the face detected in step b) 116 is characterized as skin and the face detected in step c) 122 is characterized as a 3D object.

[0272] Steps a) to d) may be performed by using at least one device, such as at least one mobile device 124, such as a mobile phone, a smartphone, etc., where access to the device is protected by using face authentication. Other devices are also possible, such as access control devices that control access to buildings, machines, automobiles, etc. The method may include allowing access to the device if the detected face is authenticated.

[0273] The method may include at least one registration step. In the registration step, a user of the device may be registered. Registration may include the process of enrolling and / or subscribing and / or instructing the user for subsequent use of the device. Generally, registration may be performed at the first use of the device and / or when the device is started. However, embodiments in which multiple users may be registered, for example, continuously, such that registration may be performed and / or repeated at any time during the use of the device are feasible. Registration may include generating a user account and / or a user profile. Registration may include inputting and storing user data, particularly image data, via at least one user interface. Specifically, at least one 2D image of the user is stored in at least one database. The registration step may include imaging at least one image of the user, particularly multiple images. Images may be recorded from different directions and / or the user may change his orientation. Additionally, the registration step may include generating at least one 3D image and / or depth map of the user, which may be used for comparison in step d). The database may be a database of the device, such as the database of the processing unit 114, and / or may be an external database such as a cloud. The method includes identifying the user by comparing the 2D image of the user with a first image. The method according to the present invention may allow a significant improvement in the presentation attack detection ability of the biometric authentication method. To improve overall authentication, in addition to the 2D image of the user, personal specific material fingerprints and 3D topological features may also be stored during the registration process. This may allow multi-factor authentication within a device by using 2D, 3D, and material-derived features.

[0274] The method using beam profile analysis technology according to the present invention can provide a concept for reliably detecting human skin by analyzing the reflection of laser spots on the face (especially in the NIR range) and differentiating it from the reflection from attack materials generated by mimicking the face. In addition, beam profile analysis provides depth information simultaneously by analyzing the same camera frame. Therefore, 3D as well as skin safety features can be provided by exactly the same technology.

[0275] Since a 2D image of the face can also be recorded simply by turning off the laser illumination, a completely secure face recognition pipeline can be established to solve the above problems.

[0276] When the laser wavelength moves towards the NIR region, the reflection properties of human skin of different races become more similar. At a wavelength of 940 nm, the difference is minimal. Therefore, different races do not play a role in skin authentication.

[0277] Time-consuming analysis of a series of frames may not be required because attack detection (through skin classification) is presented by only one frame. The time frame for performing the complete method can be ≤500 ms, preferably ≤250 ms. However, embodiments where multiple frames can be used to perform skin detection may be feasible. Depending on the confidence in identifying the reflection features in the second image and the speed of the method, the method may include sampling the reflection features over multiple frames to achieve more stable classification.

[0278] Figure 3 The experimental results are shown, especially the density as a function of the skin score. The x-axis shows the score and the y-axis shows the frequency. The score is a measure of the classification quality, with values ranging from 0 to 1, where 1 indicates a very high skin similarity and 0 indicates a very low skin similarity. The decision threshold may be approximately 0.5. A reference distribution of the skin scores for genuine presentations was generated using 10 subjects. Skin scores were also recorded for presentation attacks (PA) of Class A, Class B, and Class C (as defined in the relevant ISO standard). The experimental setup (evaluation target, ToE) includes proprietary hardware devices such as Figure 2As shown, it includes necessary sensors and a computing platform that executes PAD software. The ToE was tested using 6 types of Class A PAI (Presentation Attack Instruments), 5 types of Class B PAI, and 1 type of Class C PAI. For each PAI category, 10 PAIs were used. The PAI categories used in this study are listed in the table below. In the table, APCER is the Attack Presentation Classification Error Rate, which means the number of successful attacks / the total number of attacks * 100. In the table, BPCER is the BonaFide Presentation Classification Error Rate, that is, the number of rejected unlock attempts / the total number of unlock attempts * 100. Both Class A and Class B attacks are based on 2D PAI, while Class C attacks are based on 3D masks. For Class C attacks, a custom rigid mask built using a 3D printer was used. A test group consisting of 10 subjects was used to obtain the reference distribution of the skin scores for bona fide presentations.

[0279]

[0280] Experiments conducted using these PAIs show that two types of presentations (bona fide or PA) can be clearly distinguished according to the skin scores. Papers, 3D prints, and skin can be clearly distinguished using the method according to the present invention.

[0281] List of reference numbers

[0282] 110 Facial detection step

[0283] 112 Camera

[0284] 114 Processing unit

[0285] 116 Skin detection step

[0286] 118 Lighting unit

[0287] 120 3D detection step

[0288] 122 Authentication step

[0289] 124 Mobile device

[0290] 126 Optical sensor

Claims

1. A method for face authentication, comprising the following steps: a) At least one face detection step (110), wherein the face detection step (110) includes: using at least one camera (112) to capture at least one first image, wherein the first image includes at least one two-dimensional image of a scene suspected of including a face, and wherein the face detection step (110) includes: using at least one processing unit (114) to detect the face in the first image by identifying at least one predefined or predetermined geometric feature characteristic for the face in the first image; b) At least one skin detection step (116), wherein the skin detection step (116) includes: projecting at least one illumination pattern including a plurality of illumination features onto the scene by using at least one illumination unit (118), and using the at least one camera (112) to capture at least one second image, wherein the second image includes a plurality of reflection features generated by the scene in response to the illumination of the illumination features, wherein each of the reflection features includes at least one beam profile, and wherein the skin detection step includes: using the processing unit (114) to determine first beam profile information of at least one of the reflection features by analyzing the beam profile of at least one of the reflection features within an image region of the second image corresponding to the image region of the first image including the identified geometric features, and determining at least one material property of the reflection feature from the first beam profile information, wherein if the material property corresponds to at least one property characteristic for skin, the detected face is characterized as skin; c) At least one 3D detection step (120), wherein the 3D detection step (120) includes: determining a depth map of at least a part of the scene by determining at least one depth information of the reflection features within an image region of the second image corresponding to the image region of the first image including the identified geometric features, and wherein the 3D detection step (120) includes using 3D topology data of the face in front of the camera (112); d) At least one authentication step (122), wherein the authentication step (122) includes: authenticating the detected face by using at least one authentication unit if the face detected in step b) (116) is characterized as skin and the face detected in step c) (120) is characterized as a 3D object.

2. The method according to claim 1, wherein, Steps a) to d) are performed by using at least one device, wherein access to the device is protected by using face authentication, and wherein the method includes: allowing access to the device if the detected face is authenticated.

3. The method according to claim 1 or 2, wherein The method includes at least one registration step, wherein in the registration step, a user of the device is registered.

4. The method according to any one of claims 1 to 3, wherein, The face detection step (110) includes identifying one or more of the following in the first image: nose, eyes, mouth, or eyebrows.

5. The method according to any one of claims 1 to 4, wherein In the skin detection step, at least one parameterized skin classification model is used, wherein the parameterized skin classification model is configured to classify skin and other materials.

6. The method according to claim 5, wherein, The skin classification model is a neural network.

7. The method according to claim 5 or 6, wherein The skin detection step includes: using a skin classification model that is parameterized to account for differences in reflections of different regions of the face.

8. The method according to claim 7, wherein The skin classification model is specific to the forehead or cheek of the face.

9. The method according to any one of claims 1 to 8, wherein, Determining the beam profile information of the at least one reflection feature includes: applying an image filter, wherein the image filter maps the second image or the region of interest of the second image to real numbers, and the real numbers are used to determine material properties.

10. The method according to any one of claims 1 to 9, wherein, At least one material property of the reflection feature includes: translucency, transparency, deviation from Lambertian surface reflection.

11. The method according to any one of claims 1 to 10, wherein, The illumination pattern includes: a pseudo-random dot pattern or a random dot pattern.

12. The method according to any one of claims 1 to 11, wherein, The illumination feature has a wavelength in the range of 700 nm to 1100 nm.

13. The method according to claim 12, wherein, The illumination feature has a wavelength of 940 nm.

14. The method according to any one of claims 1 to 13, wherein, Multiple second images are captured, wherein the reflection features of the multiple second images are used for skin detection in step b) (116) and / or for 3D detection in step c) (120).

15. The method according to any one of claims 1 to 14, wherein, The skin detection step (116) is performed after the face detection step (110).

16. The method according to any one of claims 1 to 15, wherein The camera (112) is or includes at least one near-infrared camera.

17. A non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform at least steps a) to d) of the method according to any one of claims 1 to 16.

18. A mobile device (124) including at least one camera (112), at least one illumination unit (118), and at least one processing unit (114), the mobile device (124) being configured to perform steps a) to c) of the method for face authentication according to any one of claims 1 to 16, and optionally step d).

19. The mobile device (124) according to claim 18, wherein, The illumination unit (118) includes at least one vertical-cavity surface-emitting laser and one or more diffractive optical elements.

20. The mobile device (124) according to claim 18 or 19, wherein, The illumination unit (118) is configured to generate an illumination pattern including a pseudo-random dot pattern or a random dot pattern.

21. The mobile device (124) according to any one of claims 18 to 20, wherein, The illumination unit (118) includes multiple light sources, wherein the light sources emit light having a wavelength in the range of 700 nm to 1100 nm.

22. The mobile device (124) according to any one of claims 18 to 21, wherein, The camera (112) includes a CMOS chip configured to capture images.

23. The mobile device (124) according to any one of claims 18 to 22, wherein, The mobile device (124) is a mobile phone or a smartphone.

Images