Optical skin detection for face unlocking

Abstract

A face recognition method is proposed, the method comprising the following steps: a) at least one face detection step (110) determining at least one first image by using at least one camera (112); b) at least one skin detection step (116) of projecting at least one illumination pattern comprising a plurality of illumination features onto a scene by using at least one illumination unit (118), determining at least one second image using said at least one camera (112), determining by analysis of a beam profile at least one first beam profile information of a reflection feature located within an image area of ​​said second image corresponding to an image area of ​​said first image comprising an identified geometric feature, and determining from said first beam profile information by using said processing unit (114) at least one material characteristic of said reflection feature, wherein said detected face is characterized as skin if said material characteristic corresponds to at least one characteristic characteristic of skin; c) at least one 3D detection step comprising: determining, by analysis of beam profiles, at least four second beam profile information of the reflection features located in image areas of the second image corresponding to image areas of the first image containing the identified geometric features; and determining, by using the processing unit (114), at least one depth level from the second beam profile information of the reflection features, wherein the detected face is characterized as a 3D object if the depth level deviates from the predetermined or predefined depth level of a planar object; d) at least one authentication step, including authenticating the detected face by using at least one authentication unit if in step b) (116) the detected face is characterized as skin and in step c) (120) the detected face is characterized as a 3D object; Includes.

Description

[Technical Field]

[0001] This invention relates to a facial recognition method, a mobile device, and various uses of the method. The apparatus, method, and use according to the present invention can be specifically employed in various fields such as daily life, security technology, gaming, transportation technology, production technology, art, photography including digital or videography for documentary or technical purposes, safety technology, information technology, agriculture, crop protection, maintenance, cosmetics, medical technology, or science. However, other applications are also possible. [Background technology]

[0002] In today's digital world, secure access to information technology is a requirement for all state-of-the-art systems. Standard concepts such as passphrases or PIN codes are now being extended, combined with, or replaced by biometric authentication methods such as fingerprints or facial recognition. While passphrases can offer a high level of security if carefully chosen for their appropriate length, this step requires caution, as it may require memorizing several potentially long phrases depending on the IT context. Furthermore, there is no guarantee that the person providing the passphrase is authorized by the passphrase or the owner of the digital device. In contrast, biometric features such as facial or fingerprints are unique and individual characteristics. Therefore, using features derived from these is not only more convenient than passphrases / PIN codes, but also more secure because it combines personal identity with the unlocking process.

[0003] Unfortunately, just as passwords can be stolen, fingerprints and faces can be artificially created to impersonate legitimate users. First-generation automated face recognition tools use digital camera images or image streams, applying two-dimensional image processing methods to extract characteristic features, while machine learning techniques are used to generate face templates, and ID recognition is performed based on those features. Second-generation face recognition algorithms use deep convolutional neural networks instead of manually created image features to generate classification models.

[0004] However, both approaches are vulnerable to attacks using, for example, high-resolution photos that can be freely downloaded from the internet today. Therefore, the concept of Presentation Attack Detection (PAD) has become important. Early approaches were designed to protect against simple attacks, such as presenting a picture of a legitimate user, by recording a series of images and testing for time-dependent features, such as small but natural changes in head position or eye blinking. These methods can again be fooled by playing back a recorded video of the user or by careful animations generated from publicly available photos. To exclude displays as potentially fraudulent objects, near-infrared (NIR) cameras can be used, as displays emit photons only in the visible region of the electromagnetic spectrum. As a further measure, 3D cameras have been introduced, which can clearly distinguish a 3D face from a tablet with a flat image or playing video. Even so, these systems can still be attacked by, to name a few approaches, high-quality masks produced by 3D printing, careful 3D placement of 2D photos, or handmade silicone or latex masks. Since masks are typically worn by humans, survival detection based on small movements also fails. However, these types of masks may be rejected by PAD systems that can classify human skin from other materials, such as those described in EP application No. 20159984.2 filed on 28 February 2020 and EP application No. 20154961.5190679 filed on 31 January 2020 (their full contents are included by reference).

[0005] Furthermore, differences in the optical properties of skin due to ethnic origin can lead to additional problems. Reliable recognition and PAD technology must be completely independent of such different origins.

[0006] Beyond security considerations, the speed of the unlocking process and the necessary computing power are crucial for providing an acceptable user experience. High-speed facial recognition can be used for several tasks after the device has been successfully unlocked, such as checking if the user is still in front of the display or performing an on-the-fly check on the person in front of the display to launch more secure applications like banking apps. In this case as well, speed and computing resources have a significant impact on the user experience.

[0007] Current 3D algorithms are computationally very demanding, requiring the processing of multiple video frames to detect presentation attacks. Therefore, achieving acceptable unlocking performance necessitates the use of expensive hardware. Furthermore, this results in high power consumption.

[0008] In summary, current methods for face unlocking do not provide the ability to reliably detect 3D mask impersonation attacks and perform this task at a speed below the limits of human detection.

[0009] US2019 / 213309A1 describes a system and method for authenticating a user's face using a ranging sensor. The ranging sensor includes a time-of-flight sensor and a reflectance sensor. The ranging sensor transmits a signal that is reflected from the user and received back by the ranging sensor. The received signal can be used to determine the distance between the user and the sensor, and the user's reflectance. The distance and reflectance allow a processor to initiate a face recognition process. [Overview of the project] [Problems that the invention aims to solve]

[0010] Therefore, an object of the present invention is to provide an apparatus and a method for the above technical problems of known apparatuses and methods. Simple presentation attacks using genuine face photos and videos can be detected, but there is still a lack of an approach to reliably detect presentation attacks using 3D face masks. Specifically, another security layer is needed to enable replacement with a passphrase / PIN code by a biometric function derived from the face in order to unlock a digital device. Furthermore, there is a need for a method that functions without being completely dependent on different skin types originating from different ethnic origins.

Means for Solving the Problems

[0011] This problem is solved by the present invention having the features of the independent patent claims. Advantageous developments of the present invention that can be realized individually or in combination are shown in the dependent claims and / or the following description and detailed embodiments.

[0012] When used hereinafter, the terms "have", "comprise", or "include", or any grammatical variations thereof, are used in a non-exclusive manner. Thus, these terms can refer to both situations where there are no additional features in the entity described in this context other than the features introduced by these terms, and situations where there are one or more additional features. As an example, the expressions "A has B", "A comprises B", and "A includes B" refer to both situations where there are no other elements in A other than B (i.e., situations where A is composed solely and exclusively of B), and situations where, in addition to B, one or more elements, such as element C, elements C and D, or even further elements, are present in entity A.

[0013] Furthermore, it should be noted that terms such as "at least one", "one or more", or similar expressions indicating that a feature or element may be present one or more times are typically used only once when introducing each feature or element. In the following, in most cases, when referring to each feature or element, it should be noted that the expressions "at least one" or "one or more" are not repeated, despite the fact that those features or elements may appear one or more times.

[0014] Furthermore, when used hereinafter, terms such as "preferably", "more preferably", "particularly", "more particularly", "specifically", "more specifically", or similar terms are used in relation to any feature without limiting the possibility of alternatives. Thus, the features introduced by these terms are any features and are not intended to limit the scope of the claims in any way. As would be recognized by those skilled in the art, the present invention can be implemented using alternative features. Similarly, features introduced by expressions such as "in one embodiment of the present invention" or similar expressions are any features without any limitation regarding alternative embodiments of the present invention, without any limitation regarding the scope of the present invention, and without any limitation regarding the possibility of combining such introduced features with any other optional or non-optional features of the present invention.

[0015] A first aspect of the present invention discloses a method for facial recognition. The face to be recognized may specifically be a human face. As used herein, the term “facial recognition” is a broad term and should be given a common and idiomatic meaning to those skilled in the art, and should not be limited to any special or customized meaning. Specifically, the term may mean, without limitation, verifying that a recognized object or part of a recognized object is a human face. Specifically, recognition may include distinguishing a genuine human face from attack material created to mimic a face. Recognition may include verifying the identity of each user and / or assigning an identity to a user. Recognition may include generating and / or providing identification information to other devices, such as at least one authentication device for authenticating access to a mobile device, machine, automobile, building, etc. The identification information may be certified by recognition. For example, the identification information may be at least one identification token and / or include at least one identification token. If authentication is successful, it is verified that the recognized object or part of the recognized object is a genuine face and / or the object, in particular, the user's identity.

[0016] This method involves the following steps: a) at least one face detection step, the face detection step comprising determining at least one first image by using at least one camera, the first image comprising at least one two-dimensional image of a scene that is believed to contain a face, and the face detection step comprising detecting a face in the first image by using at least one processing unit to identify at least one predefined or predetermined geometric feature characteristic of a face in the first image; b) At least one skin detection step, the skin detection step comprising projecting at least one illumination pattern including a plurality of illumination features onto a scene by using at least one illumination unit, and determining at least one second image using at least one camera, the second image including a plurality of reflective features generated by the scene in response to illumination by the illumination features, each of the reflective features including at least one beam profile, the skin detection step comprising determining at least one first beam profile information of a reflective feature located in an image region of the second image corresponding to an image region of the first image including identified geometric features by analysis of the beam profile, and determining at least one material property of a reflective feature from the first beam profile information by using a processing unit, wherein the detected face is characterized as skin if the material property corresponds to a characteristic property of at least one of the skin; c) at least one 3D detection step, the 3D detection step comprising: determining at least four second beam profile information of a reflection feature located within an image region of a second image corresponding to an image region of a first image containing identified geometric features by analyzing beam profiles; and determining at least one depth level from the second beam profile information of the reflection feature by using a processing unit, wherein the detected face is characterized as a 3D object if the depth level deviates from a predetermined or predefined depth level of a planar object; d) at least one authentication step, the authentication step comprising authenticating the detected face by using at least one authentication unit, where the face detected in step b) is characterized as skin and the face detected in step c) is characterized as a 3D object, Includes.

[0017] The method steps may be performed in a predetermined order, or in a different order. Furthermore, there may be one or more additional method steps that are not listed. In addition, one, more than one, or even all of the method steps may be repeated.

[0018] As used herein, the term “camera” is a broad term and should be given the usual and idiomatic meaning to those skilled in the art, and should not be limited to any special or customized meaning. Specifically, the term may refer to a device having at least one image element configured to record or capture spatially resolved one-dimensional, two-dimensional, or even three-dimensional optical data or information. The camera may be a digital camera. As an example, the camera may comprise at least one camera chip, such as at least one CCD chip and / or at least one CMOS chip, configured to record an image. The camera may be at least one near-infrared camera, or may comprise at least one near-infrared camera.

[0019] As used herein, the term “image” may, without limitation, specifically refer to data recorded by using a camera, such as multiple electronic readings from image elements, such as pixels of a camera chip. A camera may include, in addition to at least one camera chip or image chip, one or more optical elements, such as one or more lenses. For example, a camera may be a fixed-focus camera having at least one lens fixedly adjusted relative to the camera. Alternatively, however, a camera may have one or more variable lenses that can be adjusted automatically or manually.

[0020] The camera may be the camera of a notebook computer, tablet, or, more specifically, a mobile device such as a smartphone or other mobile phone. Therefore, specifically, the camera may be part of a mobile device comprising at least one camera in addition to one or more data processing devices, such as one or more data processors. However, other cameras are also possible. As used herein, the term “mobile device” is a broad term and should be given the usual and idiomatic meaning to those skilled in the art, and should not be limited to any special or customized meaning. Specifically, the term may refer, without limitation, to a mobile electronic device, more specifically to a mobile communication device such as a mobile phone or smartphone. Additionally or alternatively, a mobile device may refer to a tablet computer or other type of portable computer.

[0021] Specifically, the camera may be, and may comprise, at least one photosensor having at least one photosensitive area. As used herein, “photosensor” generally refers to a photosensitive device for detecting a light beam, such as for detecting an illumination and / or light spot generated by at least one light beam. As further used herein, “photosensitive area” generally refers to an area of ​​a photosensor that is illuminated externally by at least one light beam and generates at least one sensor signal in response to said illumination. Specifically, the photosensitive area may be located on the surface of each photosensor. However, other embodiments are also possible. The camera may comprise a plurality of photosensors, each having a photosensitive area. As used herein, the term “photosensor, each having at least one photosensitive area” refers to a configuration comprising a plurality of single photosensors, each having one photosensitive area, and a configuration comprising a single coupled photosensor having a plurality of photosensitive areas. The term “photosensor” further refers to a photosensitive device configured to generate one output signal. If the camera includes multiple light sensors, each light sensor may be embodied by providing exactly one photosensitive area that can be illuminated, for example, so that exactly one photosensitive area exists within each light sensor, and by generating exactly one uniform sensor signal for the entire light sensor in response to illumination of the photosensitive area. Thus, each light sensor may be a single-area light sensor. The use of single-area light sensors, however, makes the camera configuration particularly simple and efficient. Thus, as an example, commercially available light sensors, such as commercially available silicon photodiodes, each having exactly one photosensitive area, may be used in the configuration. However, other embodiments are also possible.

[0022] The optical sensor may specifically include or comprise at least one photodetector, preferably an inorganic photodetector, more preferably an inorganic semiconductor photodetector, and most preferably a silicon photodetector. Specifically, the optical sensor may have sensitivity in the infrared spectral range. The optical sensor may comprise at least one sensor element comprising a matrix of pixels. All pixels of the matrix, or at least one group of optical sensors in the matrix, may specifically be identical. A group of identical pixels in the matrix may specifically be provided for different spectral ranges, or all pixels may be identical with respect to spectral sensitivity. Furthermore, the pixels may be identical in size and / or with respect to their electronic or optoelectronic properties. Specifically, the optical sensor may be an array of at least one inorganic photodiode having sensitivity in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers, or comprise such an array. Specifically, the optical sensor may have sensitivity in the near-infrared region, particularly in the range of 700 nm to 1100 nm, where silicon photodiodes are applicable. The infrared light sensor that can be used in the light sensor may be a commercially available infrared light sensor, such as the infrared light sensor sold under the brand name Hertzstueck® by trinamX GmbH at Ludwigshafen am Rhein, Germany, D-67056. Thus, as an example, the light sensor may include at least one intrinsic photovoltaic light sensor, more preferably at least one semiconductor photodiode selected from the group consisting of Ge photodiodes, InGaAs photodiodes, extended InGaAs photodiodes, InAs photodiodes, InSb photodiodes, and HgCdTe photodiodes. Additionally or alternatively, the light sensor may include at least one exogenous photovoltaic light sensor, more preferably at least one semiconductor photodiode selected from the group consisting of Ge:Au photodiodes, Ge:Hg photodiodes, Ge:Cu photodiodes, Ge:Zn photodiodes, Si:Ga photodiodes, and Si:As photodiodes.Additionally or alternatively, the optical sensor may include at least one photoconductive sensor, such as a PbS or PbSe sensor, a bolometer, preferably selected from the group consisting of a VO bolometer and an amorphous Si bolometer.

[0023] The photosensor may have sensitivity in one or more of the ultraviolet, visible, or infrared spectral ranges. Specifically, the photosensor may have sensitivity in the visible spectral range of 500 nm to 780 nm, most preferably 650 nm to 750 nm, or 690 nm to 700 nm. Specifically, the photosensor may have sensitivity in the near-infrared region. Specifically, the photosensor may have sensitivity in the near-infrared region, particularly in the range of 700 nm to 1000 nm, where silicon photodiodes are applicable. Specifically, the photosensor may have sensitivity in the infrared spectral range, specifically in the range of 780 nm to 3.0 μm. For example, the photosensor may be, or include, at least one element selected from the group consisting of a photodiode, photocell, photoconductor, phototransistor, or any combination thereof, independently. For example, the light sensor may be, or may include, at least one element selected from the group consisting of CCD sensor elements, CMOS sensor elements, photodiodes, photocells, photoconductors, phototransistors, or any combination thereof. Any other type of photosensitive element may be used. The photosensitive element may generally be made entirely or partially from inorganic materials and / or entirely or partially from organic materials. Most commonly, one or more commercially available photodiodes, such as inorganic semiconductor photodiodes, may be used.

[0024] An optical sensor may include at least one sensor element that includes a matrix of pixels. Therefore, as an example, an optical sensor may be part of or constitute a pixelated optical device. For example, an optical sensor may be at least one CCD device and / or CMOS device, and / or include them. As an example, an optical sensor may be part of or constitute a CCD device and / or CMOS device having a matrix of pixels, where each pixel forms a photosensitive area.

[0025] As used herein, the term “sensor element” generally refers to a device or combination of devices configured to sense at least one parameter. In this case, the parameter may specifically be an optical parameter, and the sensor element may specifically be an optical sensor element. The sensor element may be formed as a single, unified device or as a combination of several devices. The sensor element includes a matrix of optical sensors. The sensor element may include at least one CMOS sensor. The matrix may consist of independent pixels, such as independent optical sensors. Thus, it can constitute a matrix of inorganic photodiodes. However, alternatively, a commercially available matrix, such as one or more CCD detectors, such as a CCD detector chip, and / or CMOS detectors, such as a CMOS detector chip, may be used. Thus, generally, the sensor element may be at least one CCD device and / or CMOS device, and / or may include it, and / or the optical sensors may form a sensor array or be part of a sensor array such as the matrix described above. Therefore, as an example, the sensor element may have an array of pixels, such as a rectangular array having m rows and n columns, where m and n are independently positive integers. Preferably, multiple columns and multiple rows are given, i.e., n>1, m>1. Therefore, as an example, n may be 2 to 16 or more, and m may be 2 to 16 or more. Preferably, the ratio of the number of rows to the number of columns is close to 1. As an example, n and m may be selected such that 0.3 ≤ m / n ≤ 3 by selecting m / n = 1:1, 4:3, 16:9 or similar. As an example, the array may be a square array having an equal number of rows and columns by selecting m=2, n=2 or m=3, n=3, etc.

[0026] The matrix may consist of independent pixels, such as independent light sensors. Thus, a matrix of inorganic photodiodes can be constructed. However, alternatively, one or more commercially available matrices, such as CCD detectors, such as CCD detector chips, and / or CMOS detectors, such as CMOS detector chips, can be used. Therefore, generally, a light sensor may be and / or include at least one CCD and / or CMOS device, and / or the light sensor of a camera may form a sensor array or be part of a sensor array such as the matrix described above.

[0027] The matrix may be a rectangular matrix having at least one row, preferably multiple rows and multiple columns. For example, the rows and columns may be oriented substantially vertically. As used herein, the term “substantially vertical” refers to a vertical orientation with a tolerance of, for example, ±20° or less, preferably ±10° or less, and more preferably ±5° or less. Similarly, the term “substantially parallel” refers to a parallel orientation with a tolerance of, for example, ±20° or less, preferably ±10° or less, and more preferably ±5° or less. Therefore, as an example, tolerances smaller than 20°, specifically smaller than 10°, or even smaller than 5° may be permitted. To provide a wide field of view, the matrix may have at least 10 rows, preferably at least 500 rows, and more preferably at least 1000 rows. Similarly, the matrix may have at least 10 columns, preferably at least 500 columns, and more preferably at least 1000 columns. The matrix may include at least 50 optical sensors, preferably at least 100,000 optical sensors, and more preferably at least 5,000,000 optical sensors. The matrix may contain a number of pixels ranging from several megapixels. However, other embodiments are also possible. Therefore, in configurations where axial rotational symmetry is expected, a circular or concentric arrangement of the optical sensors in the matrix, also called pixels, may be preferred.

[0028] Therefore, as an example, the sensor element may be part of or constitute a pixelated optical device. For example, the sensor element may be at least one CCD device and / or CMOS device, and / or may include them. As an example, the sensor element may be part of or constitute a CCD device and / or CMOS device having a matrix of pixels, where each pixel forms a photosensitive area. The sensor element may employ a rolling shutter or a global shutter to read the matrix of the light sensor.

[0029] The camera may further include at least one transfer device. The camera may further include one or more additional elements, such as one or more additional optical elements. The camera may include a transfer device such as at least one lens and / or at least one lens system, and at least one optical element selected from the group consisting of at least one diffractive optical element. The term “transfer device,” also called a “transfer system,” can generally refer to one or more optical elements adapted to modify a light beam, such as by changing one or more of the beam parameters of the light beam, the width of the light beam, or the direction of the light beam. The transfer device may be adapted to guide the light beam to a light sensor. The transporter may specifically include one or more of the following: at least one lens, for example, selected from the group consisting of at least one focusable lens, at least one aspherical lens, at least one spherical lens, and at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflection element, preferably at least one mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or beam splitting mirror; and at least one multi-lens system. The transporter may have a focal length. As used herein, the term “focal length” of a transporter refers to the distance at which incident parallel light rays that may strike the transporter are focused to a “focus,” also called a “focal point.” Thus, the focal length constitutes an indicator of the transporter’s ability to focus an incident light beam. Therefore, the transporter may include one or more imaging elements that may have the effect of a focusing lens. For example, the transporter may have one or more lenses, in particular one or more refractive lenses, and / or one or more convex mirrors. In this example, the focal length can be defined as the distance from the center of the thin refractive lens to the primary focal point of the thin lens. For focusing thin refractive lenses, such as convex or biconvex thin lenses, the focal length can be considered positive and can give the distance at which a beam of parallel light striking the thin lens as a transfer device can be focused into a single spot.Furthermore, the transfer device may include at least one wavelength-selective element, for example, at least one optical filter. Additionally, the transfer device may be designed to apply a predefined beam profile to electromagnetic radiation, for example, at the location of the sensor region, specifically within the sensor area. Any of the above embodiments of the transfer device can, in principle, be implemented individually or in any desired combination.

[0030] The transfer device may have an optical axis. As used herein, the term “optical axis of the transfer device” generally refers to the mirror-symmetric or rotationally symmetric axis of a lens or lens system. The transfer system may, as an example, include a beam path in which the elements of the transfer system within the beam path are arranged rotationally symmetrically with respect to the optical axis. Furthermore, one or more optical elements arranged within the beam path may be offset or tilted with respect to the optical axis. However, in this case, the optical axis may be defined sequentially by interconnecting the centers of the optical elements within the beam path, for example, by interconnecting the centers of the lenses, and in this context, the light sensor is not considered an optical element. The optical axis may generally refer to a beam path in which the camera may have a single beam path along which the light beam travels from an object to a light sensor, or it may have multiple beam paths. As an example, a single beam path may be given, or the beam path may be divided into two or more sub-beam paths. In the latter case, each sub-beam path may have its own optical axis. In the case of multiple light sensors, the light sensors may be arranged in one and identical beam path or partial beam path. Alternatively, however, the light sensors may also be arranged in different partial beam paths.

[0031] The transfer device may configure a coordinate system in which the longitudinal coordinates are aligned with the optical axis, and d is the spatial offset from the optical axis. The coordinate system may be a polar coordinate system in which the optical axis of the transfer device forms the z axis, and the distance from the z axis and the polar angle can be used as additional coordinates. Directions parallel or antiparallel to the z axis can be considered longitudinal directions, and coordinates aligned with the z axis can be considered longitudinal coordinates. Any direction perpendicular to the z axis can be considered transverse directions, and polar coordinates and / or polar angles can be considered transverse coordinates.

[0032] The camera is configured to determine at least one image of a scene, in particular a first image. As used herein, the term “scene” may refer to a spatial region. The scene may include a face being authenticated and its surrounding environment. The first image itself may include pixels, and the pixels of the image correlate to pixels in a matrix of sensor elements. Thus, when referring to a “pixel,” it may refer to a unit of image information generated by a single pixel of the sensor element, or to a single pixel of the sensor element directly. The first image is at least one two-dimensional image. As used herein, the term “two-dimensional image” may generally refer to an image that has information about horizontal coordinates, such as height and width dimensions. The first image may be an RGB (red-green-blue) image. The term “determine at least one first image” may refer to capturing and / or recording the first image.

[0033] The face detection step includes detecting a face in a first image by using at least one processing unit to identify at least one predefined or predetermined geometric feature characteristic of a face within the first image. Specifically, the face detection step includes detecting a face in a first image by using at least one processing unit to identify at least one predefined or predetermined geometric feature characteristic of a face within the first image.

[0034] Where further used herein, the term “processing unit” generally refers to any data processing device adapted to perform a specified operation, such as by using at least one processor and / or at least one application-specific integrated circuit. Thus, as an example, at least one processing unit may include software code stored therein, which includes a number of computer commands. The processing unit may provide one or more hardware elements for performing one or more of the specified operations, and / or provide one or more processors having software performed thereon for performing one or more of the specified operations. The operations may be performed by at least one processing unit, including evaluating an image. Thus, as an example, one or more instructions may be implemented in software and / or hardware. Thus, as an example, the processing unit may comprise one or more computers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), or programmable devices such as field-programmable gate arrays (FPGAs) configured to perform the evaluation described above. However, additionally or alternatively, the processing unit may also be fully or partially embodied in hardware. The processing unit and the camera may be integrated fully or partially into a single device. Therefore, generally speaking, the processing unit may also form part of the camera. Alternatively, the processing unit and the camera may be embodied as separate devices, either completely or partially.

[0035] The processing unit may be one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs), and / or one or more computers, preferably one or more microcomputers, and / or one or more data processing devices, such as microcontrollers, field-programmable gate arrays, or digital signal processors. Additional components may include data acquisition devices, such as one or more devices for receiving and / or pre-processing sensor signals, such as one or more pre-processing devices and / or one or more AD converters and / or one or more filters. Furthermore, the processing unit may include one or more measuring devices, such as one or more measuring devices for measuring current and / or voltage. Furthermore, the processing unit may include one or more data storage devices. Furthermore, the processing unit may include one or more interfaces, such as one or more wireless interfaces and / or one or more wired interfaces.

[0036] The processing unit can be configured to display, visualize, analyze, distribute, communicate, or further process information, such as information obtained by a camera. The processing unit can be connected to or incorporated into, for example, at least one of the following: a display, projector, monitor, LCD, TFT, loudspeaker, multi-channel sound system, LED pattern, or further visualization device. It can also be further connected to or incorporated into at least one of the following: a communication device or communication interface, connector, or port capable of sending encrypted or unencrypted information using one or more of the following: email, text message, telephone, Bluetooth, Wi-Fi, infrared, or internet interface, port, or connection. It can further be connected to or incorporate at least one of the following: a system-on-a-chip such as a processor, graphics processor, CPU, Open Multimedia Applications Platform (OMAP®), integrated circuit, product from Apple A-series or Samsung S3C2-series, microcontroller or microprocessor; one or more memory blocks such as ROM, RAM, EEPROM, or flash memory; a timing source such as an oscillator or phase-locked loop; a counter timer, real-time timer, or power-on-reset-generator, voltage regulator, power management circuit, or DMA controller. The individual units can further be connected to or integrated with an Internet of Things or Industry 4.0 type network by a bus such as an AMBA bus.

[0037] The processing unit may be connected by, or may have, further external interfaces or ports such as serial or parallel interfaces or ports, one or more of USB, Centronics Port, FireWire, HDMI®, Ethernet, Bluetooth, RFID, Wi-Fi, USART, or SPI, or analog interfaces or ports such as one or more of standardized interfaces or ports to further devices such as 2D camera devices using an ADC or DAC, or an RGB interface such as CameraLink. The processing unit may be further connected by one or more of inter-processor interfaces or ports, FPGA-to-FPGA interfaces, or serial or parallel interface ports. The processing unit may also be connected to one or more of optical disc drives, CD-RW drives, DVD+RW drives, flash drives, memory cards, disk drives, hard disk drives, solid state disks, or solid state hard disks.

[0038] The processing unit may be connected by or have one or more further external connectors, such as one or more of the following: phone connectors, RCA connectors, VGA connectors, male / female hermaphrodite connectors, USB connectors, HDMI connectors, 8P8C connectors, BCN connectors, IEC60320 C14 connectors, fiber optic connectors, D subminiature connectors, RF connectors, coaxial connectors, SCART connectors, and XLR connectors, and / or may incorporate at least one suitable socket for one or more of these connectors.

[0039] Detecting a face in a first image may involve identifying at least one predefined or predetermined geometric feature characteristic of the face. The term “facial geometric feature” as used herein is broad and should be given a common and idiomatic meaning to those skilled in the art, and should not be limited to any special or customized meaning. Specifically, the term may refer, without limitation, to at least one geometric-based feature describing the shape of a face and its components, particularly one or more of the nose, eyes, mouth, or eyebrows. The processing unit may have at least one database in which the facial geometric features are stored, such as in a lookup table. Techniques for identifying at least one predefined or predetermined geometric feature characteristic of a face are generally known to those skilled in the art. For example, face detection can be carried out as described in the literature “Deep face recognition: A survey” by Masi, Lacopo et al., 2018, 31st SIBGRAPI conference on graphics, patterns and images (SIBGRAPI), IEEE, 2018, the entire content of which is incorporated by reference.

[0040] The processing unit may be configured to perform at least one image analysis and / or image processing to identify geometric features. The image analysis and / or image processing may use at least one feature detection algorithm. Image analysis and / or image processing may include one or more of the following: filtering; selection of at least one region of interest; background correction; decomposition into color channels; decomposition into hue, saturation, and / or luminance channels; frequency decomposition; singular value decomposition; application of blob detector; application of corner detector; application of determinant of Hessian filter; application of principal curvature-based region detector; application of gradient position and direction histogram algorithm; application of oriented gradient descriptor histogram; application of edge detector; application of differential edge detector; application of Canny edge detector; application of Laplace operator of Gaussian filter; application of difference Gaussian filter; application of Sobel operator; application of Laplace operator; application of Schall operator; application of Prewitt operator; application of Roberts operator; application of Kirsch operator; application of high-pass filter; application of low-pass filter; application of Fourier transform; application of Radon transform; application of Hough transform; application of wavelet transform; thresholding; and generation of a binary image. The region of interest may be determined manually by the user or automatically, for example, by recognizing features in the first image.

[0041] Specifically, a skin detection step can be performed after the face detection step, which includes projecting at least one illumination pattern containing multiple illumination features onto the scene by using at least one illumination unit. However, embodiments are possible in which the skin detection step is performed before the face detection step.

[0042] As used herein, the term “illumination unit” may also be written as “illumination source” and may generally refer to at least one arbitrary device configured to produce at least one illumination pattern. An illumination unit may be configured to provide an illumination pattern for illuminating a scene. An illumination unit may be adapted to illuminate a scene directly or indirectly, where the illumination pattern is emitted by the surface of the scene, and in particular reflected or scattered, thereby directed at least partially towards a camera. An illumination unit may be configured to illuminate a scene, for example, by directing a light beam onto the scene (the scene reflects the light beam). An illumination unit may be configured to produce an illumination light beam for illuminating a scene.

[0043] The irradiation unit may comprise at least one light source. The irradiation unit may comprise multiple light sources. The irradiation unit may comprise an artificial irradiation source, in particular at least one laser source, and / or at least one incandescent lamp, and / or at least one semiconductor light source, such as at least one light-emitting diode, in particular organic and / or inorganic light-emitting diodes. As an example, the light emitted by the irradiation unit may have wavelengths of 300 to 1100 nm, particularly 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range, such as in the range of 780 nm to 3.0 μm, may be used. Specifically, light in the near-infrared region, particularly in the range of 700 nm to 1100 nm, to which silicon photodiodes are applicable, may be used.

[0044] The irradiation unit may be configured to generate at least one irradiation pattern in the infrared region. The irradiation feature may have a wavelength in the near-infrared (NIR) region. The irradiation feature can have a wavelength of approximately 940 nm. At this wavelength, there is no melanin absorption, so dark and light skin tones reflect light in almost the same way. However, other wavelengths in the NIR region are possible, such as one or more of 805 nm, 830 nm, 835 nm, 850 nm, 905 nm, and 980 nm. Furthermore, using light in the near-infrared region, the light is undetectable or only weakly detectable by the human eye, but still detectable by silicon sensors, especially standard silicon sensors.

[0045] The irradiation unit may be configured to emit light at a single wavelength. In other embodiments, the irradiation unit may be configured to emit light at multiple wavelengths to allow for additional measurements at other wavelength channels.

[0046] As used herein, the term “ray” generally refers to a line perpendicular to the wavefront of light that indicates the direction of energy flow. As used herein, the term “beam” generally refers to a collection of rays. Hereafter, the terms “ray” and “beam” are used synonymously. Where further used herein, the term “light beam” generally refers to a quantity of light, specifically a quantity of light traveling essentially in the same direction, including the possibility that the light beam has an expansion angle or divergence angle. A light beam can have spatial extent. Specifically, a light beam can have a non-Gaussian beam profile. The beam profile may be selected from the group consisting of trapezoidal beam profiles; triangular beam profiles; and conical beams. A trapezoidal beam profile may have a plateau region and at least one edge region. A light beam may be a Gaussian light beam or a linear combination of Gaussian light beams, as outlined in more detail below. However, other embodiments are also possible.

[0047] The irradiation unit may be at least one multiple beam light source, or may include a multiple beam light source. For example, the irradiation unit may include at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the irradiation unit may have at least one laser and / or laser source. Various types of lasers may be used, such as semiconductor lasers, double heterostructure lasers, external cavity lasers, separated and contained heterostructure lasers, quantum cascade lasers, dispersed Bragg reflector lasers, polariton lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, indium arsenide lasers, transistor lasers, diode-pumped lasers, dispersed feedback lasers, quantum well lasers, interband cascade lasers, gallium arsenide lasers, semiconductor ring lasers, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alternatively, non-laser light sources such as LEDs and / or light bulbs may be used. The illumination unit may include one or more diffractive optical elements (DOEs) adapted to generate an illumination pattern. For example, the illumination unit may be adapted to generate and / or project a point cloud, and may include one or more of the following: at least one digital photoprocessing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one array of light-emitting diodes; at least one array of laser light sources. Given their generally defined beam profiles and other characteristics of handling, the use of at least one laser source as the illumination unit is particularly preferred. The illumination unit may be integrated into the camera housing or separate from the camera.

[0048] Furthermore, the irradiation unit may be configured to emit modulated or unmodulated light. When multiple irradiation units are used, different irradiation units may have different modulation frequencies, which can later be used to distinguish between light beams, as outlined in more detail below.

[0049] One or more light beams generated by the illumination unit may generally propagate parallel to the optical axis or at an angle to the optical axis, for example, at an angle to the optical axis. The illumination unit may be configured such that one or more light beams propagate from the illumination unit toward the scene along the optical axis of the illumination unit and / or camera. For this purpose, the illumination unit and / or camera may include at least one reflective element, preferably at least one prism, to deflect the illumination light beams toward the optical axis. As an example, one or more light beams, such as laser light beams, and the optical axis may have an angle of less than 10°, preferably less than 5°, and even less than 2°. However, other embodiments are possible. Furthermore, one or more light beams may be on the optical axis or off the optical axis. As an example, one or more light beams may be parallel to the optical axis, or even coincide with the optical axis, at a distance of less than 10 mm, preferably less than 5 mm, and even less than 1 mm from the optical axis.

[0050] As used herein, the term “at least one illumination pattern” refers to at least one arbitrary pattern that includes at least one illumination feature adapted to illuminate at least a portion of a scene. As used herein, the term “illumination feature” refers to at least one feature that extends at least partially from the pattern. An illumination pattern may include a single illumination feature. An illumination pattern may include multiple illumination features. An illumination pattern may be selected from the group consisting of at least one dot pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; and at least one pattern including an arrangement of periodic or aperiodic features. An illumination pattern may include regular and / or constant and / or periodic patterns such as triangular patterns, rectangular patterns, hexagonal patterns, or even convex tile patterns. An illumination pattern may show at least one illumination feature selected from the group consisting of at least one dot; at least one line; at least two lines such as parallel or intersecting lines; at least one dot and one line; at least one arrangement of periodic or aperiodic features; and at least one feature of any shape. The illumination pattern may include at least one pattern selected from the group consisting of: at least one point pattern, in particular a pseudo-random point pattern; a random point pattern or a quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern containing at least one known feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern; at least one pattern containing a convex, uniform tiling; at least one line pattern containing at least one line; and at least one line pattern containing at least two lines, such as parallel or intersecting lines. For example, the illumination unit may be adapted to generate and / or project a point cloud. The illumination unit may include at least one optical projector adapted to generate a point cloud such that the illumination pattern contains multiple point patterns. The illumination pattern may include a periodic grid of laser spots.The irradiation unit may include at least one mask adapted to generate an irradiation pattern from at least one light beam generated by the irradiation unit.

[0051] The distance between two features of the illumination pattern, and / or the area of ​​at least one illumination feature, may depend on the circle of confusion in the image. As outlined above, the illumination unit may comprise at least one light source configured to produce at least one illumination pattern. Specifically, the illumination unit comprises at least one laser light source and / or at least one laser diode designated to produce laser radiation. The illumination unit may include at least one diffractive optical element (DOE). The illumination unit may include at least one point projector, such as at least one laser source and DOE, adapted to project at least one periodic point pattern. Where further used herein, the term “projecting at least one illumination pattern” may mean providing at least one illumination pattern for illuminating at least one scene.

[0052] The skin detection step includes determining at least one second image, also called a reflected image, using a camera. The method may include determining multiple second images. The reflected features of the multiple second images may be used for skin detection in step b) and / or for 3D detection in step c).

[0053] The second image includes multiple reflective features generated by the scene in response to illumination by an illumination feature. As used herein, the term “reflective feature” may specifically refer to a feature in the image plane generated by the scene in response to illumination, which is at least one illumination feature. Each reflective feature includes at least one beam profile, also called a reflective beam profile. As used herein, the term “beam profile” of a reflective feature may generally refer to at least one intensity distribution of a reflective feature, such as a light spot of a light sensor, as a function of pixels. The beam profile may be selected from the group consisting of trapezoidal beam profiles; triangular beam profiles; conical beam profiles; and linear combinations of Gaussian beam profiles.

[0054] The evaluation of the second image may include identifying the reflection features of the second image. The processing unit may be configured to perform at least one image analysis and / or image processing to identify the reflection features. The image analysis and / or image processing may use at least one feature detection algorithm. The image analysis and / or image processing may include: filtering; selection of at least one region of interest; formation of a difference image between the image generated by the sensor signal and at least one offset; inversion of the sensor signal by inverting the image generated by the sensor signal; formation of a difference image between images generated by the sensor signal at different times; background correction; decomposition into color channels; decomposition into hue; saturation; luminance channels; frequency decomposition; singular value decomposition; application of a blob detector; application of a corner detector; application of a Hessian filter determinant; application of a principal curvature-based region detector; application of a maximum stable extreme region detector; application of a generalized Hough transform; application of a ridge detector; application of an affine-invariant feature detector; application of an affine adaptation point of interest operator; application of a Harris affine region detector; application of a Hessian affine region detector; application of a scale-invariant feature transform. This may include one or more of the following: application of scale-space extreme value detectors; application of local feature detectors; application of accelerated robust feature algorithms; application of gradient position and direction histogram algorithms; application of oriented gradient descriptor histograms; application of Deriche edge detectors; application of differential edge detectors; application of spatiotemporal point of interest detectors; application of Moravec corner detectors; application of Canny edge detectors; application of Laplace operators for Gaussian filters; application of differential Gaussian filters; application of Sobel operators; application of Laplace operators; application of Schall operators; application of Prewitt operators; application of Roberts operators; application of Kirsch operators; application of high-pass filters; application of low-pass filters; application of Fourier transforms; application of Radon transforms; application of Huff transforms; application of wavelet transforms; thresholding; and generation of binary images. The region of interest may be determined manually by the user or automatically, for example, by recognizing features in an image generated by a light sensor.

[0055] For example, the illumination unit may be configured to generate and / or project a point cloud such that multiple illumination regions are generated on a photosensor, such as a CMOS detector. Furthermore, disturbances such as speckle and / or external light and / or multiple reflections may be present on the photosensor. The processing unit may be adapted to determine at least one region of interest, for example, one or more pixels illuminated by a light beam used to determine the longitudinal coordinates of each reflection feature, which are described in more detail below. For example, the processing unit may be adapted to perform filtering methods, such as blob analysis and / or edge filtering and / or object recognition methods.

[0056] The processing unit may be configured to perform at least one image correction, which may include at least one background subtraction. The processing unit may be adapted to remove the influence of background light from the beam profile, for example, by imaging without further illumination.

[0057] The processing unit can be configured to determine the beam profile of each reflective feature. As used herein, the term “determine the beam profile” means identifying and / or selecting at least one reflective feature provided by the optical sensor and evaluating at least one intensity distribution of the reflective feature. As an example, a region of a matrix may be used and evaluated to determine an intensity distribution, such as a three-dimensional or two-dimensional intensity distribution, along an axis or line passing through the matrix. As an example, the center of illumination by the optical beam may be determined by determining at least one pixel having the best illumination, and a cross-sectional axis may be selected through the center of illumination. The intensity distribution may be an intensity distribution as a function of coordinates along this cross-sectional axis passing through the center of illumination. Other evaluation algorithms are also possible.

[0058] The reflection feature may cover or extend over at least one pixel of the second image. For example, the reflection feature may cover or extend across multiple pixels. The processing unit may be configured to determine and / or select all pixels connected to and / or belonging to the reflection feature, e.g., the light spot. The processing unit determines the center of intensity,

number

number

[0059] The processing unit is configured to determine, by analyzing the beam profile, at least one first beam profile information for a reflection feature located within the image region of a second image corresponding to the image region of a first image containing the identified geometric feature. The method may include identifying the image region of the second image corresponding to the image region of the first image containing the identified geometric feature. Specifically, the method may include matching pixels of the first and second images and selecting pixels of the second image corresponding to the image region of the first image containing the identified geometric feature. The method may further include considering additional reflection features located outside the image region of the second image.

[0060] As used herein, the term “beam profile information” may refer to any information and / or properties derived from and / or associated with the beam profile of the reflectance features. The first beam profile information and the second beam profile information may be identical or different. For example, the first beam profile information may be intensity distribution, reflectance profile, intensity center, and material features. For skin detection in step b), beam profile analysis may be used. Specifically, beam profile analysis classifies materials using the reflectance properties of coherent light projected onto the surface of an object. The classification of materials may be carried out as described in WO2020 / 187719, EP application 20159984.2 filed on 28 February 2020 and / or EP application 20154961.5 filed on 31 January 2020, the full contents of which are incorporated by reference. Specifically, a periodic grid of laser spots, such as a hexagonal grid as described in EP application 20170905.2 filed on April 22, 2020, is projected, and the reflected images are recorded by a camera. Analysis of the beam profile of each reflected feature recorded by the camera may be performed by a feature-based method. Feature-based methods are described below. Feature-based methods can be used in combination with machine learning methods that enable parameterization of skin classification models. Alternatively, or in combination, skin can be classified using a convolutional neural network with the reflected images as input.

[0061] Other methods for authenticating a user's face are known from US2019 / 213309A1, for example. However, these methods use time-of-flight (ToF) sensors. The well-known operating principle of a ToF sensor is to measure the time it takes to transmit light and receive the reflected light. In contrast, the proposed beam profile analysis uses a projected illumination pattern. It is not possible to use such a projected pattern with a ToF sensor. Using an illumination pattern is advantageous, for example, in terms of covering, and therefore it is possible to consider various parts of the face. This can improve the reliability and security of user facial recognition.

[0062] The skin detection step may include determining at least one material property of the reflective feature from beam profile information by using a processing unit. Specifically, the processing unit is configured to identify a reflective feature produced by irradiating biological tissue, in particular human skin, if its reflective beam profile satisfies at least one predetermined or predefined criterion. As used herein, the term “at least one predetermined or predefined criterion” means at least one property and / or value suitable for distinguishing biological tissue, in particular human skin, from other materials. The predetermined or predefined criterion may be, or include, at least one predetermined or predefined value and / or threshold and / or threshold range that reference a material property. A reflective feature may be indicated as produced by biological tissue if its reflective beam profile satisfies at least one predetermined or predefined criterion. As used herein, the term “indicate” means any indication, such as an electronic signal and / or at least one visual or acoustic indication. The processing unit is otherwise configured to identify the reflective feature as non-skin. As used herein, the term “biological tissue” generally refers to biological material containing living cells. Specifically, the processing unit may be configured for skin detection. The term “identification” of being produced by biological tissue, in particular human skin, may mean determining and / or verifying whether the object under test or the surface under test is or contains biological tissue, in particular human skin, and / or distinguishing biological tissue, in particular human skin, from other tissues, in particular other surfaces. Methods according to the present invention may enable the distinction of human skin from one or more of inorganic tissues, metal surfaces, plastic surfaces, foams, paper, wood, displays, screens, and fabrics. Methods according to the present invention may enable the distinction of human biological tissue from artificial or inanimate surfaces.

[0063] The processing unit may be configured to determine the material properties m of the surface emitting the reflective features by evaluating the beam profile of the reflective features. As used herein, the term “material properties” refers to at least one arbitrary property of a material configured for characterizing and / or identifying and / or classifying the material. For example, material properties may be properties selected from the group consisting of roughness, depth of light transmission into the material, properties characterizing the material as a biological or non-biological material, reflectance, specular reflectance, diffuse reflectance, surface properties, measure of light transmission, scattering, specifically backscattering behavior, etc. At least one material property may be properties selected from the group consisting of scattering coefficient, light transmission, transparency, deviation from Lambertian surface reflection, speckle, etc. As used herein, the term “determine at least one material property” may mean assigning the material property to each reflective feature, in particular the detected face. The processing unit may include at least one database containing lists and / or tables, such as lookup lists and / or lookup tables, of predefined and / or predetermined material properties. A list and / or table of material properties can be determined and / or generated by performing at least one test measurement, for example, by performing material testing using a sample with known material properties. The list and / or table of material properties can be determined and / or generated at the manufacturer's site and / or by the user. Material properties may be further assigned to one or more material classifications, such as material name, material group such as biological or non-biological material, translucent or opaque material, metal or non-metal, skin or non-skin, fur or non-fur, carpet or non-carpet, reflective or non-reflective, specular or non-specular, foam or non-foam, hair or non-hair, roughness group, etc. A processing unit may include at least one database containing lists and / or tables containing material properties and associated material names and / or material groups.

[0064] The reflective properties of skin can be characterized by the simultaneous occurrence of direct reflection at the surface (like Lambertian reflection) and scattering below the surface (volume scattering). This results in a wider laser spot on the skin compared to the materials mentioned above.

[0065] The first beam profile information may be a reflection profile. For example, although we do not wish to be bound by this theory, human skin may have a reflection profile also called a backscatter profile, which includes a portion produced by back reflection from the surface, called surface reflection, and a portion produced by very diffuse reflection from light passing through the skin, called the diffuse back reflection. For the reflection profile of human skin, see "Lasertechnikinder Medizin:Grundlagen, Systeme, Anwendungen," "Wirkungvon Laserstrahlung auf Gewebe," 1991, pp. 10171-266, Juergen Eichler, Theo Seiler, Springer Verlag, ISBN 0939-0979. Surface reflection of the skin may increase as the wavelength increases toward the near-infrared. Furthermore, the transmission depth may increase as the wavelength increases from visible light to the near-infrared. The diffuse back reflection may increase as the transmission depth of light increases. These properties can be used to distinguish skin from other materials by analyzing its backscatter profile.

[0066] Specifically, the processing unit may be configured to compare the reflected beam profile with at least one predetermined and / or pre-recorded and / or pre-defined beam profile. The predetermined and / or pre-recorded and / or pre-defined beam profile may be stored in a table or lookup table, may be determined empirically, for example, and may be stored, as an example, in at least one data storage device of the detector. For example, the predetermined and / or pre-recorded and / or pre-defined beam profile may be determined at the initial startup of the apparatus performing the method according to the present invention. For example, the predetermined and / or pre-recorded and / or pre-defined beam profile may be stored in at least one data storage device of the processing unit or apparatus, for example, by software, specifically by an app downloaded from an app store, etc. Reflected features may be shown as generated by biological tissue when the reflected beam profile and the predetermined and / or pre-recorded and / or pre-defined beam profile are identical. The comparison may include superimposing the reflected beam profile and the predetermined or pre-defined beam profile so that their intensity centers match. The comparison may include determining the deviation between the reflected beam profile and a predetermined and / or predetermined recorded and / or predetermined beam profile, for example, the sum of squared point distances. The processing unit may be adapted to compare the determined deviation to at least one threshold, and if the determined deviation is lower than and / or equal to the threshold, the surface is indicated as biological tissue and / or the detection of biological tissue is confirmed. The threshold may be stored in a table or lookup table, may be determined empirically, for example, and may be stored in at least one data storage device of the processing unit.

[0067] Additionally or alternatively, the first beam profile information may be configured to apply at least one image filter to the image of the area. Where further used herein, the term “image” refers to a two-dimensional function f(x,y), where brightness and / or color values ​​are given for any x,y position in the image. The position may be discretized corresponding to the recording pixel. The brightness and / or color may be discretized corresponding to the bit depth of the light sensor. Where used herein, the term “image filter” refers to at least one mathematical operation applied to the beam profile and / or at least one particular region of the beam profile. Specifically, the image filter Ф maps the image f or the region of interest in the image to a real number Ф(f(x,y)) = φ, where φ represents a feature, in particular a material feature. The image may be affected by noise, and so may the features. Therefore, the features may be random variables. The features may follow a normal distribution. If the features do not follow a normal distribution, they may be transformed to follow a normal distribution by a Box-Cox Transformation or the like.

[0068] The processing unit applies at least one material-dependent image filter Φ2 to the image to extract at least one material feature φ 2m It may be configured to determine the material-dependent image filter. As used herein, the term "material-dependent" image filter refers to an image having a material-dependent output. The output of a material-dependent image filter is referred herein to as "material feature φ 2m " or "Material-dependent characteristics φ 2m This is indicated as follows. The material feature may be, or may include, at least one piece of information relating to at least one material property of the surface of the scene that generated the reflection feature.

[0069] The material-dependent image filter is a luminance filter; a spot shape filter; a squared norm gradient; a standard deviation; a smoothing filter such as a Gaussian filter or a median filter; a contrast filter based on gray level generation; an energy filter based on gray level generation; a uniformity filter based on gray level generation; a dissimilarity filter based on gray level generation; a low energy filter; a threshold region filter; or a linear combination thereof; or a linear combination of a luminance filter, a spot shape filter, a squared norm gradient, a standard deviation, a smoothing filter, an energy filter based on gray level generation, a uniformity filter based on gray level generation, a dissimilarity filter based on gray level generation, a low energy filter, or a threshold region filter, or |ρ Ф2other,Фm |≧0.40 of these linear combinations (Ф m is one or more of a luminance filter, a spot shape filter, a squared norm gradient, a standard deviation, a smoothing filter, an energy filter based on gray level generation, a uniformity filter based on gray level generation, a dissimilarity filter based on gray level generation, a low energy filter, or a threshold region filter, or a linear combination thereof) further correlated material-dependent image filter Ф 2other may be at least one filter selected from the group consisting of. Further material-dependent image filter Ф 2other is the material-dependent image filter Ф m one or more of, and |ρ Ф2other,Фm |≧0.60, preferably |ρ Ф2other,Фm |≧0.80 may be correlated by.

[0070] The material-dependent image filter can be at least one arbitrary filter Φ that passes the hypothesis test. As used herein, the term "passes the hypothesis test" refers to the fact that the null hypothesis H0 is rejected and the alternative hypothesis H1 is accepted. The hypothesis test may include verifying the material dependence of the image filter by applying the image filter to a predefined data set. The data set may include a plurality of beam profile images. As used herein, the term "beam profile image" refers to NB This refers to the sum of the Gaussian radial basis functions.

number

number

number

[0071] The values ​​of x and y are:

number

[0072] Next, each image f k , for the feature value φ corresponding to the filter Φ k This can be calculated,

number

number

number

number

number

number

[0073] Hypothesis testing may include determining the mean sum of squares between the following:

number

number

[0074] Here, I x This is a regularized incomplete beta function.

number

number

[0075] Below, we describe image filters assuming that the reflected image contains at least one reflected feature, in particular a spot image. The spot image f is a function

number

[0076] For example, a material-dependent image filter may be a luminance filter. A luminance filter can return the luminance measurement of a spot as a material feature. The material feature is,

number

number

number

[0077] For example, a material-dependent image filter may be a filter that has an output dependent on the spot shape. This material-dependent image filter can return a value correlated with the light transmittance of the material as a material feature. The light transmittance of the material affects the spot shape. The material feature is,

number

number

number

[0078] For example, the material-dependent image filter may be a square-norm gradient. This material-dependent image filter may return values ​​correlated with the soft-to-hard transition and / or roughness measurements of the spot as material features. The material features are,

number

number

[0079] In the formula, μ is

number

[0080] For example, the material-dependent image filter may be a smoothing filter such as a Gaussian filter or a median filter. In one embodiment of the smoothing filter, this image filter can refer to the observation that volume scattering exhibits less speckle contrast compared to diffuse scattering materials. This image filter can quantify the spot smoothness corresponding to the speckle contrast as a material feature. The material feature is,

number

number

number

number

[0081] The material characteristics of this filter are:

number

[0082] In the formula, Var represents the variance function.

[0083] For example, the image filter may be a gray level generation-based contrast filter. This material filter is a gray level generation matrix.

number

[0084] The material characteristics of a gray level generation-based contrast filter are:

number

[0085] For example, the image filter may be a gray level generation-based energy filter. This material filter is based on the gray level generation matrix defined above.

[0086] The material characteristics of the gray level generation-based energy filter are:

number

[0087] For example, the image filter may be a gray level generation-based uniformity filter. This material filter is based on the gray level generation matrix defined above. The material characteristics of the gray level generation-based uniformity filter are:

number

[0088] For example, the image filter may be a gray level generation-based dissimilarity filter. This material filter is based on the gray level generation matrix defined above. The material characteristics of the gray level generation-based dissimilarity filter are:

number

[0089] For example, the image filter may be a low-energy filter. This material filter has low vectors L5=[1,4,6,4,1] and E5=[-1,-2,0,-2,-1] and material L5(E5) T and E5 (L5) T It is based on image f. k These are the matrices:

number

[0090] Here, the material characteristics of the Rho energy filter are:

number

[0091] For example, a material-dependent image filter may be a threshold region filter. This material feature may relate two areas in the image plane. The first area Ω1 may be an area where the function f is greater than α times the maximum value of f. The second area Ω2 may be an area where the function f is less than α times the maximum value of f, but greater than a threshold of ε times the maximum value of f. Preferably, α may be 0.5 and ε may be 0.05. Due to speckle or noise, the areas may not simply correspond to the inner and outer circles of the spot center. For example, Ω1 may include speckle or unconnected areas of the outer circle. The material feature is,

number

[0092] The processing unit determines the material properties of the surface from which the reflection features were generated, by analyzing the material features φ. 2m The system may be configured to use at least one predetermined relationship between the material properties of the surface from which the reflection features were generated. The predetermined relationship may be one or more of empirical relationships, semi-empirical relationships, and analytically derived relationships. The processing unit may include at least one data storage device for storing the predetermined relationships, such as a lookup list or a lookup table.

[0093] While the feature-based approaches described above have sufficient accuracy to distinguish between skin and surface-only scattering materials, distinguishing between skin and carefully selected attack materials, including volume scattering, is more difficult. Step b) may involve using artificial intelligence, particularly convolutional neural networks. Using reflected images as input to a convolutional neural network may allow for the generation of a classification model with sufficient accuracy to distinguish between skin and other volume scattering materials. By selecting important regions in the reflected images, only physically valid information is passed to the network, thus requiring only a compact training dataset. Furthermore, a very compact network architecture can be generated.

[0094] Specifically, the skin detection step may utilize at least one parameterized skin classification model. The parameterized skin classification model may be configured to classify skin and other materials by using a second image as input. The skin classification model may be parameterized by using one or more of machine learning, deep learning, neural networks, or other forms of artificial intelligence. As used herein, the term “machine learning” is a broad term and should be given its usual and idiomatic meaning to those skilled in the art, and should not be limited to any special or customized meaning. Specifically, the term may refer, without limitation, to methods of using artificial intelligence (AI) for automated model building, particularly for the parameterization of models. The term “skin classification model” may refer to a classification model configured to distinguish human skin from other materials. The property characteristic for skin can be determined by applying an optimization algorithm to at least one optimization target of the skin classification model. The machine learning may be based on at least one neural network, particularly a convolutional neural network. The weights and / or topology of the neural network may be predetermined and / or predefined. Specifically, training of a skin classification model can be carried out using machine learning. The skin classification model may include at least one machine learning architecture and model parameters. For example, the machine learning architecture may be one or more of the following, or may include them: linear regression, logistic regression, random forest, naive Bayes classification, nearest neighbor method, neural network, convolutional neural network, generative adversarial network, support vector machine, or gradient boosting algorithm. The terms “training,” also called “learning,” as used herein are broad terms and should be given in a common and idiomatic sense to those skilled in the art, and should not be limited to any special or customized meaning.This term may specifically refer, without limitation, to the process of constructing a skin classification model, particularly the process of determining and / or updating the parameters of the skin classification model. The skin classification model may be at least partially data-driven. For example, the skin classification model may be based on experimental data, such as data determined by illuminating multiple humans and artificial objects such as masks and recording the reflection patterns. For example, training may involve using at least one training dataset, the training dataset including images of humans and multiple artificial objects having known material properties, particularly a second image.

[0095] The skin detection step may include using at least one 2D face and face landmark detection algorithm configured to provide at least two locations of characteristic points on a human face. For example, these locations may be the eye locations, the forehead, or the cheeks. The 2D face and face landmark detection algorithm can provide the locations of characteristic points on a human face, such as the eye locations. Because there are subtle differences in the reflections of different zones of the face (e.g., the forehead and cheeks), region-specific models can be trained. Preferably, at least one region-specific parameterized skin classification model is used in the skin detection step. The skin classification model may include multiple region-specific parameterized skin classification models, such as different regions, and / or the skin classification model may be trained using region-specific data, such as by filtering the images used for training. For example, two different regions may be used for training, such as from the inner corner of the eye to below the nose, and the forehead region may be used in particular if sufficient reflective features cannot be identified within this region. However, other regions are also possible.

[0096] A detected face is characterized as skin if its material properties correspond to at least one characteristic property of skin. The processing unit may be configured to identify reflective features as generated by irradiating biological tissue, particularly skin, if its corresponding material properties satisfy at least one predetermined or predefined criterion. A reflective feature may be identified as generated by human skin if its material properties indicate "human skin". A reflective feature is identified as generated by human skin if its material properties fall within at least one threshold and / or at least one range. The at least one threshold and / or range may be stored in a table or lookup table, may be determined empirically, for example, or may be stored in at least one data storage device of the processing unit. The processing unit is configured to identify reflective features as otherwise background. Thus, the processing unit may be configured to assign a material property (e.g., skin yes or no) to each projection spot.

[0097] The 3D detection step may be performed after the skin detection step and / or face detection step. However, other embodiments are also possible in which the 3D detection step is performed before the skin detection step and / or face detection step. The material properties are based on information regarding the longitudinal coordinate z, φ 2m After determining the vertical coordinate z in step d), so that it can be considered for evaluation, φ 2m This can be determined by evaluating the following:

[0098] The 3D detection step includes determining second beam profile information for at least four reflection features located within the image region of the second image corresponding to the image region of the first image containing the identified geometric features, by analyzing the beam profiles. The 3D detection step may also include determining second beam profile information for at least four of the reflection features by analyzing each beam profile. The second beam profile information may include the quotient Q of the beam profile area.

[0099] As used herein, the term “beam profile analysis” may generally refer to the evaluation of a beam profile and may include at least one mathematical operation and / or at least one comparison and / or at least one symmetrization and / or at least one filtering and / or at least one normalization. For example, beam profile analysis may include at least one of the following steps: histogram analysis, calculation of difference measurements, application of a neural network, and application of a machine learning algorithm. The processing unit may be configured to symmetrize and / or normalize and / or filter the beam profile in particular to remove noise or asymmetry from recordings at larger angles, edge recordings, etc. The processing unit may filter the beam profile by removing high spatial frequencies, such as by spatial frequency analysis and / or median filtering. Aggregation may be performed by averaging all intensities at the same distance to the center of the light spot intensity. The processing unit may be configured to normalize the beam profile to maximum intensity in particular to account for intensity differences due to recorded distances. The processing unit may be configured to remove the influence of background light from the beam profile, for example, by imaging without illumination.

[0100] The processing unit analyzes the beam profile of each reflection feature and determines at least one vertical coordinate z for reflection features located within the image region of the second image corresponding to the image region of the first image containing the identified geometric features. DPR The processing unit may be configured to determine the longitudinal coordinate z by using a so-called depth technique from photon ratio, also known as beam profile analysis. DPR It may be configured to determine the depth from photon ratio (DPR) technique, see WO2018 / 091649A1, WO2018 / 091638A1 and WO2018 / 091640A1, the full contents of which are included by reference.

[0101] The processing unit may be configured to determine at least one first area and at least one second area for each reflected beam profile of a reflected feature in at least one region of interest and / or for each of the reflected features(plural). The processing unit is configured to integrate the first and second areas.

[0102] Analysis of a single beam profile of a reflection feature may include determining at least one first area and at least one second area of ​​the beam profile. The first area of ​​the beam profile may be area A1, and the second area of ​​the beam profile may be area A2. The processing unit may be configured to integrate the first and second areas. The processing unit may be configured to derive a combined signal, in particular a quotient Q, by one or more of the following: dividing the integrated first and integrated second areas, dividing by a multiple of the integrated first and integrated second areas, or dividing by a linear combination of the integrated first and integrated second areas. The processing unit may be configured to determine at least two areas of the beam profile and / or to divide the beam profile into at least two segments having different areas of the beam profile, where overlap of areas is possible as long as the areas do not coincide. For example, the processing unit may be configured to determine multiple areas, such as two, three, four, five, or up to ten areas. The processing unit may be configured to divide the optical spot into at least two areas of the beam profile, and / or to divide the beam profile into at least two segments containing different areas of the beam profile. The processing unit may be configured to determine the integral of the beam profile across at least two of the areas. The processing unit may be configured to compare at least two of the determined integrals. Specifically, the processing unit may be configured to determine at least one first area and at least one second area of ​​the beam profile. As used herein, the term “area of ​​beam profile” generally refers to any region of the beam profile at the photosensor position used to determine the quotient Q. The first area and the second area of ​​the beam profile may be adjacent areas, overlapping areas, or both. The first area and the second area of ​​the beam profile do not have to be the same area.For example, the processing unit may be configured to divide the sensor area of ​​a CMOS sensor into at least two sub-regions, and the processing unit may be configured to divide the sensor area of ​​a CMOS sensor into at least one left portion and at least one right portion, and / or at least one upper portion and at least one lower portion, and / or at least one inner portion and at least one outer portion. Additionally or alternatively, the camera may include at least two optical sensors, the photosensitive areas of the first optical sensor and the second optical sensor may be arranged such that the first optical sensor determines a first area of ​​the beam profile of the reflective feature, and the second optical sensor determines a second area of ​​the beam profile of the reflective feature. The processing unit may be adapted to integrate the first and second areas. The processing unit may be configured to use at least one predetermined relationship between the quotient Q and the longitudinal coordinate to determine the longitudinal coordinate. The predetermined relationship may be one or more of empirical relationships, semi-empirical relationships, and analytically derived relationships. The processing unit may include at least one data storage device for storing predetermined relationships, such as a lookup list or a lookup table.

[0103] A first area of ​​the beam profile may substantially include edge information of the beam profile, a second area of ​​the beam profile may substantially include central information of the beam profile, and / or, the first area of ​​the beam profile may substantially include information about the left portion of the beam profile, and the second area of ​​the beam profile may substantially include information about the right portion of the beam profile. The beam profile may have a center, i.e., the center point of the maximum value of the beam profile and / or the center point of the plateau of the beam profile and / or the geometric center of the light spot, and a falling edge extending from the center. The second area may include the inner region of the cross section, and the first area may include the outer region of the cross section. As used herein, the term “substantially central information” generally means that the proportion of edge information is low compared to the proportion of central information, i.e., the proportion of intensity distribution corresponding to the center. Preferably, the central information has a proportion of edge information of less than 10%, more preferably less than 5%, and most preferably, the central information does not include edge content. As used herein, the term “substantially edge information” generally means that the proportion of central information is low compared to the proportion of edge information. Edge information may include information from the entire beam profile, particularly from the central and edge regions. Edge information has a proportion of less than 10%, preferably less than 5%, of central information, and more preferably, edge information does not include central information. If the beam profile is near the center or around it and substantially includes central information, at least one area of ​​the beam profile may be determined and / or selected as the second area of ​​the beam profile. If the beam profile includes at least a portion of the falling edge of the cross section, at least one area of ​​the beam profile may be determined and / or selected as the first area of ​​the beam profile. For example, the entire area of ​​the cross section may be determined as the first area.

[0104] Other selections for the first area A1 and the second area A2 may also be possible. For example, the first area may include the substantially outer region of the beam profile, and the second area may include the substantially inner region of the beam profile. For example, in the case of a two-dimensional beam profile, the beam profile may be divided into a left portion and a right portion, where the first area substantially includes the left portion of the beam profile, and the second area substantially includes the right portion of the beam profile.

[0105] Edge information may include information about the number of photons in a first area of ​​the beam profile, and center information may include information about the number of photons in a second area of ​​the beam profile. The processing unit may be configured to determine the area integral of the beam profile. The processing unit may be configured to determine the edge information by integration and / or addition of the first area. The processing unit may be configured to determine the center information by integration and / or addition of the second area. For example, the beam profile may be a trapezoidal beam profile, and the processing unit may be configured to determine the trapezoidal integral. Furthermore, if a trapezoidal beam profile is assumed, the determination of the edge signal and center signal may be replaced by equivalent evaluations utilizing the characteristics of the trapezoidal beam profile, such as determining the slope and position of the edges, determining the height of the central plateau, and deriving the edge signal and center signal by geometric considerations.

[0106] In one embodiment, A1 may correspond to the entire or complete area of ​​the feature point on the light sensor. A2 may be the central area of ​​the feature point on the light sensor. The central area may be a constant value. The central area may be smaller than the entire area of ​​the feature point. For example, in the case of a circular feature point, the central area may have a radius of 0.1 to 0.9, preferably 0.4 to 0.6, of the total radius of the feature point.

[0107] In one embodiment, the illumination pattern may include at least one line pattern. A1 may correspond to an area on the photosensor, particularly on the photosensitive area of ​​the photosensor, having the full line width of the line pattern. The line pattern on the photosensor may be enlarged and / or displaced compared to the line pattern of the illumination pattern so that the line width on the photosensor is amplified. In particular, in the case of a matrix of photosensors, the line width of the line pattern on the photosensor may change from one column to another. A2 may be the central area of ​​the line pattern on the photosensor. The line width of the central area may be a constant value, particularly corresponding to the line width of the illumination pattern. The line width of the central area may be smaller than the full line width. For example, the central area may have a line width of 0.1 to 0.9 of the full line width, preferably 0.4 to 0.6 of the full line width. The line pattern may be segmented on the photosensor. Each column of the matrix of the photosensor may include central intensity information of the central area of ​​the line pattern and edge intensity information of the region extending further outward from the central area of ​​the line pattern to the edge region.

[0108] In one embodiment, the irradiation pattern may include at least one dot pattern. A1 may correspond to an area having the total radius of the dots in the dot pattern on the light sensor. A2 may be the central area of ​​the dots in the dot pattern on the light sensor. The central area may be a constant value. The central area may have a radius corresponding to the total radius. For example, the central area may have a radius of 0.1 to 0.9 of the total radius, preferably 0.4 to 0.6 of the total radius.

[0109] The irradiation pattern may include both at least one dot pattern and at least one line pattern. Other embodiments are possible in addition to, or instead of, line and dot patterns.

[0110] The processing unit may be configured to derive the quotient Q by one or more of the following: dividing the integrated first area and the integrated second area; dividing by a multiple of the integrated first area and the integrated second area; or dividing by a linear combination of the integrated first area and the integrated second area.

[0111] The processing unit may be configured to derive the quotient Q by one or more of the following: dividing the first area by the second area, dividing the first area by a multiple of the second area, or dividing the first area by a linear combination of the second area.

number

[0112] Additionally or alternatively, the processing unit may be adapted to determine either or both central information or edge information from at least one slice or cut of the light spot. This can be achieved, for example, by replacing the surface integral of quotient Q with a line integral along the slice or cut. To improve accuracy, several slices or cuts passing through the light spot may be used for averaging. In the case of an elliptical spot profile, averaging across several slices or cuts may yield improved distance information.

[0113] For example, in the case of a light sensor having a matrix of pixels, the processing unit is: - Determine the pixel with the best sensor signal and form at least one central signal; - Evaluate the sensor signals of the matrix and form at least one sum signal; - Determining the quotient Q by combining the center signal and the sum signal; - The system may be configured to evaluate the beam profile by determining at least one longitudinal coordinate z of the object by evaluating the quotient Q.

[0114] As used herein, “sensor signal” generally refers to a signal generated by an optical sensor and / or at least one pixel of an optical sensor in response to illumination. Specifically, the sensor signal may be or include at least one electrical signal, such as at least one analog electrical signal and / or at least one digital electrical signal. More specifically, the sensor signal may be or include at least one voltage signal and / or at least one current signal. More specifically, the sensor signal may include at least one photocurrent. Furthermore, the raw sensor signal may be used, or a display device, optical sensor, or other element may be adapted to process or preprocess the sensor signal, such as by preprocessing by filtering, thereby generating a secondary sensor signal that can also be used as a sensor signal. The term “center signal” generally refers to at least one sensor signal that contains substantially central information of the beam profile. As used herein, the term “best sensor signal” refers to either or both of the local maximum value or maximum value of the region of interest. For example, the central signal may be the signal of the pixel with the best sensor signal among multiple sensor signals generated by pixels in the entire matrix or a region of interest within the matrix, the region of interest may be predetermined or determinable within the image generated by the pixels of the matrix. The central signal may originate from a single pixel or from a group of optical sensors, in the latter case, for example, the sensor signals of the group of pixels may be added, integrated, or averaged to determine the central signal. The group of pixels from which the central signal originates may be, for example, a group of adjacent pixels such as pixels located less than a predetermined distance from the actual pixel with the best sensor signal, or a group of pixels that generate sensor signals within a predetermined range from the best sensor signal. The group of pixels from which the central signal originates may be selected to be as large as possible to enable the maximum dynamic range. The processing unit may be adapted to determine the central signal by integrating multiple sensor signals, for example, multiple pixels around the pixel with the best sensor signal.For example, the beam profile may be a trapezoidal beam profile, and the processing unit may be adapted to determine the trapezoidal integral, in particular the integral of the trapezoidal plateau.

[0115] As described above, the central signal may generally be a single sensor signal, such as a sensor signal from a pixel at the center of a light spot; or a combination of multiple sensor signals, such as a combination of sensor signals originating from pixels at the center of a light spot; or a secondary sensor signal derived by processing sensor signals derived from one or more of the aforementioned possibilities. The determination of the central signal may be performed electronically, or entirely or partially by software, since the comparison of sensor signals can be performed fairly easily by conventional electronic equipment. Specifically, the central signal may be selected from a group consisting of: the best sensor signal; the average of a group of sensor signals within a predetermined tolerance range from the best sensor signal; the average of sensor signals from a group of pixels containing the pixel with the best sensor signal and a predetermined group of adjacent pixels; the sum of sensor signals from a group of pixels containing the pixel with the best sensor signal and a predetermined group of adjacent pixels; the sum of a group of sensor signals within a predetermined tolerance range from the best sensor signal; the average of a group of sensor signals exceeding a predetermined threshold; the sum of a group of sensor signals exceeding a predetermined threshold; the integral of sensor signals from a group of optical sensors containing the optical sensor with the best sensor signal and a predetermined group of adjacent pixels; the integral of a group of sensor signals within a predetermined tolerance range from the best sensor signal; and the integral of a group of sensor signals exceeding a predetermined threshold.

[0116] Similarly, the term “sum signal” generally refers to a signal that substantially contains edge information of the beam profile. For example, the sum signal can be derived by adding, integrating, or averaging sensor signals from the entire matrix or a region of interest within the matrix, where the region of interest is predetermined or determinable within the image generated by the optical sensors of the matrix. When summing, integrating, or averaging sensor signals, the actual optical sensors from which the sensor signals are generated may be excluded from, or included in, the addition, integration, or averaging. The processing unit may be adapted to determine the sum signal by integrating the signals from the entire matrix or a region of interest within the matrix. For example, the beam profile may be a trapezoidal beam profile, and the processing unit may be adapted to determine the integral of the entire trapezoid. Furthermore, if a trapezoidal beam profile is assumed, the determination of edge and center signals may be replaced by equivalent assessments utilizing the characteristics of the trapezoidal beam profile, such as determining the slope and position of the edges, determining the height of the central plateau, and deriving edge and center signals by geometric considerations.

[0117] Similarly, the center signal and edge signal can also be determined by using segments of the beam profile, such as a circular segment of the beam profile. For example, the beam profile can be divided into two segments by a dividing line or chord that does not pass through the center of the beam profile. Thus, one segment will substantially contain edge information and the other segment will substantially contain center information. For example, the edge signal may be further subtracted from the center signal to further reduce the amount of edge information in the center signal.

[0118] The quotient Q may be a signal generated by combining the center signal and the sum signal. Specifically, the decision may include one or more of the following: forming the quotient of the center signal and the sum signal, or vice versa; forming the quotient of a multiple of the center signal and a multiple of the sum signal, or vice versa; forming the quotient of a linear combination of the center signals and a linear combination of the sum signals, or vice versa. Additionally or alternatively, the quotient Q may include any signal or combination of signals that includes at least one information item relating to a comparison between the center signal and the sum signal.

[0119] As used herein, the term “longitudinal coordinate of a reflective feature” refers to the distance between a light sensor and a point in the scene emitting the corresponding illumination feature. The processing unit may be configured to use at least one predetermined relationship between the quotient Q and the longitudinal coordinate to determine the longitudinal coordinate. The predetermined relationship may be one or more of empirical relationships, semi-empirical relationships, and analytically derived relationships. The processing unit may include at least one data storage device for storing the predetermined relationships, such as a lookup list or a lookup table.

[0120] The processing unit may be configured to run a depth algorithm from at least one photon ratio that calculates distances for all zero-order and higher-order reflection features.

[0121] The 3D detection step may include determining at least one depth level from the second beam profile information of the reflection feature by using a processing unit.

[0122] The processing unit may be configured to determine a depth map of at least a portion of a scene by determining depth information for at least one reflective feature located within an image region of a second image corresponding to an image region of a first image containing identified geometric features. As used herein, the term “depth” or depth information may refer to the distance between an object and a light sensor and may be given by longitudinal coordinates. As used herein, the term “depth map” may refer to the spatial distribution of depth. The processing unit may be configured to determine depth information for reflective features by one or more of the following techniques: depth from photon ratio, structured light, beam profile analysis, time of flight, shape from motion, depth from focal point, triangulation, depth from defocus, stereo sensor. The depth map may be a thinly loaded depth map containing a small number of entries, or the depth may be a heavily loaded depth map containing a large number of entries.

[0123] A detected face is characterized as a 3D object if its depth level deviates from a predetermined or predefined depth level for a planar object. Step c) may include using 3D topology data of the face in front of the camera. The method may include determining curvature from at least four reflective features located within an image region of a second image corresponding to an image region of a first image containing the identified geometric features. The method may include comparing the curvature determined from the at least four reflective features with a predetermined or predefined depth level for a planar object. If the curvature exceeds the assumed curvature of the planar object, the detected face is characterized as a 3D object; otherwise, it is characterized as a planar object. The predetermined or predefined depth levels for planar objects may be stored in at least one data storage of the processing unit, such as a lookup list or lookup table. The predetermined or predefined levels for planar objects may be determined experimentally and / or theoretical levels for planar objects. The predetermined or predetermined depth levels of a planar object may be at least one limit value for at least one curvature and / or a range for at least one curvature.

[0124] The 3D features determined in step c) allow for the distinction between high-resolution photographs and 3D facial structures. Combining steps b) and c) can enhance the reliability of authentication against attacks. 3D features can be combined with material features to increase the security level. Since the same computation pipeline can be used to generate input data for skin classification and 3D point cloud generation, both characteristics can be computed from the same frame with less computation.

[0125] Preferably, an authentication step may be performed following steps a) to c). The authentication step may be performed in part after each of steps a) to c). Authentication may be terminated if no face is detected in step a), and / or if it is determined in step b) that the reflective features are not generated by skin, and / or if the depth map refers to a planar object in step c). The authentication step includes authenticating the detected face using at least one authentication unit if the face detected in step b) can be characterized as skin, and the face detected in step c) can be characterized as a 3D object.

[0126] Steps a) to d) can be carried out using at least one device, such as a mobile phone or smartphone, and access to the device is protected by facial recognition. Other devices are also possible, such as access control devices that control access to buildings, machinery, automobiles, etc. The method may include granting access to the device if the detected face is authenticated.

[0127] The method may include at least one registration step, in which a user of the device can be registered. As used herein, the term “registration” may refer to the process of registering and / or signing up and / or teaching in a user for subsequent use of the device. Typically, registration may be performed at the time of the first use of the device and / or when the device is started up. However, embodiments are possible in which multiple users are registered, for example, sequentially, so that registration may be performed and / or repeated at any time during use of the device. Registration may include generating a user account and / or user profile. Registration may include inputting and storing user data, in particular image data, through at least one user interface. Specifically, at least one 2D image of the user is stored in at least one database. The registration step may include imaging of at least one image of the user, in particular multiple images. Images may be recorded from different orientations and / or the user may change orientation. Furthermore, the registration step may include generating at least one 3D image and / or depth map of the user, which may be used in step d) for comparison. The database may be, for example, a processing unit database, and / or an external database such as a cloud database. The method includes identifying a user by comparing the user's 2D image with a first image. The method according to the present invention may enable a significant improvement in the presentation attack detection capability of biometric authentication methods. To improve overall authentication, in addition to the user's 2D image, 3D topological features as well as fingerprints, which are person-specific materials, can also be stored during the registration process. This allows for multi-factor authentication within a single device by using 2D, 3D, and material-derived features.

[0128] The method according to the present invention, using beam profile analysis technology, can reliably detect human skin and provide a concept that distinguishes it from reflections from attack materials designed to mimic faces by analyzing the reflection of a laser spot on the face, particularly the reflection of a laser spot in the NIR region. Furthermore, beam profile analysis simultaneously provides depth information by analyzing the same camera frame. Therefore, skin security functionality as well as 3D can be provided by the exact same technology.

[0129] Because two-dimensional facial images can be recorded simply by turning off laser irradiation, it is possible to establish a completely secure facial recognition pipeline that solves the above problems.

[0130] Shifting the laser wavelength to the NIR region makes the reflective properties of human skin more similar across ethnic origins. At a wavelength of 940 nm, the difference is minimized. Therefore, differences in ethnic origin do not affect skin authentication.

[0131] Since presentation attack detection (by skin classification) is provided by just one frame, time-consuming analysis of a series of frames may not be necessary. The time frame required to implement the complete method may be 500 milliseconds or less, preferably 250 milliseconds or less. However, embodiments in which skin detection is performed using multiple frames may be possible. Depending on the confidence in identifying reflective features in the second image and the speed of the method, the method may include sampling reflective features across several frames to achieve a more stable classification.

[0132] In addition to accuracy, execution speed and power consumption are also important requirements. Further limitations on the availability of computing resources may be introduced due to security considerations. For example, steps a) through d) may be performed in a secure zone of the processing unit to avoid software-based operations during program execution. The compact nature of the material detection network described above can solve this problem by exhibiting excellent execution time behavior in such a secure zone, whereas conventional PAD solutions require the inspection of multiple consecutive frames, resulting in high computational costs and long algorithmic response times.

[0133] A further aspect of the present invention is a computer program for facial recognition configured to cause a computer or computer network to fully or partially implement the method according to the present invention when executed on a computer or computer network, the computer program being configured to implement and / or perform at least steps a) to d) of the method according to the present invention. Specifically, the computer program may be stored on a computer-readable data carrier and / or computer-readable storage medium.

[0134] As used herein, the terms “computer-readable data carrier” and “computer-readable storage medium” may specifically refer to non-temporary data storage means such as hardware storage media on which computer executable instructions are stored. Specifically, the computer-readable data carrier or storage medium may be, or include, storage media such as random access memory (RAM) and / or read-only memory (ROM).

[0135] Therefore, specifically, one, more, or all of the method steps described above can be carried out by using a computer or computer network, preferably by using a computer program.

[0136] A further embodiment is a computer-readable storage medium that, when executed by a computer or computer network, includes instructions causing at least steps a) to d) of the method according to the present invention to be performed.

[0137] Further disclosed and proposed herein are data carriers having stored data structures that, after being loaded into the working memory or main memory of a computer or computer network, for example, a computer or computer network, can then perform methods according to one or more embodiments disclosed herein.

[0138] Further disclosed and proposed herein are computer program products having program code means stored in a machine-readable carrier for carrying out methods according to one or more embodiments disclosed herein when the program is executed on a computer or computer network. In this specification, a computer program product means a program as a tradable product. The product can generally exist in any form, such as in paper form or on a computer-readable data carrier and / or computer-readable storage medium. Specifically, computer program products can be distributed via data networks.

[0139] Finally, disclosed and proposed herein are modulated data signals containing instructions readable by a computer system or computer network for carrying out methods according to one or more embodiments disclosed herein.

[0140] Referring to the computer implementation aspects of the present invention, one or more method steps, or even all of the method steps, of the methods according to one or more embodiments disclosed herein can be carried out using a computer or computer network. Therefore, generally, any method step involving data provision and / or manipulation can be carried out using a computer or computer network. Generally, these method steps may include any method step except those typically requiring manual intervention.

[0141] Specifically, further disclosed herein are: - A computer or computer network comprising at least one processor, wherein the processor is adapted to carry out a method according to one of the embodiments described herein, - A computer-loadable data structure adapted to carry out the method according to one of the embodiments described herein while the data structure is being executed on a computer, - A computer program which, while the program is running on a computer, is adapted to carry out a method according to one of the embodiments described herein, - A computer program which includes programming means for carrying out a method according to one of the embodiments described herein while the computer program is running on a computer or computer network, - A computer program comprising a programming means according to a prior embodiment, wherein the programming means is stored in a computer-readable storage medium, and - A storage medium wherein a data structure is stored in the storage medium and is adapted to carry out a method according to one of the embodiments described herein after the data structure has been loaded into the main storage and / or working storage of a computer or computer network, - A computer program product having program code means, wherein the program code means can be stored in a storage medium or is stored in a storage medium so as to carry out a method according to one of the embodiments described herein when the program code means is executed on a computer or computer network, That is the case.

[0142] In a further embodiment, a mobile device is disclosed comprising at least one camera, at least one illumination unit, and at least one processing unit. The mobile device is configured to perform at least steps a) to c) and optionally step d) of the facial recognition method according to the present invention. Step d) can be performed by using at least one authentication unit. The authentication unit may be a unit of the mobile device or an external authentication unit. For definitions and embodiments of the mobile device, please refer to the definitions and embodiments described with respect to the method.

[0143] In a further aspect of the present invention, the use of a method according to the present invention, such as one or more embodiments given above or provided in more detail below, is proposed for the purpose of detecting biometric presentation attacks.

[0144] Overall, the following embodiments are considered preferred in the context of the present invention: Embodiment 1: The following steps: a) at least one face detection step, the face detection step comprising determining at least one first image by using at least one camera, the first image comprising at least one two-dimensional image of a scene that is believed to contain a face, and the face detection step comprising detecting a face in the first image by using at least one processing unit to identify at least one predefined or predetermined geometric feature characteristic of a face in the first image; b) At least one skin detection step, the skin detection step comprising projecting at least one illumination pattern including a plurality of illumination features onto the scene by using the at least one illumination unit, and determining at least one second image using the at least one camera, the second image including a plurality of reflective features generated by the scene in response to illumination by the illumination features, each of the reflective features including at least one beam profile, the skin detection step comprising determining at least one first beam profile information of the reflective features located in an image region of the second image corresponding to an image region of the first image including identified geometric features by analysis of the beam profiles, and determining at least one material property of the reflective features from the first beam profile information by using the processing unit, wherein the detected face is characterized as skin if the material property corresponds to at least one property characteristic of skin; c) at least one 3D detection step, the 3D detection step comprising: determining at least four second beam profile information of the reflection feature located within the image region of the second image corresponding to the image region of the first image containing the identified geometric feature by analyzing beam profiles; and determining at least one depth level from the second beam profile information of the reflection feature by using the processing unit, wherein the detected face is characterized as a 3D object if the depth level deviates from a predetermined or predefined depth level of a planar object; d) at least one authentication step, the authentication step comprising authenticating the detected face by using at least one authentication unit when the detected face is characterized as skin in step b) and the detected face is characterized as a 3D object in step c), Facial recognition methods, including those mentioned above.

[0145] Embodiment 2: The method of the prior embodiment, wherein steps a) to d) are performed by at least one device, access to the device is secured by using facial recognition, and the method includes granting access to the device when the detected face is authenticated.

[0146] Embodiment 3: The method according to the preceding embodiment, comprising at least one registration step, in which a user of the device is registered, at least one 2D image of the user is stored in at least one database, and the method identifies the user by comparing the 2D image of the user with a first image.

[0147] Embodiment 4: The method according to any one of the preceding embodiments, wherein the skin detection step uses at least one parameterized skin classification model, the parameterized skin classification model is configured to classify skin and other materials by using the second image as input.

[0148] Embodiment 5: The method according to the preceding embodiment, wherein the skin classification model is parameterized by using machine learning, and the skin-specific characteristics are determined by applying an optimization algorithm with respect to at least one optimization target of the skin classification model.

[0149] Embodiment 6: The method according to any one of the two preceding embodiments, wherein the skin detection step includes using at least one 2D face and face landmark detection algorithm configured to provide at least two locations of characteristic points of a human face, and the skin detection step uses at least one region-specific parameterized skin classification model.

[0150] Embodiment 7: The method according to any one of the prior embodiments, wherein the irradiation pattern includes a periodic grid of laser spots.

[0151] Embodiment 8: The method according to any one of the prior embodiments, wherein the irradiation feature has a wavelength in the near-infrared (NIR) region.

[0152] Embodiment 9: The irradiation feature is the method according to the prior embodiment, having a wavelength of 940 nm.

[0153] Embodiment 10: The method according to any one of the prior embodiments, wherein a plurality of second images are determined, and the reflective features of the plurality of second images are used for skin detection in step b) and / or for 3D detection in step c).

[0154] Embodiment 11: The method according to any one of the prior embodiments, wherein the camera is at least one near-infrared camera or comprises at least one near-infrared camera.

[0155] Embodiment 12: A computer program for facial recognition, configured to cause a computer or computer network to fully or partially perform the method described in any one of the prior embodiments when run on a computer or computer network, wherein the computer program is configured to perform and / or execute at least steps a) to d) of the method described in any one of the prior embodiments.

[0156] Embodiment 13: A computer-readable storage medium that, when executed by a computer or computer network, causes an instruction to perform at least steps a) to d) of the method described in any one of the preceding embodiments that reference the method.

[0157] Embodiment 14: A mobile device comprising at least one camera, at least one illumination unit, and at least one processing unit, wherein the mobile device is configured to perform at least steps a) to c) and optionally step d) of a facial recognition method described in any one of the preceding embodiments that reference the method.

[0158] Embodiment 15: Use of the method according to any one of the prior embodiments for detecting a biometric presentation attack. [Brief explanation of the drawing]

[0159] Further optional details and features of the present invention will become apparent from the following description of preferred exemplary embodiments, in conjunction with the dependent claims. In this context, certain features may be implemented alone or in combination with other features. The present invention is not limited to exemplary embodiments. Exemplary embodiments are schematically shown in the figures. The same reference numerals in the individual figures refer to the same element or an element having the same function, or an element corresponding to one another with respect to function.

[0160] Specifically, in the drawings: [Figure 1] This figure shows one embodiment of the facial recognition method according to the present invention. [Figure 2] This figure shows one embodiment of a mobile device according to the present invention. [Figure 3] This figure shows the experimental results. [Modes for carrying out the invention]

[0161] Detailed description of the embodiment Figure 1 shows a flowchart of the facial recognition method according to the present invention. Facial recognition may refer to verifying that a recognized object or part of a recognized object is a human face. Specifically, authentication may include distinguishing a genuine human face from attack material created to mimic a face. Authentication may include verifying the identity of each user and / or assigning an identity to a user. Authentication may include generating and / or providing identification information to other devices, such as at least one authentication device for authenticating access to a mobile device, machine, automobile, building, etc. The identification information may be certified by authentication. For example, the identification information may be at least one identification token and / or include at least one identification token. If authentication is successful, it is verified that the recognized object or part of a recognized object is a genuine face and / or the object, in particular, the identity of the user.

[0162] This method involves the following steps: a) (Reference No. 110) at least one face detection step, the face detection step comprising determining at least one first image by using at least one camera 112, the first image comprising at least one two-dimensional image of a scene that is believed to contain a face, and the face detection step comprising detecting a face in the first image by using at least one processing unit 114 to identify at least one predefined or predetermined geometric feature characteristic of a face in the first image; b) (Reference No. 116) At least one skin detection step, the skin detection step comprising projecting at least one illumination pattern including a plurality of illumination features onto a scene by using at least one illumination unit 118, and determining at least one second image using at least one camera 112, the second image including a plurality of reflective features generated by the scene in response to illumination by the illumination features, each of the reflective features including at least one beam profile, the skin detection step comprising determining at least one first beam profile information of a reflective feature located in an image region of the second image corresponding to an image region of the first image including identified geometric features, by analyzing the beam profile, and determining at least one material property of a reflective feature from the first beam profile information by using a processing unit 114, wherein the detected face is characterized as skin if the material property corresponds to a characteristic property of at least one of the skin; c) (Reference No. 120) At least one 3D detection step, the 3D detection step comprising: determining at least four second beam profile information of a reflection feature located within an image region of a second image corresponding to an image region of a first image containing identified geometric features by analyzing beam profiles; and determining at least one depth level from the second beam profile information of the reflection feature by using a processing unit 114, wherein the detected face is characterized as a 3D object if the depth level deviates from a predetermined or predefined depth level of a planar object; d) (Reference No. 122) at least one authentication step, the authentication step comprising authenticating the detected face by using at least one authentication unit, where the face detected in step b) is characterized as skin and the face detected in step c) is characterized as a 3D object, Includes.

[0163] The method steps may be performed in a predetermined order, or in a different order. Furthermore, there may be one or more additional method steps that are not listed. In addition, one, more than one, or even all of the method steps may be repeated.

[0164] Camera 112 may refer to a device having at least one image element configured to record or capture spatially resolved one-dimensional, two-dimensional, or even three-dimensional optical data or information. Camera 112 may be a digital camera. For example, camera 112 may comprise at least one camera chip, such as at least one CCD chip and / or at least one CMOS chip, configured to record an image. Camera 112 may be at least one near-infrared camera, or may comprise a near-infrared camera. The image may relate to data recorded by using camera 112, such as multiple electronic readings from image elements, such as pixels of the camera chip. Camera 112 may include one or more optical elements, such as one or more lenses, in addition to at least one camera chip or image chip. For example, camera 112 may be a fixed-focus camera having at least one lens fixedly adjusted relative to the camera. Alternatively, however, camera 112 may comprise one or more variable lenses that can be adjusted automatically or manually.

[0165] Camera 112 may be the camera of a mobile device 124 such as a notebook computer, tablet, or, more specifically, a mobile phone such as a smartphone. Specifically, camera 112 may be part of a mobile device 124 that includes at least one camera 112 in addition to one or more data processing devices such as one or more data processors. However, other cameras are also possible. Mobile device 124 may refer to a mobile electronic device, more specifically a mobile communication device such as a mobile phone or smartphone. Additionally or alternatively, mobile device 124 may refer to a tablet computer or other type of portable computer. One embodiment of the mobile device according to the present invention is shown in Figure 2.

[0166] Specifically, the camera 112 may be at least one photosensor 126 having at least one photosensitive area, or may have a photosensor 126. Specifically, the photosensor 126 may be at least one photodetector, preferably an inorganic photodetector, more preferably an inorganic semiconductor photodetector, most preferably a silicon photodetector, or may include such a detector. Specifically, the photosensor 126 may have sensitivity in the infrared spectral range. The photosensor 126 may include at least one sensor element including a matrix of pixels. All pixels of the matrix, or at least one group of photosensors in the matrix, may be specifically identical. A group of identical pixels in the matrix may be specifically provided for different spectral ranges, or all pixels may be identical with respect to spectral sensitivity. Furthermore, the pixels may be identical in size and / or with respect to their electronic or photoelectronic properties. Specifically, the photosensor 126 may be an array of at least one inorganic photodiode having sensitivity in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers, or may include such arrays. Specifically, the light sensor 126 may have sensitivity in the near-infrared region, particularly in the range of 700 nm to 1100 nm, where silicon photodiodes are applicable. The infrared light sensor that can be used in the light sensor may be a commercially available infrared light sensor, such as the infrared light sensor sold under the brand name Hertzstueck® by trinamX GmbH, D-67056 Ludwigshafen am Rhein, Germany. Thus, as an example, the light sensor 126 may include at least one photovoltaic light sensor, more preferably at least one semiconductor photodiode selected from the group consisting of Ge photodiodes, InGaAs photodiodes, extended InGaAs photodiodes, InAs photodiodes, InSb photodiodes, and HgCdTe photodiodes.Additionally or alternatively, the photosensor may include at least one exogenous photovoltaic photosensor, more preferably at least one semiconductor photodiode selected from the group consisting of Ge:Au photodiode, Ge:Hg photodiode, Ge:Cu photodiode, Ge:Zn photodiode, Si:Ga photodiode, and Si:As photodiode. Additionally or alternatively, the photosensor 126 may include at least one photoconductive sensor, such as a PbS or PbSe sensor, a bolometer, preferably a bolometer selected from the group consisting of a VO bolometer and an amorphous Si bolometer.

[0167] Specifically, the photosensor 126 may have sensitivity in the near-infrared region. More specifically, the photosensor 126 may have sensitivity in the near-infrared region, particularly in the range of 700 nm to 1000 nm, where silicon photodiodes are applicable. Specifically, the photosensor 126 may have sensitivity in the infrared spectral range, specifically in the range of 780 nm to 3.0 μm. For example, the photosensor 126 may be, or include, at least one element selected from the group consisting of CCD sensor elements, CMOS sensor elements, photodiodes, photocells, photoconductors, phototransistors, or any combination thereof. Any other type of photosensitive element may be used. Photosensitive elements can generally be made entirely or partially from inorganic materials and / or entirely or partially from organic materials. Most commonly, one or more commercially available photodiodes, such as inorganic semiconductor photodiodes, may be used.

[0168] Camera 112 may further include at least one transfer device (not shown here). Camera 112 may include a transfer device such as at least one lens and / or at least one lens system, and at least one optical element selected from the group consisting of at least one diffractive optical element. The transfer device may be adapted to guide the light beam to the light sensor 126. Specifically, the transfer device may include one or more of the following: at least one lens, e.g., at least one lens selected from the group consisting of at least one adjustable focus lens, at least one aspherical lens, at least one spherical lens, and at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflection element, preferably at least one mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or beam splitting mirror; and at least one multi-lens system. The transfer device may have a focal length. Thus, the focal length constitutes an indicator of the transfer device's ability to focus the incident light beam. Thus, the transfer device may include one or more imaging elements that may have a focusing lens effect. For example, the transfer device may have one or more lenses, in particular one or more refractive lenses, and / or one or more convex mirrors. In this example, the focal length can be defined as the distance from the center of the thin refractive lens to the principal focus of the thin lens. In the case of a focusing thin refractive lens, such as a convex or biconvex thin lens, the focal length can be considered positive and can give a distance at which a beam of parallel light striking the thin lens as a transfer device can be focused into a single spot. Furthermore, the transfer device may include at least one wavelength-selective element, for example, at least one optical filter. Furthermore, the transfer device may be designed to apply a predefined beam profile to electromagnetic radiation, for example, at the location of the sensor area, specifically in the sensor region. Any of the above embodiments of the transfer device can, in principle, be realized individually or in any desired combination.

[0169] The transfer device may have an optical axis. The transfer device may constitute a coordinate system in which the longitudinal coordinates are aligned with the optical axis and d is a spatial offset from the optical axis. The coordinate system may be a polar coordinate system in which the optical axis of the transfer device forms the z axis, and the distance from the z axis and the polar angle can be used as additional coordinates. A direction parallel or antiparallel to the z axis can be considered a longitudinal direction, and coordinates aligned with the z axis can be considered longitudinal coordinates. Any direction perpendicular to the z axis can be considered a transverse direction, and polar coordinates and / or polar angles can be considered transverse coordinates.

[0170] Camera 112 is configured to determine at least one image of a scene, in particular a first image. The scene may refer to a spatial region. The scene may include a face being authenticated and the surrounding environment. The first image itself may include pixels, and the pixels of the image correlate to pixels in a matrix of sensor elements. The first image is generally at least one two-dimensional image having information about lateral coordinates, such as height and width dimensions.

[0171] The face detection step 110 includes detecting a face in a first image by using at least one processing unit 114 to identify at least one predefined or predetermined geometric feature of a face in the first image. For example, at least one processing unit 114 may include software code stored therein, which includes a number of computer commands. The processing unit 114 may provide one or more hardware elements for performing one or more specified operations, and / or provide one or more processors having software performed thereon for performing one or more specified operations. Operations including evaluating an image may be performed by at least one processing unit 114. Therefore, for example, one or more instructions may be implemented in software and / or hardware. Therefore, for example, the processing unit 114 may comprise one or more computers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), or programmable devices such as field-programmable gate arrays (FPGAs) configured to perform the evaluation described above. However, additionally or alternatively, the processing unit may also be fully or partially embodied in hardware. The processing unit 114 and the camera 112 may be integrated completely or partially into a single device. Therefore, generally, the processing unit 114 may also form part of the camera 112. Alternatively, the processing unit 114 and the camera 112 may be embodied completely or partially as separate devices.

[0172] Detecting a face in a first image may involve identifying at least one predefined or predetermined geometric feature of the face. A geometric feature of a face may be at least one geometry-based feature describing the shape of the face and one or more of its components, particularly the nose, eyes, mouth, or eyebrows. The processing unit 114 may have at least one database in which the geometric features of faces are stored, such as in a lookup table. Techniques for identifying at least one predefined or predetermined geometric feature of a face are generally known to those skilled in the art. For example, face detection can be carried out as described in the literature "Deep face recognition: A survey" by Masi, Lacopo et al., 2018, 31st SIBGRAPI conference on graphics, patterns and images (SIBGRAPI), IEEE, 2018, the entire content of which is incorporated by reference.

[0173] The processing unit 114 may be configured to perform at least one image analysis and / or image processing to identify geometric features. The image analysis and / or image processing may use at least one feature detection algorithm. Image analysis and / or image processing may include one or more of the following: filtering; selection of at least one region of interest; background correction; decomposition into color channels; decomposition into hue, saturation, and / or luminance channels; frequency decomposition; singular value decomposition; application of blob detector; application of corner detector; application of determinant of Hessian filter; application of principal curvature-based region detector; application of gradient position and direction histogram algorithm; application of oriented gradient descriptor histogram; application of edge detector; application of differential edge detector; application of Canny edge detector; application of Laplace operator of Gaussian filter; application of difference Gaussian filter; application of Sobel operator; application of Laplace operator; application of Schall operator; application of Prewitt operator; application of Roberts operator; application of Kirsch operator; application of high-pass filter; application of low-pass filter; application of Fourier transform; application of Radon transform; application of Hough transform; application of wavelet transform; thresholding; and generation of a binary image. The region of interest may be determined manually by the user or automatically, for example, by recognizing features in the first image.

[0174] Specifically, a skin detection step 116 can be performed after the face detection step 110, which includes projecting at least one illumination pattern containing multiple illumination features onto the scene by using at least one illumination unit 118. However, embodiments are possible in which the skin detection step 116 is performed before the face detection step 110.

[0175] The illumination unit 118 may be configured to provide an illumination pattern for illuminating a scene. The illumination unit 118 may be adapted to illuminate a scene directly or indirectly, such that the illumination pattern is emitted by the surface of the scene, particularly reflected or scattered, and thereby directed at least partially towards the camera. The illumination unit 118 may be configured to illuminate a scene, for example, by directing a light beam towards the scene, which reflects the light beam. The illumination unit 118 may be configured to generate an illumination light beam for illuminating a scene.

[0176] The irradiation unit 118 may have at least one light source. The irradiation unit 118 may have multiple light sources. The irradiation unit 118 may have an artificial irradiation source, in particular at least one laser source and / or at least one incandescent lamp and / or at least one semiconductor light source, for example at least one light-emitting diode, in particular organic and / or inorganic light-emitting diodes. The irradiation unit 118 may be configured to generate at least one irradiation pattern in the infrared region. The irradiation feature may have a wavelength in the near-infrared (NIR) region. The irradiation feature may have a wavelength of about 940 nm. At this wavelength, there is no absorption by melanin, so dark and light colors reflect light almost equally. However, other wavelengths in the NIR region are possible, such as one or more of 805 nm, 830 nm, 835 nm, 850 nm, 905 nm, and 980 nm. Furthermore, using light in the near-infrared region, the light is undetectable or only weakly detected by the human eye, but still detectable by silicon sensors, in particular standard silicon sensors.

[0177] The irradiation unit 118 may be at least one multi-beam light source, or may include a multi-beam light source. For example, the irradiation unit 118 may include at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the irradiation unit 118 may have at least one laser and / or laser source. Various types of lasers may be used, such as semiconductor lasers, double heterostructure lasers, external cavity lasers, separated-containment heterostructure lasers, quantum cascade lasers, dispersed Bragg reflector lasers, polariton lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, indium arsenide lasers, transistor lasers, diode-pumped lasers, dispersed feedback lasers, quantum well lasers, interband cascade lasers, gallium arsenide lasers, semiconductor ring lasers, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alternatively, non-laser light sources such as LEDs and / or light bulbs may be used. The illumination unit 118 may include one or more diffractive optical elements (DOEs) adapted to generate an illumination pattern. For example, the illumination unit 118 may be adapted to generate and / or project a point cloud, and for example, the illumination unit 118 may include one or more of the following: at least one digital photoprocessing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one array of light-emitting diodes; at least one array of laser light sources. Given their generally defined beam profiles and other characteristics of handling, the use of at least one laser source as the illumination unit 118 is particularly preferred. The illumination unit 118 may be integrated into the housing of the camera 112 or may be separate from the camera 112.

[0178] The illumination pattern includes at least one arbitrary pattern which includes at least one illumination feature adapted to illuminate at least a portion of the scene. The illumination pattern may include a single illumination feature. The illumination pattern may include multiple illumination features. The illumination pattern may be selected from the group consisting of at least one dot pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; and at least one pattern which includes an arrangement of periodic or aperiodic features. The illumination pattern may include regular and / or constant and / or periodic patterns such as triangular patterns, rectangular patterns, hexagonal patterns, or even convex tile patterns. The illumination pattern may show at least one illumination feature selected from the group consisting of at least one point; at least one line; at least two lines such as parallel or intersecting lines; at least one point and one line; at least one arrangement of periodic or aperiodic features; and at least one feature of any shape. The illumination pattern may include at least one pattern selected from the group consisting of: at least one point pattern, in particular a pseudo-random point pattern; a random point pattern or quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern containing at least one known feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern; at least one pattern containing a convex, uniform tiling; at least one line pattern containing at least one line; and at least one line pattern containing at least two lines, such as parallel or intersecting lines. For example, the illumination unit 118 may be adapted to generate and / or project a point cloud. The illumination unit 118 may include at least one optical projector adapted to generate a point cloud such that the illumination pattern may include multiple point patterns. The illumination pattern may include a periodic grid of laser spots. The illumination unit 118 may include at least one mask adapted to generate an illumination pattern from at least one optical beam generated by the illumination unit 118.

[0179] The skin detection step 116 includes determining at least one second image, also called a reflected image, using the camera 112. The method may include determining a plurality of second images. The reflected features of the plurality of second images may be used for skin detection in step b) and / or for 3D detection in step c). The reflected features may specifically be features in the image plane generated by the scene in response to illumination, which are at least one illumination features. Each of the reflected features includes at least one beam profile, also called a reflected beam profile. The beam profile of a reflected feature may generally refer to at least one intensity distribution of the reflected feature, such as a light spot of a photosensor, as a function of pixels. The beam profile may be selected from the group consisting of trapezoidal beam profiles; triangular beam profiles; conical beam profiles and linear combinations of Gaussian beam profiles.

[0180] The evaluation of the second image may include identifying the reflection features of the second image. The processing unit 114 may be configured to perform at least one image analysis and / or image processing to identify the reflection features. The image analysis and / or image processing may use at least one feature detection algorithm. The image analysis and / or image processing may include: filtering; selection of at least one region of interest; formation of a difference image between the image generated by the sensor signal and at least one offset; inversion of the sensor signal by inverting the image generated by the sensor signal; formation of a difference image between images generated by the sensor signal at different times; background correction; decomposition into color channels; decomposition into hue; saturation; luminance channel; frequency decomposition; singular value decomposition; application of a blob detector; application of a corner detector; application of a Hessian filter determinant; application of a principal curvature-based region detector; application of a maximum stable extreme region detector; application of a generalized Hough transform; application of a ridge detector; application of an affine-invariant feature detector; application of an affine adaptation point of interest operator; application of a Harris affine region detector; application of a Hessian affine region detector; application of a scale-invariant feature transform. This may include one or more of the following: application of a scale-space extreme value detector; application of a local feature detector; application of a high-speed robust feature algorithm; application of a gradient position and direction histogram algorithm; application of a oriented gradient descriptor histogram; application of a Deriche edge detector; application of a differential edge detector; application of a spatiotemporal point of interest detector; application of a Moravec corner detector; application of a Canny edge detector; application of a Gaussian filter with the Laplace operator; application of a differential Gaussian filter; application of the Sobel operator; application of the Laplace operator; application of the Schall operator; application of the Prewitt operator; application of the Roberts operator; application of the Kirsch operator; application of a high-pass filter; application of a low-pass filter; application of a Fourier transform; application of the Radon transform; application of the Huff transform; application of a wavelet transform; thresholding; and generation of a binary image. The region of interest may be determined manually by the user or automatically by recognizing features in the image generated by the optical sensor 126.

[0181] For example, the illumination unit 118 may be configured to generate and / or project a point cloud such that multiple illumination regions are generated on the light sensor 126, for example, a CMOS detector. Furthermore, disturbances such as speckle and / or external light and / or multiple reflections may be present on the light sensor 126. The processing unit 114 may be adapted to determine at least one region of interest, for example, one or more pixels illuminated by the light beam used to determine the longitudinal coordinates of each reflection feature, which will be described in more detail below. For example, the processing unit 114 may be adapted to perform filtering methods, for example, blob analysis and / or edge filtering and / or object recognition methods.

[0182] The processing unit 114 may be configured to perform at least one image correction. The image correction may include at least one background subtraction. The processing unit 114 may be adapted to remove the influence of background light from the beam profile, for example, by imaging without further illumination.

[0183] The processing unit 114 can be configured to determine the beam profile of each reflection feature. Determining the beam profile may include identifying and / or selecting at least one reflection feature provided by the optical sensor 126, and evaluating at least one intensity distribution of the reflection feature. As an example, a region of a matrix may be used and evaluated to determine an intensity distribution, such as a three-dimensional or two-dimensional intensity distribution, along an axis or line passing through the matrix. As an example, the center of illumination by the light beam may be determined by determining at least one pixel having the best illumination, and a cross-sectional axis may be selected through the center of illumination. The intensity distribution may be an intensity distribution as a function of coordinates along this cross-sectional axis passing through the center of illumination. Other evaluation algorithms are also possible.

[0184] The processing unit 114 is configured to determine, by analyzing the beam profile, at least one first beam profile information of a reflection feature located within the image region of the second image corresponding to the image region of the first image containing the identified geometric features. The method may include identifying the image region of the second image corresponding to the image region of the first image containing the identified geometric features. Specifically, the method may include matching pixels of the first and second images and selecting pixels of the second image corresponding to the image region of the first image containing the identified geometric features. The method may further include considering additional reflection features located outside the image region of the second image.

[0185] Beam profile information may be, or may include, any information and / or characteristics derived from and / or related to the beam profile of the reflection features. The first beam profile information and the second beam profile information may be identical or different. For example, the first beam profile information may be intensity distribution, reflection profile, intensity center, and material features. For skin detection in step b) 116, beam profile analysis may be used. Specifically, beam profile analysis classifies materials using the reflection characteristics of coherent light projected onto the surface of an object. The classification of materials may be carried out as described in WO2020 / 187719, EP application 20159984.2 filed on 28 February 2020 and / or EP application 20154961.5 filed on 31 January 2020, the full contents of which are incorporated by reference. Specifically, a periodic grid of laser spots, such as a hexagonal grid as described in EP application 20170905.2 filed on April 22, 2020, is projected, and the reflected images are recorded by a camera. Analysis of the beam profile of each reflected feature recorded by the camera may be performed by a feature-based method. See the description above for details on feature-based methods. Feature-based methods can be used in combination with machine learning methods that enable parameterization of skin classification models. Alternatively, or in combination, skin can be classified using a convolutional neural network with the reflected images as input.

[0186] The skin detection step 116 may include determining at least one material property of the reflective feature from the beam profile information by using a processing unit 114. Specifically, the processing unit 114 is configured to identify a reflective feature produced by irradiating biological tissue, in particular human skin, if its reflective beam profile satisfies at least one predetermined or predefined criterion. The at least one predetermined or predefined criterion may be at least one property and / or value suitable for distinguishing biological tissue, in particular human skin, from other materials. The predetermined or predefined criterion may be, or include, at least one predetermined or predefined value and / or threshold and / or threshold range that reference a material property. The reflective feature may be indicated as produced by biological tissue if the reflective beam profile satisfies at least one predetermined or predefined criterion. Otherwise, the processing unit is configured to identify the reflective feature as non-skin. Specifically, the processing unit 114 may be configured for skin detection, in particular identifying whether the detected face is human skin or not. If the material is biological tissue, particularly human skin, identification may include determining and / or verifying whether the object under test or the surface under test is or contains biological tissue, particularly human skin, and / or distinguishing biological tissue, particularly human skin, from other tissues, particularly other surfaces. The methods according to the present invention may enable the distinction of human skin from one or more of inorganic tissues, metal surfaces, plastic surfaces, foams, paper, wood, displays, screens, and fabrics. The methods according to the present invention may enable the distinction of human biological tissue from artificial or inanimate surfaces.

[0187] The processing unit 114 may be configured to determine the material property m of the surface emitting the reflective feature by evaluating the beam profile of the reflective feature. The material property may be at least one arbitrary property of the material configured for characterizing and / or identifying and / or classifying the material. For example, the material property may be a property selected from the group consisting of roughness, depth of light transmission into the material, properties characterizing the material as a biological or non-biological material, reflectance, specular reflectance, diffuse reflectance, surface properties, measure of light transmission, scattering, specifically backscattering behavior, etc. At least one material property may be a property selected from the group consisting of scattering coefficient, light transmission, transparency, deviation from Lambertian surface reflection, speckle, etc. Determining at least one material property may include assigning the material property to the detected face. The processing unit 114 may include at least one database containing lists and / or tables, such as lookup lists and / or lookup tables, of predefined and / or predetermined material properties. A list and / or table of material properties can be determined and / or generated by performing at least one test measurement, for example, by performing a material test using a sample with known material properties. The list and / or table of material properties can be determined and / or generated at the manufacturer's site and / or by the user. Material properties may be further assigned to one or more material classifications, such as material name, material group such as biological or non-biological material, translucent or opaque material, metal or non-metal, skin or non-skin, fur or non-fur, carpet or non-carpet, reflective or non-reflective, specular or non-specular, foam or non-foam, hair or non-hair, roughness group, etc. Processing unit 114 may include at least one database containing lists and / or tables containing material properties and associated material names and / or material groups.

[0188] Feature-based approaches have sufficient accuracy to distinguish between skin and surface-only scattering materials, but distinguishing between skin and carefully selected attack materials, including volume scattering, is more difficult. Step b) 116 may involve using artificial intelligence, particularly convolutional neural networks. Using reflected images as input to a convolutional neural network may allow for the generation of a classification model with sufficient accuracy to distinguish between skin and other volume scattering materials. By selecting important regions in the reflected images, only physically valid information is passed to the network, thus requiring only a compact training dataset. Furthermore, a very compact network architecture can be generated.

[0189] Specifically, in the skin detection step 116, at least one parameterized skin classification model may be used. The parameterized skin classification model may be configured to classify skin and other materials by using a second image as input. The skin classification model may be parameterized by using one or more of machine learning, deep learning, neural networks, or other forms of artificial intelligence. Machine learning may include methods of using artificial intelligence (AI) for automated model building, in particular for model parameterization. The skin classification model may include a classification model configured to distinguish human skin from other materials. Characteristics characteristic of skin can be determined by applying an optimization algorithm with respect to at least one optimization target of the skin classification model. Machine learning may be based on at least one neural network, in particular a convolutional neural network. The weights and / or topology of the neural network may be predetermined and / or predefined. Specifically, training of the skin classification model can be carried out using machine learning. The skin classification model may include at least one machine learning architecture and model parameters. For example, the machine learning architecture may be one or more of the following, or may include them: linear regression, logistic regression, random forest, naive Bayes classification, nearest neighbor, neural network, convolutional neural network, generative adversarial network, support vector machine, or gradient boosting algorithm. Training may include the process of building a skin classification model, in particular the process of determining and / or updating the parameters of the skin classification model. The skin classification model may be at least partially data-driven. For example, the skin classification model may be based on experimental data, such as data determined by illuminating multiple humans and artificial objects such as masks and recording the reflection patterns. For example, training may include using at least one training dataset, the training dataset including images of humans and multiple artificial objects having known material properties, in particular a second image.

[0190] The skin detection step 116 may include using at least one 2D face and face landmark detection algorithm configured to provide at least two locations of characteristic points on a human face. For example, these locations may be the eye locations, the forehead, or the cheeks. The 2D face and face landmark detection algorithm can provide the locations of characteristic points on a human face, such as the eye locations. Because there are subtle differences in the reflections of different zones of the face (e.g., the forehead and cheeks), region-specific models can be trained. Preferably, at least one region-specific parameterized skin classification model is used in the skin detection step 116. The skin classification model may include multiple region-specific parameterized skin classification models, such as different regions, and / or the skin classification model may be trained using region-specific data, such as by filtering the images used for training. For example, two different regions may be used for training, such as from the inner corner of the eye to below the nose, and the forehead region may be used in particular if sufficient reflective features cannot be identified within this region. However, other regions are also possible.

[0191] A detected face is characterized as skin if its material properties correspond to at least one characteristic property of skin. The processing unit 114 may be configured to identify a reflective feature as generated by irradiating biological tissue, particularly skin, if its corresponding material properties satisfy at least one predetermined or predefined criterion. A reflective feature may be identified as generated by human skin if its material properties indicate "human skin". A reflective feature is identified as generated by human skin if its material properties fall within at least one threshold and / or at least one range. The at least one threshold and / or range may be stored in a table or lookup table, may be determined empirically, for example, or may be stored in at least one data storage device of the processing unit. The processing unit 114 is configured to identify a reflective feature as otherwise background. Thus, the processing unit 114 may be configured to assign a material property (e.g., skin yes or no) to each projection spot.

[0192] The 3D detection step 120 may be performed after the skin detection step 116 and / or the face detection step 110. However, other embodiments are also possible in which the 3D detection step 120 is performed before the skin detection step 116 and / or the face detection step 110.

[0193] The 3D detection step 120 includes determining second beam profile information for at least four reflection features located within the image region of the second image, corresponding to the image region of the first image, by analyzing the beam profile, including the identified geometric features. The second beam profile information may include the quotient Q of the beam profile area.

[0194] Beam profile analysis may include evaluation of the beam profile and may include at least one mathematical operation and / or at least one comparison and / or at least one symmetrization and / or at least one filtering and / or at least one normalization. For example, beam profile analysis may include at least one of the following steps: histogram analysis step, calculation of difference measurements, application of a neural network, and application of a machine learning algorithm. Processing unit 114 may be configured to symmetrize and / or normalize and / or filter the beam profile in particular to remove noise or asymmetry from recordings at larger angles, edge recordings, etc. Processing unit 114 may filter the beam profile by removing high spatial frequencies, such as by spatial frequency analysis and / or median filtering. Aggregation may be performed by averaging all intensities at the same distance to the center of the light spot intensity. Processing unit 114 may be configured to normalize the beam profile to the maximum intensity in particular to account for intensity differences due to recorded distances. The processing unit 114 may be configured to remove the influence of background light from the beam profile, for example, by imaging without illumination.

[0195] The processing unit 114 analyzes the beam profile of each reflection feature and determines at least one vertical coordinate z for reflection features located within the image region of the second image corresponding to the image region of the first image containing the identified geometric features. DPR The processing unit 114 may be configured to determine the longitudinal coordinate z by using a so-called depth technique from photon ratio, also known as beam profile analysis. DPR It may be configured to determine the depth from photon ratio (DPR) technique, see WO2018 / 091649A1, WO2018 / 091638A1 and WO2018 / 091640A1, the full contents of which are included by reference.

[0196] The vertical coordinate of the reflection feature may be the distance between the light sensor 126 and a point in the scene emitting the corresponding illumination feature. Analysis of one beam profile of the reflection feature may include determining at least one first area and at least one second area of ​​the beam profile. The first area of ​​the beam profile may be area A1, and the second area of ​​the beam profile may be area A2. The processing unit 114 may be configured to integrate the first area and the second area. The processing unit 114 may be configured to derive a combined signal, in particular a quotient Q, by one or more of the following: dividing the integrated first area and the integrated second area, dividing by a multiple of the integrated first area and the integrated second area, or dividing by a linear combination of the integrated first area and the integrated second area. The processing unit 114 may be configured to determine at least two areas of the beam profile and / or divide the beam profile into at least two segments having different areas of the beam profile, where overlapping areas are possible as long as the areas do not coincide. For example, the processing unit 114 may be configured to determine multiple areas, such as two, three, four, five, or up to ten areas. The processing unit 114 may be configured to divide the optical spot into at least two areas of the beam profile, and / or to divide the beam profile into at least two segments containing different areas of the beam profile. For at least two of the areas, the processing unit 114 may be configured to determine the integral of the beam profile across each area. The processing unit may be configured to compare at least two of the determined integrals. Specifically, the processing unit 114 may be configured to determine at least one first area and at least one second area of ​​the beam profile. The areas of the beam profile can be any region of the beam profile at the optical sensor position used to determine the quotient Q. The first area and the second area of ​​the beam profile may be adjacent regions, overlapping regions, or both.The first area and the second area of ​​the beam profile do not have to be the same area. For example, the processing unit 114 may be configured to divide the sensor area of ​​the CMOS sensor into at least two sub-regions, and the processing unit may be configured to divide the sensor area of ​​the CMOS sensor into at least one left portion and at least one right portion, and / or at least one upper portion and at least one lower portion, and / or at least one inner portion and at least one outer portion. Additionally or alternatively, the camera 112 may include at least two light sensors 126, and the photosensitive areas of the first light sensor 126 and the second light sensor 126 may be arranged such that the first light sensor 126 determines the first area of ​​the beam profile of the reflective features, and the second light sensor 126 determines the second area of ​​the beam profile of the reflective features. The processing unit 114 may be adapted to integrate the first area and the second area. The processing unit 114 may be configured to use at least one predetermined relationship between the quotient Q and the vertical coordinate to determine the vertical coordinate. The predetermined relationship may be one or more of empirical relationships, semi-empirical relationships, and analytically derived relationships. The processing unit 114 may include at least one data storage device for storing the predetermined relationships, such as a lookup list or a lookup table.

[0197] The 3D detection step may include determining at least one depth level from the second beam profile information of the reflection feature by using a processing unit.

[0198] The processing unit 114 may be configured to determine a depth map of at least a part of the scene by determining depth information of at least one of the reflection features located within the image region of the second image corresponding to the image region of the first image including the identified geometric features. The processing unit 114 may be configured to determine the depth information of the reflection features by one or more of the following techniques: depth from photon ratio, structured light, beam profile analysis, time of flight, shape from motion, depth from focus, triangulation, depth from defocus, stereo sensors. The depth map may be a thinly populated depth map including a small number of entries. Alternatively, the depth may be a heavily populated one including a large number of entries.

[0199] The detected face is characterized as a 3D object if the depth level deviates from a predetermined or predefined depth level of a planar object. Step c) 120 may include using 3D topology data of the face in front of the camera. The method may include determining a curvature from at least four of the reflection features located within the image region of the second image corresponding to the image region of the first image including the identified geometric features. The method may include comparing the curvature determined from at least four reflection features with a predetermined or predefined depth level of a planar object. If the curvature exceeds the assumed curvature of the planar object, the detected face is characterized as a 3D object; otherwise, it is characterized as a planar object. The predetermined or predefined depth level of the planar object may be stored in at least one data storage of the processing unit as a look-up list or a look-up table. The predetermined or predefined level of the planar object may be determined experimentally and / or may be a theoretical level of the planar object. The predetermined or predefined depth level of the planar object may be at least one limit value for at least one curvature and / or a range for at least one curvature.

[0200] The 3D features determined in step c) 120 can distinguish a high-quality photo from a 3D facial structure. By combining step b) 116 and step c) 120, the reliability of authentication against attacks can be enhanced. The 3D features can be combined with material features to increase the security level. Since the same computational pipeline can be used to generate input data for skin classification and 3D point cloud generation, both characteristics can be calculated with a small amount of computation from the same frame.

[0201] Preferably, following step a) 110, step b) 116, and step c) 120, an authentication step 122 may be performed. The authentication step 122 may be performed partially after each of steps a) to c). If no face is detected in step a) 110, and / or if it is determined in step b) 116 that the reflection features are not generated by the skin, and / or if the depth map in step c) 120 refers to a planar object, the authentication may be aborted. The authentication step includes authenticating the detected face using at least one authentication unit when the face detected in step b) 116 is characterized as being skin and the face detected in step c) 120 is characterized as being a 3D object.

[0202] Steps a) to d) can be implemented by using at least one device, such as at least one mobile device 124 like a mobile phone or a smartphone, and access to the device is protected by using face authentication. Other devices are also possible, such as an access control device that controls access to buildings, machines, automobiles, etc. The method may include permitting access to the device when the detected face is authenticated.

[0203] The method may include at least one registration step. The registration step may register a user of the device. Registration may include a process of registering and / or signing up and / or teaching in a user for subsequent use of the device. Typically, registration may be performed when the device is first used and / or when the device is started up. However, embodiments are possible in which multiple users are registered, for example, sequentially, so that registration may be performed and / or repeated at any time during use of the device. Registration may include generating a user account and / or user profile. Registration may include inputting and storing user data, in particular image data, through at least one user interface. Specifically, at least one 2D image of the user is stored in at least one database. The registration step may include imaging at least one image of the user, in particular multiple images. Images may be recorded from different orientations and / or the user may change orientation. Furthermore, the registration step may include generating at least one 3D image and / or depth map of the user, which may be used in step d) for comparison. The database may be, for example, the database of processing unit 114, and / or an external database such as a cloud. The method includes identifying the user by comparing the user's 2D image with a first image. The method according to the present invention may enable a significant improvement in the presentation attack detection capability of the biometric authentication method. To improve overall authentication, in addition to the user's 2D image, 3D topological features as well as fingerprints, which are person-specific materials, can also be stored during the registration process. This allows for multi-factor authentication within a single device by using 2D, 3D, and material-derived features.

[0204] The method according to the present invention, using beam profile analysis technology, can reliably detect human skin and provide a concept to distinguish it from reflections from attack materials designed to mimic a face by analyzing the reflection of a laser spot on the face, particularly the reflection of a laser spot in the NIR region. Furthermore, beam profile analysis simultaneously provides depth information by analyzing the same camera frame. Therefore, skin security functionality as well as 3D can be provided by the exact same technology.

[0205] Since two-dimensional images of faces can also be recorded simply by turning off the laser irradiation, a completely secure face recognition pipeline can be established that solves the above problems.

[0206] Shifting the laser wavelength to the NIR region makes the reflective properties of human skin more similar across ethnic origins. At a wavelength of 940 nm, the difference is minimized. Therefore, differences in ethnic origin do not affect skin authentication.

[0207] Presentation attack detection (by skin classification) is provided by just one frame, so time-consuming analysis of a series of frames may not be necessary. The time frame for implementing the complete method may be 500 milliseconds or less, preferably 250 milliseconds or less. However, embodiments in which skin detection is performed using multiple frames may be possible. Depending on the confidence for identifying reflective features in the second image and the speed of the method, the method may include sampling reflective features across several frames to arrive at a more stable classification.

[0208] Figure 3 shows the experimental results, particularly the density of skin scores as a function. The X-axis represents the score, and the Y-axis represents the frequency. The score is a measure of classification quality, with values ​​between 0 and 1, where 1 indicates very high skin similarity and 0 indicates very low skin similarity. The threshold for judgment can be around 0.5. A baseline distribution of skin scores for bona fide presentations was created using 10 subjects. Skin scores were also recorded for presentation attacks (PAs) of levels A, B, and C (as defined in the relevant ISO standards). The experimental setup (evaluation target, TOE) included proprietary hardware equipment, such as shown in Figure 2, and included the necessary sensors and a computing platform on which the PAD software was run. The TOE was tested using six types of PAIs (presentation attack instruments) targets at level A, five types of PAIs at level B, and one type of PAI at level C. Ten PAIs were used for each type of PAI. The types of PAIs used in this study are shown in the table below. In the table, APCER is the attack presentation classification error rate, represented as the number of successful attacks / total attacks * 100. In the table, BPCER is the good-faith presentation classification error rate, represented as the number of rejected unlock attempts / total unlock attempts * 100. Level A and Level B attacks are all based on 2D PAI, while Level C attacks are based on 3D masks. For Level C attacks, custom rigid masks fabricated using a 3D printer were used. A test crew of 10 subjects was used to obtain a reference distribution of skin scores for good-faith presentations.

[0209] [Table 2]

[0210] Experiments using these PAIs showed that the two presentation classes (goodwill or PA) could be clearly distinguished based on skin scores. Using the method according to the present invention, a clear distinction between paper, 3D printed, and skin is possible. [Explanation of symbols]

[0211] 110 Face detection step 112 Cameras 114 Processing Units 116 Skin detection step 118 Irradiation Unit 120 3D detection steps 122 Authentication Steps 124 Mobile Devices 126 Light Sensor

Claims

1. The following steps: a) at least one face detection step (110), the face detection step (110) comprising at least one camera (112) determining at least one first image, the first image comprising at least one two-dimensional image of a scene that is thought to contain a face, and the face detection step (110) comprising at least one processing unit (114) detecting a face in the first image by identifying at least one predefined or predetermined geometric feature characteristic of a face in the first image; b) At least one skin detection step (116), the skin detection step (116) comprising: at least one irradiation unit (118) projecting at least one irradiation pattern including a plurality of irradiation features onto the scene; and at least one camera (112) determining at least one second image, the second image including a plurality of reflective features generated by the scene in response to irradiation by the irradiation features, each of the reflective features including at least one beam profile; the skin detection step determining at least one first beam profile information of the reflective features located within an image region of the second image corresponding to an image region of the first image including identified geometric features, by analysis of the beam profiles; and the processing unit (114) determining at least one material property of the reflective features from the first beam profile information, wherein the detected face is characterized as skin if the material property corresponds to at least one property characteristic of skin; c) at least one 3D detection step (120), the 3D detection step (120) comprising: determining at least four second beam profile information of the reflection feature located within an image region of the second image corresponding to an image region of the first image containing the identified geometric feature by analyzing beam profiles; and the processing unit (114) determining at least one depth level from the second beam profile information of the reflection feature, wherein the detected face is characterized as a 3D object if the depth level deviates from a predetermined or predefined depth level of a planar object; d) at least one authentication step (122), the authentication step (122) comprising at least one authentication unit authenticating the detected face when the detected face is characterized as skin in step b) (116) and the detected face is characterized as a 3D object in step c) (120), Facial recognition methods, including those mentioned above.

2. The facial recognition method according to Claim 1, wherein steps a) to d) are performed by at least one device, access to the device is protected by facial recognition, and the facial recognition method includes granting access to the device when the detected face is recognized.

3. The facial recognition method according to claim 2, wherein the facial recognition method comprises at least one registration step, in which a user of the device is registered, and at least one 2D image of the user is stored in at least one database, and the facial recognition method comprises identifying the user by comparing the 2D image of the user with a first image.

4. The facial recognition method according to any one of claims 1 to 3, wherein the skin detection step uses at least one parameterized skin classification model, the parameterized skin classification model is configured to take the second image as input and classify skin and other materials.

5. The facial recognition method according to claim 4, wherein the skin classification model is parameterized by machine learning, and the skin-specific characteristics are determined by applying an optimization algorithm with respect to at least one optimization target of the skin classification model.

6. The facial recognition method according to claim 4 or 5, wherein the skin detection step includes using at least one 2D face and facial landmark detection algorithm configured to provide at least two locations of characteristic points of a human face, and the skin detection step (116) uses at least one region-specific parameterized skin classification model.

7. The facial recognition method according to any one of claims 1 to 6, wherein the irradiation pattern includes a periodic grid of laser spots.

8. The irradiation feature has a wavelength in the near-infrared (NIR) region, as described in any one of claims 1 to 7, for the face recognition method.

9. The facial recognition method according to claim 8, wherein the irradiation feature has a wavelength of 940 nm.

10. A facial recognition method according to any one of claims 1 to 9, wherein a plurality of second images are determined, and the reflective features of the plurality of second images are used for skin detection in step b) (116) and / or for 3D detection in step c) (120).

11. The facial recognition method according to any one of claims 1 to 10, wherein the camera (112) is at least one near-infrared camera or comprises at least one near-infrared camera.

12. A computer program for facial recognition, configured to cause a computer or computer network to fully or partially perform the facial recognition method described in any one of claims 1 to 11 when executed on a computer or computer network, wherein the computer program for facial recognition is configured to perform and / or execute at least steps a) to d) of the facial recognition method described in any one of claims 1 to 11.

13. A computer-readable storage medium that, when executed by a computer or computer network, includes instructions causing at least steps a) to d) of the facial recognition method described in any one of claims 1 to 11 to be performed.

14. A non-temporary computer-readable storage medium that, when executed by one or more processors, includes instructions causing the one or more processors to perform at least steps a) to d) of the facial recognition method described in any one of claims 1 to 11.

15. A mobile device (124) comprising at least one camera (112), at least one illumination unit (118), and at least one processing unit (114), wherein the mobile device (124) is configured to perform at least steps a) to c) and optionally step d) of the facial recognition method described in any one of claims 1 to 11.

16. The use of a facial recognition method according to any one of claims 1 to 11 for a facial recognition computer program according to claim 12, or a computer-readable storage medium according to claim 13, or a non-temporary computer-readable storage medium according to claim 14, or a mobile device (124) according to claim 15, to detect a biometric presentation attack.

Images